Near real-time pipeline

The near real-time pipeline feature facilitates real-time data usage in the Customer Data Platform (CDP) across campaigns, metrics, and 360 views. Some of the personalized experiences and benefits of this feature are:

• Dynamic segmentation 
• Faster issue resolution
• Informed decision-making
• Agile market response
• Enhanced fraud prevention
• Cross-channel consistency
• Improved customer retention
• Resource optimization
• Advanced analytics

Features¶

The following are the features of the near real-time pipeline:

• Event visibility in A360 Profiles: 

  • Event access in customer profiles within A360
  • Immediate insights into customer interactions and behaviors

• Customer search in 360 Profiles: 

  • Immediate search for newly added customers within the 360 Profiles platform
  • No need for scheduled batch workflows

• Event availability in Snowflake: 

  • Event population into Snowflake tables in real-time
  • No need for scheduled RT-Batch runner workflow for hourly or bi-hourly updates

Capabilities¶

• New events for existing customer records: Immediate access to real-time events associated with existing customer records within the 360 platform. Users can obtain the most recent interactions and behaviors of customers in near real-time, which helps in immediate engagement and decision-making.

  For example, marketers can promptly respond to customer actions, such as website visits or purchases, by initiating personalized campaigns or promotions based on recent behaviors.

• New customer records in 360 Profiles: Access to newly added customer records in the 360 Profiles platform within sub-minute intervals, but not immediately processed by the Ingestion Rules Engine (IRE). Users can retrieve new customer data shortly after it is added to the system, promoting quick access to customer profiles.

  For example, customer support representatives can access basic information about new customers to address initial inquiries or provide preliminary assistance before detailed processing.

Process flow to track events¶

1. The DW-Tracker API tracks events in real-time.

2. The events move through the CDP data pipeline and are then consumed by A360 Profiles.

3. The A360 Profiles UI displays these events with a short delay of 1 to 2 minutes.

Important considerations¶

• Public event visibility: The PUBLIC.EVENT showing real-time events does not mean that data is immediately available in the 360 profile. It has a lag of up to 1 minute.
• Event surfacing in CDP UI: Events surface in the CDP UI within 3 minutes.
• Transition to real-time data: After enabling this feature, the real-time data is not written to Parquet tables.
• Limited real-time support: The current implementation supports only real-time events but does not support entities such as transactions or custom entities.
• Search configuration limitation: The search configuration columns defined in the a360.search.config tenant property remain unchangeable during the first run of real-time enablement. 
• Possible race condition: A possible race condition exists because the entities other than EVENT undergo a full-refresh and might result in the loss of A360 data when the Data Warehouse (DW) runs. The data resurfaces when the DW runs again.
• Stale streams: If real-time data is not received for any tenant, the pipeline streams become stale and require recreation.
• HBase row key population: The response no longer populates the HBase row key.
• Anonymous events: Anonymous events are invisible as they lack a Standard Customer Number (SCN) because the cookie is null, or the cookie exists but the SCN is null.