Information for: DEVELOPERS   PARTNERS

Data Feeds via SFTP

Overview

The Customer Data Platform (CDP) platform is built around several entities (e.g. Customer, Product, etc…). The CDP data feeds provides a standard set of flat-file feeds to define how you can send the relevant data into the CDP platform. They are designed to support all the use cases CDP applications provide to you.

While the feeds are presented as a fixed set of entities and attributes, they can be extended with client-specific attributes as well as client-specific entities as needed. You should discuss your needs with your CDP Client Success team to configure such extensions.

Feeds

Here is the list of standard feeds (required) :

Feed Feed Name
Customer Customer_[DateTime].csv
Address Address_[DateTime].csv
CustomerAddressXRef CustomerAddressXRef_[DateTime].csv
Transaction Transaction_[DateTime].csv
TransactionItem TransactionItem_[DateTime].csv
Organization Organization_[DateTime].csv
Product Product_[DateTime].csv
ProductCategory ProductCategory_[DateTime].csv
ProductCategoryXRef ProductCategoryXRef_[DateTime].csv

Here are optional feeds that may be required for specific use cases:

Feed Feed Name
Event Event_[DateTime].csv
PaymentMethod PaymentMethod_[DateTime].csv
TransactionPaymentMethodXRef TransactionPaymentMethodXRef_[DateTime].csv
Campaign Campaign_[DateTime].csv
Dispatch Dispatch_[DateTime].csv
Message Message_[DateTime].csv

Note

  • The full feeds are not required every day. After an initial full feed, it is more efficient to send only the new/updated data to CDP as incremental feeds.
  • Web and email events come in through other mechanisms, and are therefore not part of the required feeds.

Feed Format

The CDP feeds were designed to be source system-agnostic, and as such they have formatting requirements explained below. Note that if you are unable to extract the necessary feeds from your existing systems, CDP offers integration services.

  • Name: Each file should follow the naming convention entityName_yyyymmddhhmiss.csv (e.g. customer_20201224235959.csv). If multiple sources feed the same CDP entity, then we recommend adding a source identifier to the front of the file name sourceName_entityName_yyyymmddhhmiss.csv (e.g. crm_customer_20201224235959.csv).
  • Format: comma-separated values (CSV), with the conventions used by OpenCSV.
  • Record delimiter: line breaks ({CR}{LF})
    • Record delimiters cannot be present within fields and should be replaced with a whitespace character
  • Field delimiter: comma (,), tab (\t), or pipe (|)
    • If your data contains field delimiters, please use double quotes as text qualifiers
    • Without text qualifiers double quotes in your data should be escaped with a backslash: 3"3\"
  • Text qualifier (recommended): double quotes (“)
    • Double quotes within fields need to be escaped by prepending them with another double quote (“”), ” → “”. Or by prepending a double-quote with a backslash (“), ” → “.
    • If text qualifiers are not used leading and trailing whitespace will be trimmed.
  • Escape character: backslash ()
    • Escape characters within fields should be replaced by two escape characters. → \
  • A header row that names the columns is required. Column names cannot begin or end with whitespace.
  • A footer row is not supported
  • Files should be encoded using UTF-8
  • The following date/datetime formats are supported by default:
    • dd-MM-yyyy
    • yyyy-MM-dd
    • MM/dd/yyyy
    • yyyy/MM/dd
    • dd MMM yyyy
    • dd MMMM yyyy
    • yyyyMMdd HHmm
    • dd-MM-yyyy HH:mm
    • yyyy-MM-dd HH:mm
    • MM/dd/yyyy HH:mm
    • yyyy/MM/dd HH:mm
    • dd MMM yyyy HH:mm
    • dd MMMM yyyy HH:mm
    • yyyyMMdd HHmmss
    • dd-MM-yyyy HH:mm:ss
    • yyyy-MM-dd HH:mm:ss
    • MM/dd/yyyy HH:mm:ss
    • yyyy/MM/dd HH:mm:ss
    • dd MMM yyyy HH:mm:ss
    • dd MMMM yyyy HH:mm:ss
    • yyyy-MM-dd HH:mm:ss.SSS
    • Additional date/datetime formats can be added custom by your CDP integration team
  • The timezone of the file should be communicated to the CDP integration team
  • Each file can be compressed via zip or gzip compression. Each file must be compressed separately, multiple files cannot be compressed together in a compressed directory. The compression method should be included in the file name (e.g. customer_20201114232015.csv.zip or customer_20201114232015.csv.gz).
  • Each file can be encrypted via pgp or gpg encryption. Each file must be encrypted separately, and should not be signed. The encryption method should be included in the file name (e.g. customer_20201114232015.csv.pgp or customer_20201114232015.csv.gpg). If you would like to encrypt your files, please reach out to your Implementation Consultant who will provide you a public key for encrypting the files.

Special notes

Note

Each file should contain only one record per primary key, the latest and greatest version of that record. Since CDP is a distributed system, files are not processed top to bottom or bottom to top. Each file is split up and processed by multiple machines. If two records exist in one file with the same identifier, then CDP cannot guarantee which version of that record will be recorded to CDP’s data warehouse.

Customer & Address feeds

  • CDP supports having the Address entity in the Customer feed (in the case of 1:1 or n:1 relationships). To support n:n relationships however requires those entities to be in separate feeds.
  • By providing 2 separate feeds, CDP supports the use case where one customer can have multiple addresses, and one address can be used by multiple customers.
  • The Address feed can therefore contain multiple lines with the same SourceCustomerNumber. However, the SourceCustomerAddressNumber should always be unique within that feed. If the same address corresponds to multiple customers, you should create multiple rows for that address (1 per customer it is linked to) and maintain the uniqueness of the SourceCustomerAddressNumber key.

Product & Product Category feeds

CDP supports having products mapped to the product categories within the product feed (1:1 or n:1 relationships). To support n:n relationships however requires a separate cross-reference feed (the optional ProductCategoryXRef feed mentioned in the sections above).