Definition of an A/B test in an Customer Data Platform (CDP) campaign¶

A/B testing is the idea of splitting an audience into multiple groups at random, following a specific split (typically based on percentage of the entire audience), to send them different variants of a given campaign. This includes the concept of a “Holdout” group, which should not receive any campaign/ treatment, to compare the metrics from the groups that received the campaign against the groups that did not (ie the “true lift” that the campaign provided).

CDP provides A/B testing capabilities, which depend on the execution channel :

• For an SFTP campaign, if you setup an A/B test, you will get additional columns automatically added to your output file to signify which variant each master customer was included in. This can then be loaded into other systems to actually execute that A/B test basd on that random distribution of contacts across multiple variants. The “Holdout” group will have a specific “Holdout” name in that additional column, so you can keep it aside and not send that campaign/treatment to it.

• For a SendGrid campaign, if you setup an A/B test, you will effectively get the ability to design each variant’s content & email layout in the Design step (more details on the Design step itself can be found here : . Then each person will receive the variant they were randomly assigned. Remember : the “Holdout” group will simply not be sent the campaign at all, but will be extracted as part of the campaign execution for future campaign reporting.

Setting up an A/B test¶

Setting up an A/B test is easy :

1. On the A/B test step, turn the switch “on”. You will then see a default split of 2 variants with default names, each setup to contain 50% of the audience you selected in the Audience step.

2. You can rename those variants as needed, to better remember the treatment each of those should receive. For SendGrid campaigns, those names will be reflected in the tabs under the Design step, so you can design each variant in parallel. For SFTP campaigns, those names will be what you see in the additional columns in your file.

3. To add another variant, click on the “+ Add Variant” button. To remove a variant, click on the small “x” next to the variant.

4. To include a holdout, select the holdout checkbox, and enter the size (in percent of the total audience) that the holdout should contain.

5. You can adjust the size of a variant by entering a number (representing the percentage of the total audience that should be included in this variant). Obviously, the sum of all variants’ size (including the Holdout group) should total 100%, and the UI will let you know if that’s not the case.