Split (A/B) testing guidelines
General guidelines
-
Test email versions that differ by one variable in the template, subject line or sender name.
-
Form email variants for split tests that differ only slightly. This will simplify the measurement of user responses and test result evaluations.
-
Create an email variant that will be used as the primary. A "Primary" bulk email has no changes made to it. Test alternative variants together with the primary one and estimate the results.
Audience guidelines
-
The audience to whom the templates variants are sent must be large enough to receive a statistically significant result. It is recommended to include at least 1000 contacts in a split test.
-
Perform split testing using an audience that belongs to one segment, to minimize the effect of the time factor on the result.
-
Before starting the A/B testing, perform A/A testing to make sure that the audience segment is homogeneous. Send the same email variant to two groups of the target audience that are equal by the number of contacts. If the conversion indicators are the same for the two groups, it means that the segment is homogeneous.
Evaluation guidelines
-
Analyze responses of the recipients after the split test start (from 2 to 72 hours depending on the email type).
-
Estimate conversion rates of all emails that were included in the split test at the same time to eliminate the influence of time on the result.
-
When estimating results, pay attention to the metrics that are influenced by the tested item. For example, the call-to-action button color (CTA) may influence the number of clicks and the wording of the email subject line may influence the number of opens.