A/B Testing Guide

A/B testing compares two versions of something to figure out which performs better. This testing method helps marketers with questions like, “What is most likely to make people visit my website? Or buy our product?” It’s used to evaluate everything from website design to email blast headlines.

In A/B testing, A refers to the ‘control’ or the initial testing variable. B refers to the ‘variation’ or new version of the testing variable. This type of test lets you collect information about changes to your website and the impact of those changes.

How it Works

In an A/B test example, you take a direct mail piece and modify it to create a second version, this time with a different look or offer. This change can be as simple as a single headline, coupon, or a complete redesign of the piece. Then, half of your audience receives the original version of the mailer (known as the control), and half sees the newly edited version of your mail (the variation).

As recipients are given either the control or variation, their actions are measured to help you determine whether changing the elements had a positive, negative, or neutral effect on visitor behavior.

Why you should A/B test

A/B testing allows you to make changes to existing content while gaining insight into how they impact your audience’s behavior.

More than just answering a one-off question or settling a disagreement, A/B testing can be used to continually improve a given experience or meet a single goal, such as conversion rate over time.

A/B Testing Guide

The following is an A/B testing framework you can use to start running tests:

• Identify your control variable: As you evaluate your direct mail campaigns, you might find a few variables you want to test. But to evaluate how effective a change is, you’ll want to isolate a single variable and measure its performance.

• Create variations: Make the desired changes to an element of your mail piece. This might be changing the color, design, or swapping the offer.

• Run experiment: Kick off your experiment and wait for visitors to participate!

• Track the Results: After creating variations and kicking off your experiment, you’ll want to track the performance of your A/B Test. One of the easiest ways to track the responses of each variation is through a unique phone number on each mail piece.

• Analyze results: Once your experiment is complete, it’s time to analyze the results. The data, in the form of responses you collect show you the difference between how the two versions performed and whether there is a significant difference in engagement from your audience.