A/B testing best practices
Get clear answers from your tests. This page helps you choose what to test, set up versions that differ enough, pick the right conversion and know when a result is real. The steps themselves are on Setting up A/B tests.
What to test
Test the parts that decide whether a visitor acts:
- Headlines and hero sections: the first thing visitors see; small wording changes can matter
- Calls to action: the text, colour, size and place of a button
- The copy around a form: the heading and the promise above a sign-up form
- Offers: how you present a price, a deadline or a benefit
Footers, legal text and decoration rarely change what visitors do. For the menu, look at your web statistics instead.
Design the versions
Change one thing in an A/B test
With two versions, change one thing between A and B. If you change the headline, the picture and the button at once, you cannot tell which change made the difference.
With three or more versions, each one can try a different idea. Keep the rest of the element the same.
Make the versions clearly different
"Get started" against "Get started today" will hardly show a difference. Test real alternatives:
- "Get started" against "Try it free" against "See a demo"
- A large photo against a product picture against a video
Choose the right conversion
| Conversion | Best for |
|---|---|
| Time on element | Hero sections and text blocks, where reading time shows interest |
| Time on page | Whole-page layouts, where staying on the page is the goal |
| Click on element | Buttons, links and calls to action |
Set Seconds (1 to 60) to a realistic reading time: about 5 seconds for a short headline, 15 to 30 seconds for a section of text. To count a click somewhere else on the page, such as the submit button of a form, enter its CSS selector.
Run the test
Wait for stable scores
Thompson Sampling adapts from the first visit, but early numbers swing. With 20 impressions, one extra conversion moves the Conversion Score by about 4.5 points; with 200, by about half a point. Watch the score, not the raw Conversion Rate: the score accounts for how much data there is.
Leave the versions alone
Editing a version keeps its old numbers, so the results then mix the old and the new version. If you change a version a lot, reset the test with Delete on the report page; see Analysing results.
Read the result
When to declare a winner
Look for these signals in the RL report:
- The leader's Conversion Score stays steady over several days.
- The leader gets clearly more Impressions than the others, because the RL module sends it more visitors.
- The leader stays ahead when you change the Time range of the chart.
Then keep the winner: remove the other versions in the editor and save.
When results stay close
Similar scores after many impressions mean the versions perform the same. That is a valid result: keep the version you prefer and test something else.
Page speed and caching
The page contains every version, and the browser shows only the one the RL module picks. Until the pick arrives, all versions stay hidden. If rl.php cannot be reached, visitors get version A.
Because the HTML is the same for everyone, page caches and CDNs can keep serving it. The browser counts each impression and conversion once per version per session, so reloading a page does not inflate the numbers. The RL module flags the core Page Cache module on the status report; follow its advice there.
What's next?
Analysing results from your tests to continue exploring DXPR Builder.