In an A/B test, some visitors see version A of a page and others see version B, and you compare which one produces more conversions. The method is excellent, but it has one condition that’s easy to forget: it needs a lot of data.
How much traffic you really need
Take an example. A page converts at 2%, two inquiries per hundred visits. You want to check whether a new version lifts that to 2.4%, a one-fifth improvement. Under standard assumptions, 95% confidence and an 80% chance of detecting that difference, you need about 21,000 visits per variant. Over 42,000 in total.
If that page gets 5,000 visits a month, the test would run for more than eight months. In that time the campaigns, the season and the offer itself will change, so the result would be hard to interpret anyway.
The smaller the difference you want to detect, the more traffic you need. That’s why small changes, like a button color, practically can’t be tested reliably on sites with moderate traffic.
Rules without which the test lies
- Set the sample size before you start. Use a sample size calculator and don’t end the test early.
- Don’t peek and stop when the result “looks good”. If you check every day and stop at the first significant difference, the chance of declaring a winner that isn’t there rises sharply.
- Test in full weeks. Monday traffic and Saturday traffic behave differently.
- Pick one primary metric before you start, not after seeing the results.
- Don’t run overlapping tests on the same path.
- Check the traffic split. If it was meant to be 50/50 and came out 55/45, something in the setup is wrong and the result isn’t reliable.
What’s worth testing
Since a test needs a lot of traffic, test changes that could make a big difference:
- the headline and the promise on the first screen,
- the structure of the offer, such as packages instead of a single price,
- the length of the form and what you ask in it,
- how the price is presented,
- the order of sections on a service page.
Leave the button color for last, if at all.
What to do when there isn’t enough traffic
Most B2B company websites have too little traffic for reliable A/B tests. That doesn’t mean decisions have to be made blind:
- User testing. A few sessions where people from the target group complete tasks on the site usually reveal the biggest problems. It doesn’t give you percentages, but it gives you reasons.
- Session recordings and click maps. They show where people stop, what they look for and where they give up.
- Before-and-after comparison. Less certain than a test, because many things change in the meantime, but with a big change and the context noted, it gives a reasonable signal.
- Talking to sales. The questions customers ask salespeople show what the website doesn’t explain.
It’s worth tracking the results of changes in the same sheet as the rest of the numbers, which we described in the marketing report for leadership.
Tools
Since Google shut down Optimize in September 2023, companies have used many other tools, from simple visual testing tools to tests run in the site’s code. The choice of tool matters less than the rules in this article: no tool will save a test with too small a sample.
If your site gets traffic but no inquiries, start by checking its structure and content before you test the details. See how we work on websites.
Let’s talk about your project
The first call takes 30 minutes and is free. After it, you know whether we can help and what it will cost.
Book a call
