A/B testing

How it works

1. Hypothesis: "If we show pricing on the homepage, qualified demos will increase, because visitors will know whether it fits their budget."
2. Primary metric, chosen before starting.
3. Sample size: how many visits are needed to detect the expected difference. Calculated before the test.
4. Random split of traffic between version A (control) and B (variant).
5. Analysis once the planned sample is reached, not before.

Common mistakes

• Checking results every day and stopping when a winner appears. The best-known mistake, and it sharply increases the chance of false positives. Evan Miller explained it in his article "How Not To Run an A/B Test".
• Samples that are too small: with few conversions, differences are noise.
• Short durations: cover full weeks, because behavior differs between weekdays and weekends.
• Changing many things at once with no way of knowing which caused the effect, unless it's set up as a multivariate test.
• Measuring only clicks and not whether conversions turn into sales.

With low traffic

Many B2B websites don't have enough traffic to detect small improvements. In that case:

• Test big changes, such as a different value proposition or page structure, whose effect, if there is one, is easier to detect.
• Use more frequent metrics, such as CTA clicks, knowing they're a proxy.
• Complement with qualitative methods: usability tests, interviews, session recordings.

Tools

Google Optimize, the most widely used free option, was shut down on 30 September 2023. Today teams use tools such as VWO, Optimizely or AB Tasty, or features built into their CMS and product analytics platforms.

Related terms

• Conversion rate
• Heatmap
• Quick win
• Landing page

Sources

• Evan Miller, How Not To Run an A/B Test and his sample size calculator.