A/B Testing: Running Experiments That Give Real Answers
Key takeaway
Decide the primary metric, the minimum effect worth detecting, and the sample size before launching — then leave the test alone until it reaches that size. Stopping early because the result looks good is the most common way teams ship changes that do nothing.
Experimentation is the only reliable way to know whether a change helped. It's also easy to run in a way that produces confident, wrong conclusions — which is worse than not testing at all.
Set it up before you launch it
- One primary metric. Additional metrics are guardrails and context, not additional chances to declare victory.
- The minimum effect size worth acting on. A 0.2% lift may be real and still not worth the complexity of the change.
- Required sample size, calculated from your baseline rate and that effect size. Small-traffic sites often discover the honest answer is that they cannot detect the difference they're hoping for.
- A planned duration covering at least one full business cycle — usually one to two weeks — because behaviour differs by day.
The mistakes that invalidate results
- Peeking and stopping when the result crosses significance. Repeatedly checking inflates false positives dramatically.
- Testing many variants or metrics without accounting for multiple comparisons — test enough things and something will look significant by chance.
- Changing the test mid-flight: adjusting traffic split, editing a variant, or fixing a bug resets comparability.
- Ignoring the novelty effect — existing users react to change itself, and the first days often overstate the difference.
- Assigning by session rather than by user, so the same person sees both variants.
Reading the outcome honestly
- A non-significant result is information: it usually means the change didn't matter much, which is worth knowing.
- Report the confidence interval, not just the point estimate. "Between -1% and +6%" is a very different conversation from "+2.5%".
- Check the guardrails — a conversion lift that increases refunds or support contacts isn't a win.
- Segment afterwards only to generate hypotheses, never to rescue a failed test.
What's worth testing
Test decisions that are expensive to reverse or genuinely contested — pricing presentation, checkout flow, onboarding steps, major layout changes. Don't burn traffic testing button colours, and don't test a change that's obviously correct: fixing a broken form doesn't need an experiment. Keep a log of every test and its result; the accumulated record of what didn't work is one of the most valuable product assets a team can have.