The most common A/B test result I've seen isn't "B won" or "A won." It's "B looks a bit better, but we're not really sure, and the test ran for a week because we needed to make a decision."
That's not a failed test exactly, but it's a test that didn't do its job. Most of the time, the problem was decided before the test even started, in how it was planned.
Start with one hypothesis
Write the hypothesis as a sentence: "If we show the full price including taxes on the search results page, more people who start checkout will complete it, because they won't be surprised at the end."
That sentence names the change, the expected effect, the audience and the reason. If you can't write it, the test probably isn't ready.
Choose one primary metric
Pick the one number that decides whether the test worked. Then pick a few guardrail metrics that shouldn't get worse.
| Test | Primary metric | Guardrails |
|---|---|---|
| Full price on search results | Checkout completion | Search to details rate, revenue per visitor |
| New home screen row on TV | Plays started from home screen | Play failures, session length |
| Shorter sign-up form | Sign-up completion | First-week activity of new users |
| AI-drafted reply suggestion | Replies sent with suggestion | Edit rate, customer follow-ups |
If you watch ten metrics and declare victory on whichever moves, you'll find false wins. Decide in advance.
Work out the sample size
Small effects need a lot of traffic to detect. If your expected improvement is modest and your traffic is limited, the test might need to run for a long time, or it might not be able to give a clear answer at all.
Tools like Evan Miller's sample size calculator make this quick to check. 1 I'd do this before building anything. If the numbers say you'd need months of traffic, consider whether a test is the right tool or whether a bigger, bolder change would be easier to measure.
Fix the duration and stick to it
Decide how long the test runs before starting, based on the sample size and at least one full weekly cycle. Then don't stop early just because the result looks good on day three.
Checking results repeatedly and stopping when they look significant is one of the most common ways to get a false positive. If you need to monitor for problems, watch the guardrails, not the primary metric.
Make sure the split is clean
Check that users are randomly assigned, that the same user always sees the same version, and that both groups look similar at the start. If one group has far more mobile users, the comparison isn't fair.
For multi-platform products, check the split per platform. A feature might help on web and hurt on TV, and an overall average can hide that. A staged rollout with feature flags can look similar to a test, but it's not the same thing unless it's set up this carefully.
Read the result honestly
When the test ends:
- If the primary metric improved and guardrails held, that's a real win. Ship it.
- If nothing changed, that's useful too. The idea didn't make a measurable difference, at least not at this scale.
- If the primary improved but a guardrail got worse, you have a trade-off to discuss, not a win.
Kohavi, Tang and Xu's book on trustworthy online experiments covers these pitfalls and many others in detail. 2 It's the most useful single resource I know on the subject.
Keep a record
Write a short summary of every test: hypothesis, result, decision and anything surprising. Over time this becomes one of the most valuable documents a product team has. It stops you from re-running tests that already failed and helps new people understand what's been tried.
I'd also look at results by segment when it makes sense, like new versus returning users. Just treat segment findings as ideas for the next test, not conclusions. If you slice the data enough ways, something will always look interesting. The AI adoption metrics post has more on choosing denominators that don't mislead you.