The A/B test: checking a change honestly
On the pizzeria's site you changed the "Order" button, and conversion grew from 10% to 12%. Is it the button's doing — or just lucky visitors? An A/B test answers honestly: show variant A to one random half of users, variant B to the other, and compare.
Two groups of users, split in half at random. Each has its own conversion — the share who completed the target action. The dot is the observed conversion; the segment around it is the confidence interval: where the true conversion plausibly falls given randomness.
A/B is needed exactly when the cost of an error exceeds the cost of the test. Shipping without a test is also a decision — just one that bets on intuition.
A competent test: the required sample size is computed before launch (so there is enough power to catch the effect), no hourly peeking at results, no stopping at the first pretty number.
Decisions are made from intervals and significance, not bare points — otherwise a random spike easily passes for a win.
A/B tests are the product standard: button color and copy, email subject, price, feed ranking. Everywhere variants are compared on live users, not on intuition.
In medicine these are randomized controlled trials — the gold standard of evidence. Agronomy compares field treatments the same way. One principle: split at random and compare honestly.
Definitions
A/B is honest under true random assignment, no overlap with other tests, and independent users. Network effects (social apps, marketplaces, delivery) break independence: groups influence each other, and the naive comparison lies.
You need the metric, sample size and duration fixed in advance. Peeking, stopping at the first "significance" and choosing the metric after the fact devalue the result.
Deep dive: the math and the mechanism (optional)
What A/B measures has a name: the average treatment effect, ATE — the average difference in outcome between "saw B" and "saw A" across all users. The difference of group means IS the ATE estimate, and randomization is what makes that estimate honest.
Why AVERAGE and not per-person: one user cannot see both A and B at once, so the individual effect is unobservable (the fundamental problem of causal inference). The effect within a segment is the CATE; more on it — and on how ATE hides heterogeneity — in the segments lesson.