Multiple comparisons in A/B
We have learned to measure the effect overall (ATE) and by segment (CATE). But as soon as an experiment has many variants, metrics or segments — the pizzeria dashboard's twenty metrics, ten cities — a new trap appears: testing many hypotheses at once makes it easy to catch random "significance". Let's see why, and how to handle it.
Without a correction, "significant" passes for both real effects and ~5% of the empty ones (red). Bonferroni (FWER) lowers the threshold to α/m — the false ones nearly vanish, but real findings die with them (lots of orange "missed"). Benjamini–Hochberg (FDR) bounds not "zero errors" but the share of false among the found: the threshold is the point where the i·α/m staircase stops covering the sorted p-values — it usually catches more real effects at a controlled share of false ones.
Each square is a separate hypothesis test. There is no real effect in any of them (these are deliberately empty tests). The significance threshold is 5%. Press "run the tests".
Before a batch of tests, decide what is worse: missing a finding (then control FDR) or catching a false one (then FWER/Bonferroni). The threshold is chosen before seeing the data.
A dashboard with 50 metrics: some metric is almost always "significantly" different between segments. Don't rush to celebrate — that is expected noise. Analysts either correct the threshold or re-check the finding on new data.
This is a direct continuation of the peeking lesson: both repeated checks over time and checking many metrics at once inflate false wins.
In genetics, thousands of genes are compared against a disease — without correction, false associations are guaranteed to surface, so very strict thresholds are used there.
In analytics this is "data diving": combing through cuts and metrics until a pretty result pops out for the slide deck. Pretty — but unreproducible.
The classic of the genre is the "dead salmon": researchers put a dead fish in an MRI scanner, ran the standard analysis over thousands of voxels without corrections — and "found" brain activity. The work won an Ig Nobel Prize and became an eternal monument to multiple comparisons.
Definitions
Bonferroni is built for the worst case — independent tests. When metrics are strongly correlated (revenue and average order value), the correction becomes overly strict and cuts power; then FDR control procedures (Benjamini–Hochberg) are more honest.
Corrections only cure the declared comparisons. A "significant" segment found by combing the data cannot be saved by a correction over the list — such a result is re-tested on fresh data.
Deep dive: the math and the mechanism (optional)
The Bonferroni algorithm (FWER control). 1) Set the overall acceptable risk α (usually 0.05). 2) Divide it by the number of tests: threshold = α / m. 3) Reject H0 in the tests where p < α/m. 4) Then the probability of at least one false discovery in the whole family is ≤ α. Pros — simplicity and a hard guarantee; cons — with large m the threshold is tiny and power (the chance of catching real effects) drops sharply. A relative is Holm's method: the same idea, but stepwise and slightly more powerful.
The Benjamini–Hochberg algorithm (FDR control). 1) Collect all m p-values and sort them in ascending order: p(1) ≤ p(2) ≤ … ≤ p(m). 2) Set the acceptable share of false hits q (say, 0.05). 3) Going from the end, find the largest rank k with p(k) ≤ (k/m)·q. 4) Reject H0 for all hypotheses with ranks 1…k (including those whose p exceeds their individual bound but sit "under" the found k). 5) Then the expected share of false hits among the rejected is ≤ q. In practice: BH is the standard wherever hypotheses are many (genes, features, dozens of metrics and segments), because it finds noticeably more real effects than Bonferroni at the price of a small controlled share of errors.