Multiple comparisons in A/B
We have learned to measure the effect overall (ATE) and by segment (CATE). But as soon as an experiment has many variants, metrics or segments — the pizzeria dashboard's twenty metrics, ten cities — a new trap appears: testing many hypotheses at once makes it easy to catch random "significance". Let's see why, and how to handle it.
Without a correction, "significant" passes for both real effects and ~5% of the empty ones (red). Bonferroni (FWER) lowers the threshold to α/m — the false ones nearly vanish, but real findings die with them (lots of orange "missed"). Benjamini–Hochberg (FDR) bounds not "zero errors" but the share of false among the found: the threshold is the point where the i·α/m staircase stops covering the sorted p-values — it usually catches more real effects at a controlled share of false ones.
Each square is a separate hypothesis test. There is no real effect in any of them (these are deliberately empty tests). The significance threshold is 5%. Press "run the tests".
What it means
What decision this changes
Before a batch of tests, decide what is worse: missing a finding (then control FDR) or catching a false one (then FWER/Bonferroni). The threshold is chosen before seeing the data.
A dashboard with 50 metrics: some metric is almost always "significantly" different between segments. Don't rush to celebrate — that is expected noise. Analysts either correct the threshold or re-check the finding on new data.
This is a direct continuation of the peeking lesson: both repeated checks over time and checking many metrics at once inflate false wins.
Where it shows up
In genetics, thousands of genes are compared against a disease — without correction, false associations are guaranteed to surface, so very strict thresholds are used there.
In analytics this is "data diving": combing through cuts and metrics until a pretty result pops out for the slide deck. Pretty — but unreproducible.
The classic of the genre is the "dead salmon": researchers put a dead fish in an MRI scanner, ran the standard analysis over thousands of voxels without corrections — and "found" brain activity. The work won an Ig Nobel Prize and became an eternal monument to multiple comparisons.
Definitions
When the method lies (assumptions)
Bonferroni is built for the worst case — independent tests. When metrics are strongly correlated (revenue and average order value), the correction becomes overly strict and cuts power; then FDR control procedures (Benjamini–Hochberg) are more honest.
Corrections only cure the declared comparisons. A "significant" segment found by combing the data cannot be saved by a correction over the list — such a result is re-tested on fresh data.
Deep dive: the math and the mechanism (optional)
The Bonferroni algorithm (FWER control). 1) Set the overall acceptable risk α (usually 0.05). 2) Divide it by the number of tests: threshold = α / m. 3) Reject H0 in the tests where p < α/m. 4) Then the probability of at least one false discovery in the whole family is ≤ α. Pros — simplicity and a hard guarantee; cons — with large m the threshold is tiny and power (the chance of catching real effects) drops sharply. A relative is Holm's method: the same idea, but stepwise and slightly more powerful.
The Benjamini–Hochberg algorithm (FDR control). 1) Collect all m p-values and sort them in ascending order: p(1) ≤ p(2) ≤ … ≤ p(m). 2) Set the acceptable share of false hits q (say, 0.05). 3) Going from the end, find the largest rank k with p(k) ≤ (k/m)·q. 4) Reject H0 for all hypotheses with ranks 1…k (including those whose p exceeds their individual bound but sit "under" the found k). 5) Then the expected share of false hits among the rejected is ≤ q. In practice: BH is the standard wherever hypotheses are many (genes, features, dozens of metrics and segments), because it finds noticeably more real effects than Bonferroni at the price of a small controlled share of errors.