Hypothesis testing: p-value and the two errors
You ran an experiment and saw a difference. Is it real — or just sampling luck? Hypothesis testing answers that question. And every answer carries two kinds of errors.
Both bells are sampling distributions of the estimate (the difference of means), and they are motionless: gray H0 is always centered at 0, blue H1 at the true effect (+60 ₽). Only the threshold moves: the critical value depends solely on H0 and α (crit = z·SE, SE = σ/√n), not on the effect. Changing α moves the threshold and shows the balance of the two errors: the yellow area is α (Type I), the red one β (a missed effect). n and σ only change the bells\u2019 width. How many observations a given effect needs — next lesson, on sample size.
The metric is the average order value, 1000 ₽; the true effect is fixed (+60 ₽), the bells never move. α sets the critical value (higher α — the threshold moves toward zero, α grows and β falls — there is the two-error balance). Power (1−β) is raised NOT by the threshold but by sample size n or smaller σ — they narrow the bells without moving their centers; the button finds n for 80% power. A two-sided test catches deviation either way, so at the same α its threshold sits farther from zero.
Two curves — two possible worlds, and each is that same sampling distribution from the estimation lesson, only for the DIFFERENCE of means. The gray one (H0) stands at zero — no effect. The blue one (H1) — at the true effect (fixed here for clarity). Both bells are MOTIONLESS. Only the decision boundary moves — the critical value: it depends solely on H0 and α, NOT on the effect. So as you change α you move the threshold and watch the two errors flow into each other: the yellow area is α (Type I error), the red one is β (a missed effect). The n and σ sliders only change the bells' width without moving their centers. Play with α to feel the balance. If the formula crit = z·SE hasn't clicked yet, that's fine — this is the densest spot in the lesson: the formula itself is always computed by the software, but the trade-off between the two errors is worth carrying with you for good. (How the required sample size is computed from the effect — next lesson; the p-value and "our result" — on a separate chart further on.)
What it means
What decision this changes
Before the test, fix the default action ("don't ship") and the threshold α. The test changes the decision in one direction only: p-value below the threshold — ship. In every other case we stay with the default — which is not "we proved there is no effect".
Picture a drug trial. H0 — "does not help", H1 — "helps". A false positive: approving a placebo. A miss: rejecting a working drug. The costs differ, so the threshold (significance level) is chosen deliberately, not "5% by habit".
And remember the law of large numbers: on a tiny sample any "difference" may be noise. A sufficient sample size is part of an honest test.
Where it shows up
The same two errors are everywhere: a medical test (declaring a healthy person sick, or a sick one healthy), a spam filter (a real letter in spam, or spam in the inbox), a court (convicting the innocent or releasing the guilty).
Nowhere can both errors be zeroed at once — you can only choose the balance according to each error's price.
The most famous threshold in history is particle physics' "five sigma": CERN announced the Higgs boson discovery only when the chance of such a signal arising randomly fell to roughly one in 3.5 million (p ≈ 3·10⁻⁷). Same logic as your A/B — just a different price for a false discovery.
Definitions
When the method lies (assumptions)
A p-value is honest for one pre-specified question. Many metrics, segments, or stopping the test "once it looks significant" turn 5% false positives into tens of percent — see the lessons on multiple comparisons and peeking.
A small p does not mean a large or important effect: on a huge sample even +0.01% becomes significant. Look at the effect size and the confidence interval, not only the 0.05 threshold.
Deep dive: the math and the mechanism (optional)
An important epistemic subtlety. On the chart the teal H1 curve is drawn as if it were known — but in reality the true effect is NOT known to us (otherwise why run the test). We fix its position as a hypothesis: the effect size we want to be able to detect (in planning it is called the MDE); power and sample size in the next lesson are computed for it. We always live in exactly one world (the effect either exists or it doesn't) and never know which one — all we do is limit the chances of erring in each direction.
Why you cannot "prove H1 directly". We cannot enumerate every way the effect might exist, but we can compute exactly how a world WITHOUT the effect (H0) behaves. So the logic is always by contradiction: show that what we observed is too improbable for the H0 world. This is an asymmetry — as in court: we reject the presumption of innocence given strong evidence, but "not rejected" does not mean "innocence proven".