Hypothesis testing: p-value and the two errors
You ran an experiment and saw a difference. Is it real — or just sampling luck? Hypothesis testing answers that question. And every answer carries two kinds of errors.
Both bells are sampling distributions of the estimate (the difference of means), and they are motionless: gray H0 is always centered at 0, blue H1 at the true effect (+60 ₽). Only the threshold moves: the critical value depends solely on H0 and α (crit = z·SE, SE = σ/√n), not on the effect. Changing α moves the threshold and shows the balance of the two errors: the yellow area is α (Type I), the red one β (a missed effect). n and σ only change the bells\u2019 width. How many observations a given effect needs — next lesson, on sample size.
The metric is the average receipt, 1000 ₽; the true effect is fixed (+60 ₽), the bells never move. α sets the critical value (higher α — the threshold moves toward zero, α grows and β falls — there is the two-error balance). Power (1−β) is raised NOT by the threshold but by sample size n or smaller σ — they narrow the bells without moving their centers; the button finds n for 80% power. A two-sided test catches deviation either way, so at the same α its threshold sits farther from zero.
Two curves — two possible worlds, and each is that same sampling distribution from the estimation lesson, only for the DIFFERENCE of means. The gray one (H0) stands at zero — no effect. The blue one (H1) — at the true effect (fixed here for clarity). Both bells are MOTIONLESS. Only the decision boundary moves — the critical value: it depends solely on H0 and α, NOT on the effect. So as you change α you move the threshold and watch the two errors flow into each other: the yellow area is α (Type I error), the red one is β (a missed effect). The n and σ sliders only change the bells' width without moving their centers. Play with α to feel the balance. If the formula crit = z·SE hasn't clicked yet, that's fine — this is the densest spot in the lesson: the formula itself is always computed by the software, but the trade-off between the two errors is worth carrying with you for good. (How the required sample size is computed from the effect — next lesson; the p-value and "our result" — on a separate chart further on.)
Before the test, fix the default action ("don't ship") and the threshold α. The test changes the decision in one direction only: p-value below the threshold — ship. In every other case we stay with the default — which is not "we proved there is no effect".
Picture a drug trial. H0 — "does not help", H1 — "helps". A false alarm: approving a placebo. A miss: rejecting a working drug. The costs differ, so the threshold (significance level) is chosen deliberately, not "5% by habit".
And remember the law of large numbers: on a tiny sample any "difference" may be noise. A sufficient sample size is part of an honest test.
The same two errors are everywhere: a medical test (declaring a healthy person sick, or a sick one healthy), a spam filter (a real letter in spam, or spam in the inbox), a court (convicting the innocent or releasing the guilty).
Nowhere can both errors be zeroed at once — you can only choose the balance according to each error's price.
The most famous threshold in history is particle physics' "five sigma": CERN announced the Higgs boson discovery only when the chance of such a signal arising randomly fell to roughly one in 3.5 million (p ≈ 3·10⁻⁷). Same logic as your A/B — just a different price for a false discovery.
Definitions
A p-value is honest for one pre-specified question. Many metrics, segments, or stopping the test "once it looks significant" turn 5% false alarms into tens of percent — see the lessons on multiple comparisons and peeking.
A small p does not mean a large or important effect: on a huge sample even +0.01% becomes significant. Look at the effect size and the confidence interval, not only the 0.05 threshold.
Deep dive: the math and the mechanism (optional)
Важная эпистемическая тонкость. На графике бирюзовая кривая H1 нарисована как известная — но в реальности истинный эффект нам НЕизвестен (иначе зачем тест). Мы фиксируем её положение как гипотезу — тот размер эффекта, который хотим уметь замечать (в планировании его называют MDE); под него и считают мощность и размер выборки в следующем уроке. Мы всегда живём только в одном мире (эффект либо есть, либо нет) и никогда не знаем, в каком, — мы лишь ограничиваем шансы ошибиться в каждую сторону.
Почему нельзя «доказать H1 напрямую». Мы не можем перечислить все способы, которыми эффект мог бы существовать, но можем точно посчитать, как ведёт себя мир БЕЗ эффекта (H0). Поэтому логика всегда от противного: показать, что наблюдаемое слишком невероятно для мира H0. Это асимметрия — как в суде: отвергаем презумпцию «невиновен» при сильных уликах, но «не отвергли» не значит «доказали невиновность».