Sample size and power
The previous lesson showed: more data, fewer errors. But how much exactly? You do not guess this — you compute it in advance. Otherwise you risk running an experiment physically incapable of noticing the effect.
The metric is the average receipt, 1000 ₽. We control not the mean difference directly (the business goal sets it) but the MDE — the minimum % lift we want to catch reliably; it sets the center of the blue H1 (+30 ₽). The top chart shows two distributions of the estimate: gray under "no effect" (H0), blue under "the effect is real" (H1). The threshold splits the axis into decisions: red right of the threshold under H0 is α (noise taken for effect), yellow left of it under H1 is β (a real effect overlooked). Smaller σ or bigger n narrow the bells → β falls, power grows. The bottom chart folds this into the power-vs-n curve: n is chosen to fit the MDE, not the other way around.
First, where power comes from at all. The top chart shows two distributions of our estimate of the between-group difference. The gray bell on the left — the world with NO effect (H0): the difference is zero on average, but the estimate wanders around zero due to chance. The blue bell on the right — the world with a real effect (H1): the estimate wanders around the true difference. Horizontal axis — what difference we might measure.
Sample size is a decision made before launch. Starting an underpowered test means agreeing in advance to miss the effect and waste the traffic.
Before launching an A/B, product analysts compute the sample size: "to notice a 1-percentage-point conversion lift with 80% power we need this many users per variant". Without that calculation the test easily turns out meaningless.
And conversely: on a huge sample even a negligible, business-useless difference becomes statistically significant. So you check not only significance but the effect size.
Clinical trials size their patient counts exactly this way — undersampling means a working drug may be rejected merely for lack of data.
Sociologists size polls for the required precision; engineers size the number of load-test runs. Everywhere the data budget is computed first and collected second.
Definitions
A sample-size calculation is only as honest as the expected-effect estimate behind it. That estimate is usually optimistic; the real effect is smaller — and the "properly sized" test ends up underpowered.
The metric's variance is taken from history, and it drifts: season, traffic mix, promotions. Build in a margin and recompute on fresh data — last quarter's σ can betray you.