Variance-reduction methods
A test's sensitivity falls with the data spread σ. The most expensive fix is recruiting more users. There is a smarter one: strip the metric of its "extraneous" spread without touching the effect itself. These are variance-reduction methods — how product teams speed up A/B tests severalfold. Drag the correlation slider and watch the metric's distribution tighten.
CUPED: the better the pre-experiment covariate predicts the metric (higher ρ), the more variance is removed — down to (1−ρ²) — and the more sensitive the test at the same n.
The idea in plain terms. Part of a metric's spread is predictable in advance. An active user spent a lot before the experiment too; an inactive one — little. This "innate" between-people difference inflates the metric's variance yet has nothing to do with our effect. Subtract it, and a cleaner signal remains.
Little traffic — the default is "wait for a long test". CUPED and stratification change the equation: the same conclusion faster, or a smaller detectable effect in the same time.
In large products CUPED is on by default in the A/B platform: the covariate is the same metric over the 1–2 weeks before the test. It typically cuts the required sample or test duration by 30–50% — an enormous saving of time and traffic.
The method needs pre-experiment data and a stable audience. For new users (who have no "before") it does not work — there teams use other predictors or skip the adjustment.
Netflix, Microsoft, Booking and others have published how variance reduction accelerates their experiments: the same conclusions from less traffic, hence more tests per unit of time.
The same principle reaches beyond A/B: in any comparison, controlling for known factors (covariates) sharpens the estimate. It is the bridge to regression, whose core idea is "hold everything else equal".
Definitions
CUPED needs a covariate from the PRE-experiment period that correlates with the metric; for new users (no "before") it does not work. The covariate must not depend on the experiment itself — otherwise you may accidentally "subtract" the effect.
Variance reduction cuts random noise but does not cure bias: on unrepresentative data you get a confidently computed — yet still biased — result.
Deep dive: the math and the mechanism (optional)
The CUPED algorithm, step by step. 1) Take covariate X from the PRE-experiment period — usually the same metric over the 1–2 weeks before the test. 2) Across all users estimate θ = Cov(X, Y) / Var(X) (the regression coefficient of Y on X). 3) For every user compute the adjusted metric Y' = Y − θ·(X − X̄), where X̄ is the covariate's overall mean. 4) Then run the ordinary test (a t-test on the difference of means) on Y' instead of Y. 5) Var(Y') is (1−ρ²) times Var(Y), where ρ = corr(X, Y); the effect estimate is not biased because X does not depend on the group.
The stratification algorithm, step by step. 1) Before the experiment pick a stratification variable correlated with the metric and independent of the test (country, platform, device type, pre-experiment activity — a continuous covariate gets cut into quantile bins). 2) WITHIN each stratum k compute the between-variant difference of means Δ̂ₖ and its variance. 3) Assemble the overall effect as the weighted average Δ̂ = Σ wₖ·Δ̂ₖ with weights wₖ equal to stratum shares (post-stratification). 4) The variance of this estimate contains only WITHIN-stratum spread — the between-stratum part is gone. 5) A practical subtlety: don't overdo the number of strata (each needs enough data); over-stratifying is as harmful as ignoring it.