Variance-reduction methods
A test's sensitivity falls with the data spread ฯ. The most expensive fix is recruiting more users. There is a smarter one: strip the metric of its "extraneous" spread without touching the effect itself. These are variance-reduction methods โ how product teams speed up A/B tests severalfold. Drag the correlation slider and watch the metric's distribution tighten.
CUPED: the better the pre-experiment covariate predicts the metric (higher ฯ), the more variance is removed โ down to (1โฯยฒ) โ and the more sensitive the test at the same n.
The idea in plain terms. Part of a metric's spread is predictable in advance. An active user spent a lot before the experiment too; an inactive one โ little. This "innate" between-people difference inflates the metric's variance yet has nothing to do with our effect. Subtract it, and a cleaner signal remains.
What it means
What decision this changes
Little traffic โ the default is "wait for a long test". CUPED and stratification change the equation: the same conclusion faster, or a smaller detectable effect in the same time.
In large products CUPED is on by default in the A/B platform: the covariate is the same metric over the 1โ2 weeks before the test. It typically cuts the required sample or test duration by 30โ50% โ an enormous saving of time and traffic.
The method needs pre-experiment data and a stable audience. For new users (who have no "before") it does not work โ there teams use other predictors or skip the adjustment.
Where it shows up
Netflix, Microsoft, Booking and others have published how variance reduction accelerates their experiments: the same conclusions from less traffic, hence more tests per unit of time.
The same principle reaches beyond A/B: in any comparison, controlling for known factors (covariates) sharpens the estimate. It is the bridge to regression, whose core idea is "hold everything else equal".
Definitions
When the method lies (assumptions)
CUPED needs a covariate from the PRE-experiment period that correlates with the metric; for new users (no "before") it does not work. The covariate must not depend on the experiment itself โ otherwise you may accidentally "subtract" the effect.
Variance reduction cuts random noise but does not cure bias: on unrepresentative data you get a confidently computed โ yet still biased โ result.
Deep dive: the math and the mechanism (optional)
The CUPED algorithm, step by step. 1) Take covariate X from the PRE-experiment period โ usually the same metric over the 1โ2 weeks before the test. 2) Across all users estimate ฮธ = Cov(X, Y) / Var(X) (the regression coefficient of Y on X). 3) For every user compute the adjusted metric Y' = Y โ ฮธยท(X โ Xฬ), where Xฬ is the covariate's overall mean. 4) Then run the ordinary test (a t-test on the difference of means) on Y' instead of Y. 5) Var(Y') is (1โฯยฒ) times Var(Y), where ฯ = corr(X, Y); the effect estimate is not biased because X does not depend on the group.
The stratification algorithm, step by step. 1) Before the experiment pick a stratification variable correlated with the metric and independent of the test (country, platform, device type, pre-experiment activity โ a continuous covariate gets cut into quantile bins). 2) WITHIN each stratum k compute the between-variant difference of means ฮฬโ and its variance. 3) Assemble the overall effect as the weighted average ฮฬ = ฮฃ wโยทฮฬโ with weights wโ equal to stratum shares (post-stratification). 4) The variance of this estimate contains only WITHIN-stratum spread โ the between-stratum part is gone. 5) A practical subtlety: don't overdo the number of strata (each needs enough data); over-stratifying is as harmful as ignoring it.