Variance-reduction methods
A test's sensitivity falls with the data spread Ο. The most expensive fix is recruiting more users. There is a smarter one: strip the metric of its "extraneous" spread without touching the effect itself. These are variance-reduction methods β how product teams speed up A/B tests severalfold. Drag the correlation slider and watch the metric's distribution tighten.
CUPED: the better the pre-experiment covariate predicts the metric (higher Ο), the more variance is removed β down to (1βΟΒ²) β and the more sensitive the test at the same n.
The idea in plain terms. Part of a metric's spread is predictable in advance. An active user spent a lot before the experiment too; an inactive one β little. This "innate" between-people difference inflates the metric's variance yet has nothing to do with our effect. Subtract it, and a cleaner signal remains.
Little traffic β the default is "wait for a long test". CUPED and stratification change the equation: the same conclusion faster, or a smaller detectable effect in the same time.
In large products CUPED is on by default in the A/B platform: the covariate is the same metric over the 1β2 weeks before the test. It typically cuts the required sample or test duration by 30β50% β an enormous saving of time and traffic.
The method needs pre-experiment data and a stable audience. For new users (who have no "before") it does not work β there teams use other predictors or skip the adjustment.
Netflix, Microsoft, Booking and others have published how variance reduction accelerates their experiments: the same conclusions from less traffic, hence more tests per unit of time.
The same principle reaches beyond A/B: in any comparison, controlling for known factors (covariates) sharpens the estimate. It is the bridge to regression, whose core idea is "hold everything else equal".
Definitions
CUPED needs a covariate from the PRE-experiment period that correlates with the metric; for new users (no "before") it does not work. The covariate must not depend on the experiment itself β otherwise you may accidentally "subtract" the effect.
Variance reduction cuts random noise but does not cure bias: on unrepresentative data you get a confidently computed β yet still biased β result.
Deep dive: the math and the mechanism (optional)
The CUPED algorithm, step by step. 1) Take covariate X from the PRE-experiment period β usually the same metric over the 1β2 weeks before the test. 2) Across all users estimate ΞΈ = Cov(X, Y) / Var(X) (the regression coefficient of Y on X). 3) For every user compute the adjusted metric Y' = Y β ΞΈΒ·(X β XΜ), where XΜ is the covariate's overall mean. 4) Then run the ordinary test (a t-test on the difference of means) on Y' instead of Y. 5) Var(Y') is (1βΟΒ²) times Var(Y), where Ο = corr(X, Y); the effect estimate is not biased because X does not depend on the group.
The stratification algorithm, step by step. 1) Before the experiment pick a stratification variable correlated with the metric and independent of the test (country, platform, device type, pre-experiment activity β a continuous covariate gets cut into quantile bins). 2) WITHIN each stratum k compute the between-variant difference of means ΞΜβ and its variance. 3) Assemble the overall effect as the weighted average ΞΜ = Ξ£ wβΒ·ΞΜβ with weights wβ equal to stratum shares (post-stratification). 4) The variance of this estimate contains only WITHIN-stratum spread β the between-stratum part is gone. 5) A practical subtlety: don't overdo the number of strata (each needs enough data); over-stratifying is as harmful as ignoring it.