Segments and heterogeneous effects (ATE and CATE)

An A/B test of a new menu item gives the verdict: "B beats A by 2%". But the average result can hide that newcomers loved the novelty while regulars were put off by it. An effect is rarely the same for everyone. Let's unpack what exactly an experiment measures and why you should look at segments.

● z = 0● z = 1pooled slope (no control): 0.78

True effect of x on y0.20

Confounder z strength1.0

Set the true effect to 0 and crank up the confounder strength: the pooled red line will show a "relationship" that isn’t there. Controlling for z (within-group lines) removes it. That is what multiple regression does — estimates the effect of x while holding z constant.

What an A/B test estimates is called the ATE — the Average Treatment Effect: how much better B is than A on average across all users. It is the honest headline number of the experiment. But "on average" means averaging over very different people.

you can flip beats with ← → keys

What it means

What decision this changes

The average effect answers "roll out to everyone?"; the CATE answers "roll out to whom?": a targeted rollout can save a feature that failed the overall test.

A competent A/B readout does not stop at the ATE: analysts check the CATE across pre-specified segments so as not to ship a change that is positive on average but hurts an important group. Sometimes the best decision is to roll out only to the segment where the effect is positive.

But segment analysis is a multiple-comparisons minefield: dozens of cuts will almost surely produce a random "significant" segment. So segments are hypothesized in advance, the threshold is corrected, and findings are re-verified.

Where it shows up

A new feature often delights newcomers and annoys experienced users (it breaks a habit). The ATE can be near zero while the CATE by tenure differs sharply; the decision is made on the segments.

Price and promo changes almost always have heterogeneous effects across countries and platforms — which is why rollouts are often done by segment rather than "to everyone at once".

Definitions

ATE (average effect)

Average Treatment Effect — the average difference in outcomes between variants across everyone; this is what an A/B test estimates.

In plain words: the effect averaged over everyone.

CATE (conditional effect)

Conditional ATE — the effect within a segment (conditional on attributes). Reveals heterogeneity.

In plain words: the effect for a specific segment.

Individual effect (ITE)

the effect for a single unit; unobservable — you can never see both outcomes at once (the fundamental problem of causal inference).

In plain words: the effect for one person — never directly observable.

Effect heterogeneity

the effect differs across segments; the average (ATE) can hide segments with the opposite sign.

In plain words: the average can hide it: some were helped, others were hurt.

When the method lies (assumptions)

The ATE estimate is honest under true randomization and independence; a segment CATE inherits the same requirements plus a sufficient size for each segment (in small slices the estimate is very noisy).

Segments found by combing the data after the fact are almost certainly false. Without pre-specified hypotheses and a multiplicity correction, segment analysis breeds phantom effects.