Analysis of variance (ANOVA): comparing many groups

The t-test and z-test compare TWO groups. But A/B tests often have several variants (A/B/C/D), and life has many categories: conversion by country, revenue by plan. Comparing them pairwise is dangerous, and that is what analysis of variance — ANOVA — is for. Move the sliders and watch the ratio of spreads, F.

spread BETWEEN groups (SS_b): 1817spread WITHIN (SS_w): 789F = 31.10F > critical → the groups differ

Spread between group means8

Within-group spread σ6

F = (spread between groups) / (spread within). Pull the means apart — F grows; add noise inside — F drops. That is how ANOVA answers "is there any difference at all" for several groups at once, without breeding pairwise comparisons.

Why can't you just run a t-test for every pair? That is the multiple comparisons trap: with 4 groups there are already 6 pairs, and the chance of randomly catching "significance" rises sharply. ANOVA asks one overall question — "is there any difference between any of the groups at all?" — with a single test, without inflating the error.

you can flip beats with ← → keys

What it means

ANOVA is the right tool for A/B/C/D tests and for comparing a metric across many segments. First one test — "is there any difference at all" — then corrected pairwise comparisons. It disciplines the analysis and guards against the false discoveries from the multiple comparisons lesson.

If the data within groups is heavily skewed or has outliers, ANOVA (like the t-test) has nonparametric analogues (Kruskal–Wallis) — the same idea on ranks. As always, the data dictates the choice.

Where it shows up

Agronomy, where ANOVA was born: compare crop yields under several fertilizers at once. Today — comparing several interface versions, pricing plans, regions, models.

In analytics, ANOVA and its regression form appear wherever you need to know whether a categorical factor (group, channel, segment) explains differences in a metric — and by how much.

Definitions

ANOVA

a test for differences in means across several groups at once with a single criterion, without pairwise sweeps.

Between-group spread

how far the group means deviate from the overall mean — the "signal".

Within-group spread

how much the observations wobble inside their own group — the "noise".

F statisticF = spread between / spread within

large F → the differences between groups override the internal noise.

When the method lies (assumptions)

Classical ANOVA assumes roughly normal residuals, similar spread across groups (homoscedasticity) and independent observations. With heavy tails or outliers, take the rank analogue — the Kruskal–Wallis test.

ANOVA answers "is there a difference anywhere", not "between which groups exactly". For that you need post-hoc comparisons (e.g. Tukey's) with a multiplicity correction.