After ANOVA: which groups actually differ
ANOVA only says "there is a difference somewhere". But the business needs "WHICH group is better". The naive answer — run t-tests for all pairs — leads straight into the multiple comparisons trap. That is why special post-hoc procedures follow ANOVA.
Each row is a pair’s difference of means with an interval. Missed zero (green) → significant. Switch the Tukey correction off — the intervals narrow and more pairs turn "significant": that is the inflation of false findings under uncorrected pairwise tests.
ANOVA said "there is a difference somewhere". But between WHICH of the three groups? The chart shows confidence intervals of the pairwise differences (A−B, A−C, B−C). The rule is simple: if an interval does not cover zero (green) — the difference is significant; if it covers it (gray) — it is not. Move the between-group and within-group spreads — the intervals respond.
What it means
The proper multi-group analysis: ANOVA → if significant → a corrected post-hoc (Tukey). That way you neither drown in pairwise tests nor miss which group truly stands out.
The same in A/B/C/D tests: first check whether the variants differ at all, then corrected comparisons against the control. Otherwise, with many variants the "winner" is often just chance.
Where it shows up
Agronomy and medicine are the birthplace of post-hocs: compare several fertilizers or doses and figure out which one is actually better without harvesting false differences.
Product work: comparing several landing page versions or pricing grids — ANOVA/regression at the gate, then corrected pairwise comparisons to pick the variant.
Definitions
When the method lies (assumptions)
Post-hocs inherit ANOVA's assumptions (roughly normal residuals, similar group spread, independence). When violated — nonparametric analogues (Kruskal–Wallis + Dunn).
Controlling the family-wise error reduces the power of each individual comparison: the honest price for the right to test many pairs at once.