Two-way ANOVA and interactions
One-way ANOVA compares groups by ONE factor. But in life factors come several at a time: the plan and the channel; the drug and the age. Two-way ANOVA takes them apart together β and catches the most interesting thing: the interaction, when the effect of one factor depends on the other.
Set the interaction to 0 β the lines become parallel: the testβs effect is the same on mobile and desktop. Increase it β the lines diverge and may cross: on mobile the test helps, on desktop it hurts. A significant interaction = the "average effect" lies; look by segment.
The idea is the same β compare the spread explained by the factors with the noise spread inside. But now there are two factors: A β the button design (control/test, the X axis), B β the platform (two lines: mobile and desktop). ANOVA decomposes the spread into A's contribution, B's contribution, their interaction and the residual noise β with an F for each. On the chart you can see it by the slope and the parallelism of the lines.
With several factors, don't run a pile of separate tests β two-way ANOVA (or a regression with an interaction) takes them apart jointly and honestly catches where the effect depends on the conditions.
A significant interaction is a stop sign for the phrase "helps on average": from there the decision is made by subgroup, not by the overall mean. A direct bridge to segment analysis in A/B.
Agronomy (ANOVA's birthplace): the yield depends on the cultivar, the fertilizer, and their combination β the interaction shows which cultivar suits which fertilizer.
Product and medicine: a feature's/drug's effect often depends on the segment/patient. The interaction IS the "who exactly it suits" β the most important part of the conclusion.
Definitions
Same as one-way: roughly normal residuals, similar spread in the cells, independence. Plus you need enough observations in EVERY cell (combination of levels), otherwise the interaction is estimated very noisily.
Many factors and levels β many comparisons β the multiplicity risk again; a significant interaction is best confirmed on new data.