Confidence interval versus p-value
The p-value answers "is the difference chance?", while the confidence interval answers "how big is it and how sure are we?". They are two sides of one coin: estimation and testing. Understand their link β and you stop worshipping "p < 0.05" and start reading results like a grown-up analyst.
The interval excludes zero βΊ the test is significant (p<0.05) β one and the same statement. But the interval says more: at huge n even a tiny effect is "significant", yet the interval shows it is small; at small n the interval is wide β "data is scarce", not "there is no effect".
There is an exact connection between them. The chart shows the effect estimate with its confidence interval relative to zero. Drag the effect and the sample size: at exactly the moment the interval stops covering zero, p drops below 0.05. "p < 0.05" and "the interval excludes zero" are the same statement said two ways.
If the decision depends on "how much to invest", decide by the interval, not by "significant/not": the interval shows both the best and the worst plausible outcome.
The habit of reporting results as intervals, not just p, immediately raises decision quality: you see direction, size and confidence. "A 1.8% lift (CI 0.3β3.3%)" is grounds for a decision; "p = 0.04" is not.
It also dissolves the cult of 0.05: instead of "significant / not significant" the team discusses what effect is plausible and whether it is worth shipping. Especially vital on large samples, where almost everything becomes significant.
Medicine long ago switched to confidence intervals for effect sizes: "a 20% risk reduction (95% CI: 5β33%)" is more informative than "p < 0.05".
In a product, the interval around an A/B effect feeds the rollout decision directly: if even the interval's lower bound pays for the costs β ship; if the interval is wide and touches zero β collect more data.