Confidence intervals

A single sample mean is a point estimate, and it is almost surely slightly off. It is more honest to state not a point but a range in which the true value plausibly lies. That is a confidence interval. But what exactly does "95%" mean?

▬ blue — interval covered the truth: 0▬ red — MISSED it: 0covered —% (expected ≈ 95%)

CI = x̄ ± z·SE = x̄ ± 1.96·(σ/√n)

for a share (A/B): p̂ ± 1.96·√( p̂(1−p̂)/n )

z depends on the confidence level: 90% → 1.645, 95% → 1.96, 99% → 2.576. Higher level — wider interval.

Level:

The white dashed line is the true value (unknown to us in real life). Take a sample: it yields a mean and an interval around it. Press "take a sample" a few times — each segment is the interval from one sample.

you can flip beats with ← → keys

What it means

That is why reports say not "conversion 12%" but "12% ± 2%": the interval honestly shows how confident we are. Two estimates with overlapping intervals are too early to tell apart — we'll see this in the A/B test lesson.

And remember: a wide interval is not an error but a signal that "data is scarce". Want it narrower — collect more observations.

Where it shows up

Confidence intervals stand behind the "margin of error" in poll coverage ("45% ± 3%"), behind forecast ranges and error bars on dashboard charts.

It is the same thing everywhere: don't pretend the estimate is exact — show its uncertainty honestly.

Definitions

Point estimate

a single number — e.g. the sample mean. Almost always slightly off the truth.

Confidence interval

a range around the estimate built so that, over repeated experiments, a set share of intervals (e.g. 95%) covers the true value.

Confidence level

that very share (95%, 99%): how often the method produces a covering interval over the long run.

When the method lies (assumptions)

The formula CI = x̄ ± z·SE relies on the normality of the sampling distribution (the CLT) and a sample-based estimate of σ; for small n use the t-distribution instead of z (slightly wider). For a share you need enough successes and failures.

The interval captures only RANDOM error. Systematic bias (an unrepresentative sample) is invisible to it — you can get a narrow but wrong interval around a biased estimate.