Sequential tests: how to peek honestly

In the previous lesson we saw that peeking at a regular A/B and stopping at the first "significant" inflates false wins. But the business wants to stop tests earlier. There is an honest way — sequential methods, which price in the cost of repeated checks up front.

Stopping rule:

Data:

The share of A/A tests with a false stop (the rule fired at least once, though there is no effect). The honest target is ≤5%. Run the "no effect" scenario with the "+50" button.

naive 0.05: —α-spending: —evidence ×20 (SPRT): —(A/A tests: 0)

Same day-by-day view as in the peeking lesson, but the stopping rule is now honest. Naive flat 0.05 is checked every day → under A/A the p-value keeps dipping under it (red false stops), far above 5%. The α-spending boundary is strict early and relaxes toward the planned day → false stops stay near 5% (the group-sequential family: Pocock, O’Brien–Fleming). Switch to "evidence counter (SPRT)" to see the other family.

The main idea — this is the same day-by-day p-value chart as in the peeking lesson, except the stopping boundary is now honest. Choose "naive 0.05 threshold", the "no effect (A/A)" scenario, and run "+50 tests": the lines keep diving under the flat threshold — the share of false stops is far beyond 5% (that is the peeking from the previous lesson). Now switch to "sequential boundary (α-spending)": under the same A/A, the share returns to an honest ~5%, because the threshold is strict early in the test and relaxes only toward the planned day. That is the trick: the cost of repeated checks is budgeted in advance. Pocock and O'Brien–Fleming are specific families of such boundaries.

you can flip beats with ← → keys

What it means

What decision this changes

If you want to watch mid-test, choose a sequential design before the start. Changing the stopping rules mid-test is not allowed.

If the team needs to monitor a test and stop early, that is not a reason to break the statistics — it is a reason to switch on the sequential mode in the A/B platform. Then an early stop is honest, not self-deception.

Without such methods the rule is simple and hard: fix the sample size and the duration in advance, decide once at the end. Any "let's switch it off, it's already significant on day three" in a fixed-horizon test inflates false wins.

Where it shows up

Clinical trials have used group sequential designs for decades: interim analyses make it ethical to stop a trial when a drug clearly works or clearly harms.

Modern product A/B platforms (including at major tech companies) offer always-valid statistics precisely so product managers can peek at the dashboard honestly.

Definitions

Sequential test

a method that allows repeated in-flight checks while controlling the overall Type I error.

In plain words: a design in which interim looks are permitted in advance.

Likelihood ratio (SPRT)

the accumulated "evidence counter": how many times more likely the data is when the effect is present than when it is absent. With no effect it does not grow on average (a fair game), so it rarely reaches high values.

In plain words: an evidence counter for the effect; the "stop at 20×" rule holds the error at 5% for any number of looks.

Group sequential

planned interim cuts with tightened thresholds (O'Brien–Fleming, Pocock) and an early-stop option.

In plain words: several planned stopping points, each with its own error budget.

Always-valid p-value

a p-value/interval you may look at any moment, any number of times, without inflating the error.

In plain words: a p-value you can trust whenever you look.

Error budget (alpha spending)

distributing the allowed Type I error across several checks so the total stays at 5%.

In plain words: 5% error is a budget: spend a piece on each check.

When the method lies (assumptions)

The speed gain is not free: in the worst case sequential methods require no smaller — and sometimes a larger — maximum sample; the saving is on average, not guaranteed.

You must use a method built for the purpose. Taking a regular fixed-horizon p-value and checking it daily is not a "sequential test" — it is exactly the peeking.