Sequential tests: how to peek honestly

In the previous lesson we saw that peeking at a regular A/B and stopping at the first "significant" inflates false wins. But the business wants to stop tests earlier. There is an honest way — sequential methods, which price in the cost of repeated checks up front.

Stopping rule:

Data:

The share of A/A tests with a false stop (the rule fired at least once, though there is no effect). The honest target is ≤5%. Run the "no effect" scenario with the "+50" button.

naive 0.05: —α-spending: —evidence ×20 (SPRT): —(A/A tests: 0)

Same day-by-day view as in the peeking lesson, but the stopping rule is now honest. Naive flat 0.05 is checked every day → under A/A the p-value keeps dipping under it (red false stops), far above 5%. The α-spending boundary is strict early and relaxes toward the planned day → false stops stay near 5% (the group-sequential family: Pocock, O’Brien–Fleming). Switch to "evidence counter (SPRT)" to see the other family.

The main idea — this is the same day-by-day p-value chart as in the peeking lesson, except the stopping boundary is now honest. Choose "naive 0.05 threshold", the "no effect (A/A)" scenario, and run "+50 tests": the lines keep diving under the flat threshold — the share of false stops is far beyond 5% (that is the peeking from the previous lesson). Now switch to "sequential boundary (α-spending)": under the same A/A, the share returns to an honest ~5%, because the threshold is strict early in the test and relaxes only toward the planned day. That is the trick: the cost of repeated checks is budgeted in advance. Pocock and O'Brien–Fleming are specific families of such boundaries.

you can flip beats with ← → keys

What it means

What decision this changes

If you want to watch mid-test, choose a sequential design before the start. Changing the stopping rules mid-test is not allowed.

If the team needs to monitor a test and stop early, that is not a reason to break the statistics — it is a reason to switch on the sequential mode in the A/B platform. Then an early stop is honest, not self-deception.

Without such methods the rule is simple and hard: fix the sample size and the duration in advance, decide once at the end. Any "let's switch it off, it's already significant on day three" in a fixed-horizon test inflates false wins.

Where it shows up

Clinical trials have used group sequential designs for decades: interim analyses make it ethical to stop a trial when a drug clearly works or clearly harms.

Modern product A/B platforms (including at major tech companies) offer always-valid statistics precisely so product managers can peek at the dashboard honestly.

Definitions

Sequential test

a method that allows repeated in-flight checks while controlling the overall Type I error.

In plain words: a design in which interim looks are permitted in advance.

Group sequential

planned interim cuts with tightened thresholds (O'Brien–Fleming, Pocock) and an early-stop option.

In plain words: several planned stopping points, each with its own error budget.

Always-valid p-value

a p-value/interval you may look at any moment, any number of times, without inflating the error.

In plain words: a p-value you can trust whenever you look.

Error budget (alpha spending)

distributing the allowed Type I error across several checks so the total stays at 5%.

In plain words: 5% error is a budget: spend a piece on each check.

When the method lies (assumptions)

The speed gain is not free: in the worst case sequential methods require no smaller — and sometimes a larger — maximum sample; the saving is on average, not guaranteed.

You must use a method built for the purpose. Taking a regular fixed-horizon p-value and checking it daily is not a "sequential test" — it is exactly the peeking.