Peeking: how to ruin an honest test
The test of the new "Order" button is on its third day, and the p-value has already blinked below 0.05. Stop and celebrate? An A/B test is honest only if you play by the rules, and the most common mistake is peeking at the result and stopping the test the moment you see "significant". We'll show why this breaks everything β using an A/A test, where there is provably no difference.
The top chart is the p-value day by day as the sample accumulates (below β how it grows toward the required size). With no effect, the p-value wanders at random and will almost surely dip under 0.05 at least once β but that is a false reason to stop. The honest rule: read the result on the planned day (the black dashes), not "as soon as it turns significant".
We launch an A/A test: both variants are actually identical, there is no real difference. The top line is one experiment's p-value day by day as the sample accumulates; below you can see it growing toward the size fixed in the design. The yellow line is the 0.05 threshold, the black dashes mark the planned end day. Press "run 1 experiment".
The default is to wait for the planned end of the test. Every early stop "on significance" raises the chance of buying noise instead of an effect.
Product teams fix the test duration and don't touch it early. "Let's switch it off, it's already significant on day three" is the classic way to ship something that doesn't actually work.
This follows directly from the sample size lesson: a test must be run to its planned size, not stopped at the first convenient spike.
In science this has a name β p-hacking: data and analysis choices are shuffled until p < 0.05 pops out. This is why a share of headline results later fails to replicate.
The same temptation lives in trading and marketing: test enough "strategies" and one will randomly look great on the past β and crumble on the future.