Interactive statistics

56 free interactive lessons: from mean and median to A/B tests and Bayes. Drag the charts and build intuition. No sign-up.

Module 9: Data traps

Survivorship bias

The most insidious kind of bias is when the data you do NOT see matters more than the data you have. The classic example β€” WWII airplanes.

● bullet holes of the returned
The military wanted to reinforce where there were most holes β€” the wings and the tail. Where is the armor actually needed?

The military studied planes that returned from missions to decide where to add armor. The white dots are bullet holes. Sounds logical to reinforce where there are most of them β€” the wings and the tail?

What it means

"College dropouts become billionaires" is textbook survivorship bias. For every Zuckerberg there are thousands of dropouts with no success, and they are simply absent from that sample.

The same in product work: you survey the remaining users and hear everything is fine β€” while the key churn data sits with those who left silently and never made it to the survey.

Where it shows up

"How to succeed" books dissect companies that survived β€” that is why they get written about. Those that did the same and went broke never made the sample.

Investors see the returns of funds that still exist; the ones closed at a loss quietly vanish from the statistics, and the average return looks better than reality.

Definitions
Survivorship bias
concluding from those who "lived" into the data while ignoring those who dropped out of it.
Selection bias
the more general case: the sample is unrepresentative because not everyone makes it in.
Next β†’
Enjoying the materials?

This site is built by the DataSlice Telegram channel: data analytics in plain words β€” real cases, metrics, careers. The channel is in Russian.

Open the Telegram channel