Data leakage

The most insidious trap of machine learning — the model shines on the test set and falls apart in production. Often the culprit is leakage: information that will not exist at the real moment of prediction accidentally seeped into training. The model "peeked at the answer" — and fooled everyone, including its authors.

Test ≈ production (78% vs 77%) — an honest estimate. The test predicts the real quality.

Production quality (green) is the real one — it does not change. Leakage inflates only train/test: a "feature from the future" (say, the account closure date when predicting churn) gives away the answer; preprocessing on all the data before the split lets the test peek into training. The goal of validation is test ≈ production.

Leakage is when a feature or part of the test set carries information from the "future" or from the answer itself. The model learns on it, the test metrics soar — but in reality that feature does not exist at prediction time, and quality collapses. It is not overfitting per se: even a model honest in its complexity will lie if the data "leaked".

you can flip beats with ← → keys

What it means

Prevention is pipeline discipline: split first, then all preparation on train only (fit on train, transform on test). Features are checked for "availability in time": can they be computed strictly before the moment of the forecast.

Suspiciously high quality is a reason not to celebrate but to hunt for leakage. "Too good" in ML almost always means a validation error, not a genius model.

Where it shows up

Famous failures of competitions and startups: the model showed AUC 0.99 on the test and flopped in the field — because an identifier or a field indirectly encoding the answer had seeped into the data.

Medical models "predicted" disease from the technical labels of the scan (the oncology center's machine name), not from the scan itself — classic leakage through an extraneous feature.

Definitions

Data leakage

information unavailable at the real moment of prediction (from the future or from the answer) gets into training.

Feature from the future

a variable that becomes known only AFTER the event being predicted.

Preprocessing leakage

normalization/imputation/feature selection on all the data before the split — the test peeks into train.

Group/time leakage

the same object in train and test, or random shuffling of temporal data instead of a chronological split.

When the method lies (assumptions)

Train/test and cross-validation give an honest estimate ONLY if no training information seeped into the test. Under leakage, even formally correct validation lies.

For temporal and grouped data, ordinary random splitting is itself a leak — you need time-based/group-based validation schemes.