Overfitting and honest validation (train/test)
A model can be made arbitrarily complex and forced to pass perfectly through every training point. But that is not the goal: the model must work on NEW data it has never seen. Here hides the main trap of machine learning β and the reason all the metrics (AUC, RMSE) must be measured the right way.
The white points are the data the model learns on. The teal curve is the model itself. The yellow points are new data the model has not seen. The slider sets the model's complexity (the curve's flexibility).
Splitting into train and test (and more reliably β cross-validation) is basic machine learning hygiene. All the metrics from the previous lessons (AUC, precision/recall, RMSE) are meaningful only on held-out data. A model praised for accuracy on its training data can be useless in production.
This continues the thread about sample size and honesty: a pretty result on the same data the model was tuned on easily turns out to be self-deception β memorized noise.
Overfitting explains why a "genius" trading strategy that perfectly describes the past loses money on the future β it memorized history's accidents.
And why a model trained on one city or one camera starts failing in new conditions: it adapted to particulars, not to the essence.
Definitions
The honesty of train/test rests on the test taking NO part in training or tuning. Data leakage (test bleeding into training, or a feature "from the future") paints a pretty but fake estimate β there is a separate lesson on this in the traps module.
The split must respect the structure: for time series the test comes LATER than the training (you cannot learn on the future); for groups (one user β many rows) split by group, otherwise the estimate is inflated.
Deep dive: the math and the mechanism (optional)
Under the hood is the bias-variance trade-off. A simple model has high bias (systematically off) but low variance (stable across samples). A complex one β low bias but high variance (wiggles from sample to sample). The error on new data β biasΒ² + variance + irreducible noise, and the minimum is in the middle of the complexity range.
Overfitting is fought not only with train/test: regularization (a penalty for complexity), more data, a simpler model, early stopping. All of them shift the bias-variance balance toward the optimum.