How good is the forecast: backtesting and metrics
A model trained on the whole history always looks convincing: the line passes close to the data. But the quality of a forecast is measured not by how it described the past, but by how it handled what it had never seen. Hence backtesting: cut the history at some point, forecast the next two weeks, compare with the fact, move the point and repeat. The three windows on the chart switch with a click.
The split runs along time, never at random: a randomly chosen test day would leave the model its own neighbours โ tomorrow predicted from the day after tomorrow.
First rule: the split runs along time, not at random. In ordinary machine learning the test part is drawn at random, and rightly so โ there the observations are independent. In a series neighbouring days are nearly identical, so a randomly chosen test Tuesday ends up surrounded by a training Monday and Wednesday.
What will the metric show if the test days are picked scattered across the whole series?
What it means
The horizon of a backtest is chosen by how far ahead the forecast is actually needed. If purchasing plans are made two weeks out, testing the model one day ahead is pointless โ the number will be pretty and irrelevant.
A report on forecast quality is worth writing in three numbers: metric, horizon, baseline. ยซMAPE 8% over 14 days, the seasonal naive gives 11%ยป is substantive. ยซAccuracy 92%ยป is not.
The backtest error is systematically more optimistic than the future one: the model and its orders were picked while looking at that same history. To account for it, the final check is made on a period that took part neither in the selection nor in the debugging.
Where it shows up
In demand forecasting the cost of an error is asymmetric: under-ordering means lost sales, over-ordering means write-offs. Then instead of MAE one takes a metric with different penalties for the sign of the miss โ and optimises a quantile of the distribution rather than a point.
In finance a backtest is a mandatory procedure for any strategy, and that is where its pitfalls are best studied: overfitting to a period, survivorship bias in the instrument set, and using data that was not yet known at the time.
Weather forecasts have long been published probabilistically: ยซ60% chance of rainยป rather than ยซit will rainยป. Forecasts of business metrics are moving the same way โ an interval is more useful than a point.
Definitions
When the method lies (assumptions)
A backtest assumes the future is built like the past. If the history holds no pandemic, no competitor leaving the market and no change of pricing policy, the backtest says nothing about how the model behaves under such conditions.
Comparing models by a metric on one and the same period is multiple comparisons: the more variants you try, the higher the chance the winner won by luck. Across dozens of configurations a difference of a fraction of a percent means nothing.
Deep dive: the math and the mechanism (optional)
MASE solves both problems of MAPE at once: the model's error is divided by the average error of a naive forecast on the training sample. One means ยซas good as naiveยป, 0.7 means ยซ30% better than naiveยป. The metric is defined at zeros, symmetric, and comparable across series of different scale โ which is why it became the standard in forecasting competitions.
For interval forecasts point metrics are no use at all: an interval is judged by coverage (what share of facts landed inside the 95% interval โ it should be about 95%) and by the pinball loss, which penalises both a miss and excessive width. A model with a perfect MAE and permanently narrow intervals is more dangerous than an honest model with wide ones.