Regression quality metrics: MAE, RMSE, MAPE
We know how to draw the line. But any forecast misses — the question is by how much. To compare models and know whether to trust them, the error is rolled up into a single number. There are several such numbers, and they answer different questions. Drag the points and add an outlier — watch how each metric reacts.
Drag the points and watch the metrics. RMSE is always ≥ MAE and reacts to an outlier more sharply.
Each point has a residual — its distance to the line (the yellow ticks). All quality metrics are different ways to average these misses into one number. The smaller it is, the more accurate the model.
In practice the metric is fixed before training the model — otherwise there is a temptation to pick the one where the result looks prettier. And you watch more than one: MAE and RMSE side by side immediately show whether the data has heavy outliers (if RMSE is noticeably larger than MAE — it does).
These same metrics drive model selection in machine learning: out of a dozen models you take the one with the smaller error on new data. And what exactly counts as "error" — MAE, RMSE or something else — is determined by the business meaning of the task.
Forecasting weather, demand, prices, travel time — quality is measured exactly this way everywhere. A delivery service optimizes the RMSE of predicted time: better to be consistently off by a couple of minutes than occasionally by half an hour.
Financial and product forecasts are often judged in MAPE: "the model is off by 8% on average" is clear to management and analysts alike, and does not depend on the units.
Definitions
MAPE explodes on values near zero and is asymmetric: under- and over-forecasting are fined differently. For metrics with zeros (daily demand, rare orders) take MAE/RMSE or weighted variants.
R² grows from adding any variables and says nothing about performance on new data. A high R² with a nonlinear relationship, or when extrapolating beyond the training range, is an illusion of accuracy.