The confusion matrix: precision, recall and the accuracy trap
A classifier has four possible outcomes, and collapsing them into one "share of correct answers" is dangerous. To measure quality honestly, the outcomes are laid out in a confusion matrix, and metrics are computed from it to fit the specific task. Move the threshold and the class separability β all the numbers recompute.
Move the threshold β Precision and Recall trade off. Worse separability (overlapping classes) β both metrics fall. Accuracy is deceptive under imbalance: with little spam, "nothing is spam" gives high accuracy but Recall = 0.
Every prediction lands in one of four cells. TP (true positive) β caught spam. TN (true negative) β correctly let a regular email through. FP (false positive) β wrongly marked a real email as spam. FN (false negative) β missed spam into the inbox. These four numbers ARE the confusion matrix.
The first thing done with a trained model is building the confusion matrix and looking at precision/recall, not a single accuracy. Especially if the class of interest is rare: fraud, disease, churn. There, high accuracy means almost nothing.
When one number is needed to compare models, F1 is used β the harmonic mean of precision and recall: it is high only when both metrics are high. But even F1 is chosen deliberately, remembering that precision and recall answer different questions.
Medical screening is tuned for high recall: better to re-check a healthy person than to miss a sick one. That is why a sensitive first test is followed by a precise confirmatory one β it recovers the precision.
A spam filter, by contrast, protects precision: the user cares more about not losing a real email than about occasionally seeing spam in the inbox. The same confusion matrix, but opposite priorities β and the threshold sits in different places.
Definitions
The matrix is computed at a specific threshold: move it β and precision and recall change. Comparing models "at the default threshold of 0.5" is incorrect, especially on imbalanced classes.
Precision depends on the share of the positive class in the stream: move the model to where fraud is half as frequent β precision drops by itself, with no degradation of the model. Recall does not depend on the base rate.