Class imbalance: when ROC deceives

Fraud — 0.2% of transactions. A disease — 1% of patients. When the target class is rare, the usual metrics start lying, and ROC/AUC look deceptively good. Let's see how to evaluate a classifier honestly on a rare class. Shrink the rare class share and change the separability — watch the PR curve and the AUC.

ROC-AUC ≈ 0.92 (barely depends on rarity)Precision at recall 80% ≈ 25%

Rare class share5%

Class separability2.0

Shrink the class share: ROC-AUC barely changes (looks "fine") while the PR curve and precision at a fixed recall collapse. On a rare class you watch PR, not ROC/accuracy.

First, recall the accuracy trap under imbalance: if fraud is 0.2%, the model "everything is honest" is right 99.8% of the time — accuracy is nearly perfect and the usefulness is zero (not a single fraudster caught). On a rare class accuracy is meaningless — we saw this in the confusion matrix.

you can flip beats with ← → keys

What it means

The first question in classification: how rare is the class? If rare and important (fraud, disease) — set accuracy and bare ROC-AUC aside and look at precision/recall, PR-AUC, F1, tuning the threshold to the cost of a miss vs a false alarm.

A high AUC under heavy imbalance is a reason not to celebrate but to check precision at the operating threshold: it often turns out that 95% of the alerts are false and the system is drowning.

Where it shows up

Anti-fraud and medical screening are the classic imbalance: real cases are a handful, so everything revolves around precision/recall and the cost of errors, not around overall accuracy.

Moderation and spam: with a rare violation it is vital not to bury users under false blocks — that is direct control of precision at the required recall.

Definitions

Class imbalance

one class is far rarer than the other (fraud, disease, churn) — the usual metrics get distorted.

PR curve

precision against recall across thresholds; on a rare class more honest than ROC, since precision feels the rarity.

F12·P·R / (P+R)

the harmonic mean of precision and recall: high only when both metrics are high.

Weighting / re-sampling

remedies for imbalance: class weights during training, down/up-sampling of the rare class.

When the method lies (assumptions)

Accuracy and ROC-AUC are informative on balanced classes; under heavy imbalance they systematically flatter — you need PR metrics.

Re-sampling and weights change the base rate, so the "probabilities" the model outputs afterwards are biased and need calibration if they are read as real probabilities.