Interactive statistics

56 free interactive lessons: from mean and median to A/B tests and Bayes. Drag the charts and build intuition. No sign-up.

Module 9: Data traps

Goodhart's law: the metric as a target

The last trap is about the metrics themselves. As soon as a metric is made a target, people start gaming it, and it stops reflecting what it was introduced for.

Target metricReal valueE.g.: "calls per hour" ↑ while conversation quality ↓
While the pressure is moderate, the metric and the value grow together.

A call center decided to push the "calls per hour" metric. Move the pressure on the metric. At first both the metric and the real value (the quality of the conversation with the customer) grow together β€” all is well.

What it means

When setting goals (OKRs, KPIs), experienced teams always ask: "how can this goal be gamed to the detriment of the business?" β€” and add a counter-metric that will catch it.

This is a direct bridge to the second part of the site: in every metric hierarchy the bottom level is guardrails. They exist precisely because of Goodhart's law.

Where it shows up

Schools that drill for the test instead of knowledge; hospitals turning away severe patients to improve mortality statistics; social networks burning out their audience for momentary engagement.

Everywhere the same: one indicator got optimized β€” and the system it was supposed to measure broke.

Definitions
Goodhart's law
when a measure becomes a target, it ceases to be a good measure β€” people start gaming it.
Gaming the metric
formally improving the indicator to the detriment of the real goal it was introduced for.
Counter-metrics (guardrails)
quality constraints next to the target metric that catch gaming and degradation.
Next β†’
Enjoying the materials?

This site is built by the DataSlice Telegram channel: data analytics in plain words β€” real cases, metrics, careers. The channel is in Russian.

Open the Telegram channel