What a production metric can’t tell you

The average looked fine

The clearest example I know comes from medical imaging. A 2020 study looked at a chest X-ray model for detecting a collapsed lung that scored 0.87 AUC overall. Split by whether the X-ray showed a chest drain, it scored 0.94 with a drain and 0.77 without one. A drain is how a collapsed lung gets treated, so the model did best on patients who already had help and worst on the untreated cases, which are the ones it existed for. And 80% of the positive cases in the test set had drains, so the overall number mostly described them.

Nothing was wrong with the 0.87. It answered a question about the whole population, and the risk sat in one part of it.

There's a second way a healthy metric misleads: it measures something next to what you care about. Kohavi and colleagues describe an Office Online redesign that cut clicks per user by 64%. Clicks were the stand-in for purchases, so that looked like a disaster, but the people who did click were much more likely to buy. The proxy pointed the wrong way.

Slice on purpose, and don't over-read the slices

Gender Shades is the well-known version of this. Three commercial face classifiers had overall accuracy between 87.9% and 93.7%. For darker-skinned women their error rates were between 20.8% and 34.7%. For lighter-skinned men they were under 1%. You only see that if somebody chooses the groups and breaks the results down.

For agents I'd slice by task difficulty, input type and how much a failure costs. A wrong answer about store hours and a wrong refund aren't the same failure, even if they count the same in a success rate.

The catch is that slicing creates a lot of comparisons, and some will look alarming by chance. With enough slices there's always a worst one. Statisticians have handled this for decades, and Benjamini and Hochberg gave the standard way to limit how many flagged results are false alarms. Even without the formal method, treat a bad small slice as a lead, not a conclusion.

Read the runs behind the number

Latency people learned a long time ago that averages hide the tail. The Google SRE book has an example where a typical request takes about 50 ms but 5% of requests are 20 times slower, and an alert on average latency barely moves while the tail changes a lot. Its answer is to treat metrics as distributions and watch percentiles.

Percentiles still don't tell you why. For that you have to look at actual runs. When a metric moves I pull a sample of individual traces from before and after the change and read them. Kohavi's group has a name for the reflex this takes, Twyman's law, which says a number that looks surprising is usually wrong. Two of their Bing "wins" turned out to be improvements in how clicks were tracked, not in what users did, and only digging under the top line showed that.

Choosing the next investigation

Put together, the routine is short. Decide which outcome the metric stands in for. Break results down by the segments where failure costs the most. Treat surprising slices as leads, and read traces before believing any story about why the number changed.

A production metric is good at telling you that something changed. It's bad at telling you who it changed for, and that's usually the part that matters.