Precision, Recall, and the Mistake Most People Make Evaluating Models

Here's a sentence that sounds like good news and isn't: "our spam filter is 96.3% accurate." I've watched that number get presented in a meeting as a win, right before someone asked how many actual spam emails it catches, and the room went quiet. This post is the numbers behind why that question matters more than the accuracy figure, worked out with a small, concrete example instead of definitions alone.

The setup: 1,000 emails, 30 of them spam

Imagine a test set of 1,000 emails. 970 are legitimate, 30 are spam — a realistic ratio, since most inboxes are mostly not spam. Now consider two "models":

Model A: predicts "not spam" for everything

This isn't a real model, it's a joke baseline, but stick with it. It gets every legitimate email right (970 correct) and every spam email wrong (30 missed).

Accuracy = 970 / 1000 = 97.0%

Ninety-seven percent accuracy, and it does zero useful work — it has never once caught a spam email. This is usually the first "wait, what?" moment for anyone new to imbalanced classification: a completely useless model can post a better-sounding number than a genuinely useful one.

Model B: an actual trained classifier

Now suppose we train a real model on features like sender reputation, link count, and specific word patterns — the same kind of engineered features you'd get by running a candidate email's body through something like a word counter to check for suspiciously short bodies or repeated phrase spam patterns, then feeding those counts in as inputs. On our 1,000-email test set, it produces this confusion matrix:

Accuracy = (18 + 945) / 1000 = 96.3%

Model B's accuracy is lower than the do-nothing baseline's 97.0%, despite Model B actually catching 18 real spam emails that Model A missed entirely. If accuracy were the only number in the room, Model A would win the comparison, which should be enough on its own to explain why nobody serious evaluates imbalanced classifiers on accuracy alone.

Precision and recall, computed from the same table

Precision answers: "of the emails we flagged as spam, how many actually were?"

Precision = TP / (TP + FP) = 18 / (18 + 25) = 18 / 43 = 0.419 (41.9%)

Recall answers: "of the emails that were actually spam, how many did we catch?"

Recall = TP / (TP + FN) = 18 / (18 + 12) = 18 / 30 = 0.600 (60.0%)

Now the picture is very different from the accuracy figure. Model B catches 60% of spam (recall), but well over half of what it flags as spam is actually legitimate mail wrongly caught in the filter (a precision of 41.9%). Neither of those facts was visible in the 96.3% accuracy number. Accuracy blended a large pool of easy, correct "not spam" predictions with a much smaller and much more mistake-prone pool of spam predictions, and the large pool drowned out the small one.

Where F1 fits in

Precision and recall trade off against each other — you can push recall toward 100% by flagging almost everything as spam, tanking precision, or push precision toward 100% by only flagging emails you're extremely confident about, tanking recall. F1 is the harmonic mean of the two, a single number that penalizes models for being lopsided in either direction:

F1 = 2 * (precision * recall) / (precision + recall)
= 2 * (0.419 * 0.600) / (0.419 + 0.600)
= 2 * 0.2514 / 1.019
= 0.493

An F1 of about 0.49 is a much more honest single-number summary of Model B than 96.3% accuracy is. It's not a great score, and it shouldn't feel like one — the model is currently wrong more often than right on the specific job it exists to do, which is separating spam from legitimate mail.

Same model, different threshold: watching precision and recall trade off

Model B doesn't actually output "spam" or "not spam" directly — it outputs a probability, and somewhere in the code a threshold (usually 0.5 by default) turns that probability into a decision. Nothing forces you to keep that default. Moving the threshold on the exact same trained model, without retraining anything, changes precision and recall in opposite directions. Here's what that looks like with illustrative numbers on our same 1,000-email set:

Lowering the threshold to 0.3 means the model flags an email as spam more easily. It catches more real spam (recall jumps to 80%) but drags in far more false positives, tanking precision to 33.3%. Raising it to 0.7 does the opposite: the model only flags emails it's quite confident about, so precision climbs to 55.6%, but it now misses two-thirds of actual spam. None of these three thresholds is objectively "correct" — which one you'd actually ship depends entirely on which mistake costs you more, which is exactly the question accuracy never forces you to answer.

In practice, you'd sweep across many threshold values, not just three, and plot precision against recall at each one. That plot is a precision-recall curve, and for genuinely imbalanced problems like this one, it tends to be far more informative than a standard ROC curve. ROC curves plot true positive rate against false positive rate, and because the false positive rate is measured against a huge pool of negatives (970 legitimate emails here), a model has to be dramatically bad before its ROC curve looks bad — the same large-denominator problem that made accuracy misleading in the first place.

Precision-heavy vs. recall-heavy: it depends on what a mistake costs

There's no universally "better" side of the precision-recall trade-off — it depends entirely on the problem in front of you:

• Spam filtering leans precision-heavy. Blocking a real business email is a worse day for the user than one extra spam message reaching the inbox, so most production spam filters are tuned toward a higher threshold than the naive 0.5 default.
• Medical screening tests lean recall-heavy. Missing an actual case of a disease during initial screening can be far more costly than a false alarm that gets ruled out by a follow-up test, so screening tools are usually tuned to catch nearly everyone, accepting more false positives along the way.
• Fraud transaction blocking often lands in the middle. Blocking too many legitimate purchases (low precision) frustrates real customers and costs revenue directly; missing too much fraud (low recall) costs money directly too, so teams typically tune the threshold against an actual dollar cost estimate for each type of mistake rather than picking a round number.

Whichever side you lean toward, the decision should be made on purpose, with the confusion matrix numbers in front of you — not left as whatever the training library's default threshold happens to produce.

What changes with more than two classes

Everything above used a binary spam/not-spam example because the arithmetic stays simple enough to follow by hand, but the same trap shows up, arguably worse, in multi-class problems. Picture a support-ticket classifier sorting tickets into "billing," "bug," "how-to," and "other," where "other" makes up 80% of tickets. A model that just predicts "other" every time still posts a deceptively high accuracy, for the exact same reason Model A did above — one dominant class inflates the headline number regardless of how many classes exist. The fix scales the same way: compute precision and recall per class, not just one blended accuracy figure, and pay closest attention to the classes you actually care about catching, even if they're the smaller ones.

The mistake, named directly

The mistake isn't using accuracy. Accuracy is a perfectly fine metric when classes are roughly balanced and errors cost about the same in both directions. The mistake is using accuracy by default, without checking the class balance first, on a problem where one class vastly outnumbers the other. When 97% of your data belongs to one class, a model has to actively work to score below 97% accuracy — which means accuracy stops measuring whether the model is good at its actual job and starts measuring whether it agrees with the majority class.

I've seen this play out with churn prediction, fraud detection, and defect detection in manufacturing data — anywhere the thing you actually care about catching is the rare event, accuracy quietly becomes the wrong scoreboard. It's the same imbalance issue that shows up during data preparation, discussed from the pipeline side in building a tiny end-to-end ML pipeline, where checking value_counts() on the target column before training is step one for exactly this reason.

What to check instead

• Look at the class balance first. Run a value count on your target column before you train anything. If one class is above roughly 80–90%, accuracy alone is not going to tell you much.
• Report precision, recall, and F1 alongside accuracy, not instead of it — all three together, plus the confusion matrix itself, give a fuller picture than any single number.
• Decide which error costs more before you tune anything. A missed spam email (false negative) is mildly annoying. A legitimate business email caught in the spam folder (false positive) can mean a missed job offer. In fraud detection those costs usually flip — a missed fraud case can be expensive, so recall often matters more there than precision.
• Adjust the classification threshold, not just the model. Most classifiers output a probability, and the 0.5 cutoff for "spam" vs "not spam" is a default, not a law. Moving that threshold trades precision for recall in a way you can tune to the cost structure of your actual problem.
• Use precision-recall curves for genuinely imbalanced problems instead of ROC curves, which can look deceptively good on heavily imbalanced data for reasons similar to the accuracy trap above.

The takeaway

A 96.3% accurate spam filter and a 97.0% accurate do-nothing baseline are not close in quality, even though their headline numbers are close in value. The confusion matrix underneath both of them tells the real story: one catches 18 spam emails, the other catches zero. If you evaluate a model on accuracy alone without first checking whether your classes are balanced, you're not measuring what you think you're measuring.