Reading a Confusion Matrix When the Positive Class Is Rare

Most of the classification problems I care about have one thing in common: the interesting class is rare. Fraudulent transactions, phishing emails, a price that has been keyed in wrong, a holding that does not reconcile. When the positives are a few per cent of the data or less, the headline accuracy figure stops telling you anything useful, and the confusion matrix becomes the first thing worth looking at.

This post walks through one worked example from start to finish: the four cells of the matrix, the numbers classification_report prints, and how to turn them into a decision about where to set the threshold.

The four cells

A binary classifier makes two kinds of correct call and two kinds of mistake. Laid out as a two-by-two grid, with what really happened down the side and what the model said across the top, you get the confusion matrix. scikit-learn uses exactly this orientation: rows are the true labels and columns are the predicted labels, so for labels 0 and 1 the result reads [[TN, FP], [FN, TP]] (see the confusion_matrix reference).

It helps to give the cells business names before looking at any numbers:

• True positive: a fraudulent transaction the model flagged. This is the work you wanted done.
• False positive: a genuine transaction the model flagged. Someone spends time reviewing it and finds nothing.
• False negative: fraud the model let through. Usually the expensive one.
• True negative: a genuine transaction left alone. The bulk of the data.

A worked example

The numbers below are made up for illustration. Imagine a month of 10,000 card transactions, of which 100 (1%) are fraudulent. A model scores every transaction and flags the 150 with the highest scores. Of those 150, 80 turn out to be fraud.

Example confusion matrix for 10,000 transactions

Actual class	Predicted genuine	Predicted fraud	Total
Actual genuine	9,830 TN	70 FP	9,900
Actual fraud	20 FN	80 TP	100
Total	9,850	150	10,000

Accuracy is (9,830 + 80) / 10,000 = 99.1%. That sounds excellent until you notice that a “model” which never flags anything scores 99.0%, because 9,900 of the 10,000 transactions are genuine. The extra 0.1 percentage points are the entire value of the model, and accuracy hides it.

The two numbers that do carry information are:

• Precision = TP / (TP + FP) = 80 / 150 = 53%. Of everything flagged, how much was real? This is the hit rate the review team experiences.
• Recall = TP / (TP + FN) = 80 / 100 = 80%. Of all the fraud that happened, how much did we catch?

F1 is the harmonic mean of the two: 2 × 0.533 × 0.80 / (0.533 + 0.80) = 0.64. It is a convenient single number for comparing models, but it weights precision and recall equally, which is rarely how the business sees it.

What classification_report adds

In Python you rarely compute these by hand. The snippet below rebuilds the example and prints the matrix and the report:

import numpy as np
from sklearn.metrics import confusion_matrix, classification_report

# Rebuild the illustrative example: 1 = fraud, 0 = genuine
y_true = np.array([0] * 9_900 + [1] * 100)
y_pred = np.array([0] * 9_830 + [1] * 70 +   # genuine: 9,830 TN, 70 FP
                  [0] * 20 + [1] * 80)       # fraud:   20 FN, 80 TP

print(confusion_matrix(y_true, y_pred))
print(classification_report(y_true, y_pred,
                            target_names=["genuine", "fraud"], digits=3))

The output looks like this:

[[9830   70]
 [  20   80]]
              precision    recall  f1-score   support

     genuine      0.998     0.993     0.995      9900
       fraud      0.533     0.800     0.640       100

    accuracy                          0.991     10000
   macro avg      0.766     0.896     0.818     10000
weighted avg      0.993     0.991     0.992     10000

Three things in that report are worth reading carefully.

The fraud row is the one that matters. The genuine row will look near perfect on almost any imbalanced problem, because getting the majority class right is easy.

Support is the class count. Here it reminds you that the fraud metrics rest on only 100 examples. Catch two more or two fewer and recall moves by two points. On small positive classes I treat the third decimal place as noise.

Macro and weighted averages answer different questions. The macro average treats both classes equally, so it is dragged down by the weak fraud row. The weighted average weights each class by its support, so on imbalanced data it mostly repeats the majority class and looks flattering. If someone quotes a single “weighted F1” for a fraud model, ask for the fraud row instead. The scikit-learn model evaluation guide documents how each average is calculated.

From metrics to a threshold

A confusion matrix describes one threshold. Most classifiers produce a score, and “flag the top 150” was a choice. The useful question is which threshold gives the best trade-off for the people who act on the output.

One practical way to frame it is to put a rough cost on each type of mistake. Again with made-up numbers: say reviewing a false alarm costs 1 unit of analyst time and a missed fraud costs 25 units. At the threshold above, the cost is 70 × 1 + 20 × 25 = 570.

Now loosen the threshold so the model flags 300 transactions and catches 92 of the 100 frauds. Precision falls to 31%, which looks worse on paper. But the cost is 208 × 1 + 8 × 25 = 408, a clear improvement. The metric that looked worse produced the better outcome, because misses were much more expensive than false alarms.

scikit-learn’s precision_recall_curve gives you precision and recall at every threshold, so you can run this calculation across the whole range rather than for two hand-picked points. The cost ratio is a business input, not a modelling one, and agreeing it up front with whoever owns the review queue saves a lot of argument later.

A short checklist

• Always compare against the do-nothing baseline. On 1% positives, 99% accuracy is the floor, not an achievement.
• Read the minority-class row of the report first, and check its support.
• Evaluate on data the model has not seen, ideally a later time period than the training data, since fraud patterns drift.
• Choose the threshold from the cost of each error, then report precision and recall at that threshold.
• Re-check the matrix after deployment. Base rates change, and precision moves with them even when the model does not.

None of this needs new tools. It needs the habit of opening the confusion matrix before quoting a single number, and of asking what each cell costs.