Evaluation Metrics: Precision, Recall, ROC AUC and Thresholds

A model’s usefulness depends on how its mistakes affect the decision it supports. For churn, missing a customer who leaves costs revenue; contacting a customer who would have stayed costs a discount or an agent’s time. Evaluation metrics measure these mistakes, and the decision threshold decides the balance between them.

The confusion matrix

For a yes/no prediction, every case falls into one of four boxes:

• True positive: predicted churn, and they churned (caught).
• False positive: predicted churn, but they stayed (false alarm).
• False negative: predicted stay, but they churned (missed).
• True negative: predicted stay, and they stayed.

From these come the main metrics:

• Precision: of the customers flagged, how many really churned? (caught / flagged)
• Recall: of the customers who churned, how many were flagged? (caught / all churners)
• ROC AUC: how well the model ranks churners above stayers, across all thresholds. 0.5 is random, 1.0 is perfect.

Choosing a threshold

A classifier outputs a probability. You choose the cut-off above which a customer is flagged. The default of 0.5 is rarely the best choice.

The dataset used in every lesson of this course is a made-up churn table. Run this block first:

import numpy as np
import pandas as pd

def make_customers(n=5000, seed=7):
    """A made-up churn dataset: who cancels their subscription in the next 60 days?"""
    rng = np.random.default_rng(seed)
    tenure = rng.integers(1, 60, n)                       # months as a customer
    spend = np.round(rng.normal(40, 15, n).clip(5), 2)    # monthly spend
    tickets = rng.poisson(1.2, n)                         # support tickets last quarter
    last_login = rng.integers(0, 45, n)                   # days since last login
    logit = -1.2 - 0.05 * tenure + 0.35 * tickets + 0.06 * last_login - 0.01 * spend
    churned = rng.random(n) < 1 / (1 + np.exp(-logit))
    return pd.DataFrame({"tenure_months": tenure, "monthly_spend": spend, "support_tickets": tickets,
                         "days_since_login": last_login, "churned": churned.astype(int)})

Then see how the threshold changes the result:

from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import confusion_matrix, precision_score, recall_score, roc_auc_score

df = make_customers()
X, y = df.drop(columns="churned"), df["churned"]
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=0, stratify=y)
proba = LogisticRegression(max_iter=1000).fit(X_train, y_train).predict_proba(X_test)[:, 1]

print(f"ROC AUC: {roc_auc_score(y_test, proba):.3f}")
for threshold in (0.5, 0.3, 0.2):
    pred = (proba >= threshold).astype(int)
    tn, fp, fn, tp = confusion_matrix(y_test, pred).ravel()
    print(f"threshold {threshold}: flagged {tp + fp:4}  caught {tp:3}  missed {fn:3}  false alarms {fp:3}  "
          f"precision {precision_score(y_test, pred):.2f}  recall {recall_score(y_test, pred):.2f}")

ROC AUC: 0.832
threshold 0.5: flagged  163  caught 112  missed 197  false alarms  51  precision 0.69  recall 0.36
threshold 0.3: flagged  396  caught 215  missed  94  false alarms 181  precision 0.54  recall 0.70
threshold 0.2: flagged  600  caught 267  missed  42  false alarms 333  precision 0.45  recall 0.86

The model is the same in all three rows; only the threshold changes:

• At 0.5, flags are usually right (precision 0.69), but most churners are missed (recall 0.36).
• At 0.2, the team catches 86% of churners, but more than half of the contacted customers would have stayed.

Which is better depends on costs. If saving a churner is worth 200 and a contact costs 10, flagging more customers pays off. If the team can only make 200 calls a week, pick the threshold that flags about 200 customers. Choose the threshold from the business decision, not from habit.

Choosing a metric

Situation	Useful metrics
Classes are balanced and errors cost about the same	Accuracy, F1
Rare positive class (fraud, churn, defects)	Precision, recall, precision-recall curve
Comparing models before choosing a threshold	ROC AUC, PR AUC
Predicting numbers	MAE, RMSE

Common mistakes

1. Using accuracy on imbalanced data.
2. Keeping the 0.5 threshold without thinking about costs.
3. Reporting one metric without the confusion matrix behind it.
4. Choosing the threshold on the test set, then reporting the test score as unbiased; use validation data to choose it.

Key takeaway

Read the confusion matrix, then use precision, recall and ROC AUC to describe a model’s mistakes. Choose the decision threshold from the business costs and capacity, not the default. Next: the two traps that make models look better than they are, overfitting and data leakage.