ROC AUC vs PR AUC: Which Metric to Use for Imbalanced Data

Both metrics summarise how well a classifier ranks positives above negatives, across every possible threshold. They differ in what they reward:

• ROC AUC is the area under the ROC curve, which plots the true positive rate (recall) against the false positive rate. It answers: “If I pick one positive and one negative at random, how often does the model score the positive higher?” A random model scores 0.5; a perfect one 1.0.
• PR AUC is the area under the precision-recall curve. It answers: “As I catch more of the positives, how many of my alerts are real?” A random model scores about the share of positives in the data (for 1% fraud, about 0.01); a perfect one 1.0.

The practical rule: for rare positive classes (fraud, defects, rare diseases), look at PR AUC. ROC AUC can look comfortable while the model still produces mostly false alarms.

Why ROC AUC can mislead on rare classes

The false positive rate divides false alarms by the number of negatives. When negatives vastly outnumber positives, even thousands of false alarms are a small rate, so the ROC curve still looks good. Precision divides by the number of alerts, so those same false alarms drag it down. That is exactly what the people reviewing alerts experience.

See it on a fraud-like example

About 1% of these 20,000 transactions are positive. A logistic regression model is compared with random guessing:

import numpy as np
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score, average_precision_score

# Fraud-like data: about 1% of transactions are positive.
X, y = make_classification(n_samples=20_000, n_features=10, n_informative=4, weights=[0.99],
                           class_sep=1.0, flip_y=0, random_state=1)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.5, stratify=y, random_state=1)
scores = LogisticRegression(max_iter=1000).fit(X_train, y_train).predict_proba(X_test)[:, 1]
random_scores = np.random.default_rng(0).random(len(y_test))

print(f"positives in test set: {y_test.mean():.1%}")
print(f"{'':14}{'ROC AUC':>8}{'PR AUC':>8}")
print(f"{'model':14}{roc_auc_score(y_test, scores):>8.3f}{average_precision_score(y_test, scores):>8.3f}")
print(f"{'random guess':14}{roc_auc_score(y_test, random_scores):>8.3f}{average_precision_score(y_test, random_scores):>8.3f}")

# What the PR number means in practice: to catch half of the fraud, how many alerts are real?
from sklearn.metrics import precision_recall_curve
precision, recall, _ = precision_recall_curve(y_test, scores)
p_at_half = precision[recall >= 0.5].max()
print(f"catching 50% of fraud: {p_at_half:.0%} of alerts are real fraud")

positives in test set: 1.0%
               ROC AUC  PR AUC
model            0.898   0.054
random guess     0.464   0.010
catching 50% of fraud: 7% of alerts are real fraud

• A ROC AUC of 0.90 sounds strong.
• The PR AUC of 0.054 tells a different story. It is about five times better than random (0.010), but far from useful on its own.
• The last line makes it concrete: to catch half the fraud, 93% of alerts would be false alarms. A fraud team would reject this model, and only PR AUC warned you.

(The random guess scores 0.46 instead of exactly 0.5 because the test set has only about 100 positives, so chance varies.)

Side by side

	ROC AUC	PR AUC
Curve	True positive rate vs false positive rate	Precision vs recall
Random baseline	0.5, always	Share of positives (for example 0.01)
Sensitive to class imbalance	Hardly: can look good on rare classes	Yes: reflects how noisy alerts are
Focus	Ranking of positives vs negatives overall	Quality of the top-ranked predictions
Best for	Balanced classes; comparing models where both kinds of error matter	Rare positives where false alarms are costly
Easy to explain	“Chance a positive ranks above a negative”	“Average precision as you catch more positives”

Which should you report?

• Balanced data, or both errors matter similarly: ROC AUC is fine and easy to compare across datasets.
• Rare positives: report PR AUC, and always state the positive rate so readers know the baseline.
• Before going live: neither is enough. Choose a threshold and report precision and recall at that threshold, in business terms: “catches 60% of fraud, 1 in 4 alerts is real”. The evaluation metrics lesson covers threshold choice.

Common mistakes

1. Celebrating a high ROC AUC on a 1% positive class without checking precision.
2. Comparing PR AUC across datasets with different positive rates: the baselines differ.
3. Reporting an area but never a threshold. Users experience one threshold, not the curve.
4. Using accuracy on imbalanced data, where predicting “never fraud” scores 99%.

Interview angle

For “ROC AUC vs PR AUC”:

• What each curve plots, and each random baseline.
• Why ROC AUC stays high on imbalanced data (the false positive rate’s denominator is the huge negative class).
• When you would use each, and that you would also report precision and recall at the chosen threshold.

Practise with the machine learning interview questions and the data science interview questions.

Key takeaway

ROC AUC measures overall ranking and always has a 0.5 random baseline, so it can look good even when a rare-class model raises mostly false alarms. PR AUC focuses on precision as recall grows and has a baseline equal to the positive rate, so it shows how useful the alerts really are. Use PR AUC for rare positives, ROC AUC for balanced problems, and always finish with precision and recall at a real threshold.