Model Monitoring and Data Drift: Keeping Models Healthy

A model is trained on a snapshot of the past. As customers, prices, products and behaviour change, the data it sees in production drifts away from what it learned, and its predictions slowly get worse. Unlike a crashed service, a degraded model keeps answering, so problems go unnoticed unless you monitor it.

Two kinds of drift

• Data drift: the inputs change. For example, more new customers than before, or customers logging in less often. The model may still be right about each customer, but it now sees cases it saw rarely in training.
• Concept drift: the relationship between inputs and outcome changes. For example, fraudsters change tactics, so the same patterns no longer mean fraud. This hurts accuracy directly.

Data drift can be measured immediately from inputs. Concept drift usually shows up only when real outcomes arrive, which can take days or months.

Measuring data drift with PSI

The population stability index (PSI) compares a feature’s distribution in production with its distribution in training. It splits the training values into bins, then measures how much the share of values in each bin has moved. Common rules of thumb: below 0.1 is stable, 0.1 to 0.25 is worth watching, above 0.25 is a significant shift.

import numpy as np

def psi(expected, actual, bins=10):
    """Population stability index: how much a feature's distribution moved since training."""
    edges = np.quantile(expected, np.linspace(0, 1, bins + 1))
    edges[0], edges[-1] = -np.inf, np.inf
    e = np.histogram(expected, edges)[0] / len(expected)
    a = np.histogram(actual, edges)[0] / len(actual)
    e, a = np.clip(e, 1e-6, None), np.clip(a, 1e-6, None)
    return float(np.sum((a - e) * np.log(a / e)))

rng = np.random.default_rng(9)
training = {"monthly_spend": rng.normal(40, 15, 20_000), "days_since_login": rng.exponential(10, 20_000)}
this_week = {"monthly_spend": rng.normal(41, 15, 5_000),        # about the same
             "days_since_login": rng.exponential(20, 5_000)}    # customers log in much less often now

for feature in training:
    value = psi(training[feature], this_week[feature])
    status = "OK" if value < 0.1 else "watch" if value < 0.25 else "ALERT: investigate and consider retraining"
    print(f"{feature:17} PSI {value:.3f}  {status}")

monthly_spend     PSI 0.006  OK
days_since_login  PSI 0.382  ALERT: investigate and consider retraining

Spending looks the same as in training, but days since last login has shifted a lot: customers log in much less often than when the model was trained. Since that is one of the churn model’s strongest features, its predictions this week deserve a close look.

What to monitor

Layer	Examples
System	Request volume, latency, errors, failed batch runs
Data quality	Missing values, out-of-range values, schema changes
Data drift	PSI or distribution tests per important feature
Predictions	Share of high-risk predictions, average score over time
Outcomes	Real accuracy, precision and recall once labels arrive
Business	The metric the model is meant to improve

When to retrain

• On a schedule (for example monthly), when data changes steadily and retraining is cheap.
• On evidence: drift alerts, falling accuracy on fresh labels, or a business change.
• Always validate the new model against the current one before promoting it (see experiment tracking and model registry). Sometimes drift means the problem itself changed, and the fix is new features or a new definition, not just retraining.

Common mistakes

1. Monitoring only the service, not the data and predictions.
2. Alerting on every small change, so alerts get ignored. Focus on important features and agreed thresholds.
3. Retraining automatically without validation, which can deploy a worse model.
4. No plan for collecting real outcomes, so accuracy is never measured.

Key takeaway

Models degrade as the world changes. Monitor system health, data quality, input drift (for example with PSI), predictions and real outcomes, alert on what matters, and retrain on a schedule or on evidence, always validating before promotion. That completes the MLOps course; practise explaining it with the machine learning interview questions.