Overfitting, Cross-Validation and Data Leakage

Two problems make machine learning models look better on paper than in real use. Overfitting is a model learning its training data too closely. Data leakage is a model using information it would not have when making real predictions. Both are common, and both are easy to miss if you only look at one good-looking score.

Overfitting

An overfitted model memorises noise in the training data. The sign is a big gap between training and test scores, as with the unlimited decision tree in trees, forests and boosting (training AUC 1.000, test AUC 0.592). Fixes: simpler models, limits such as tree depth, regularisation, more data, and early stopping.

Cross-validation

A single train-test split can be lucky or unlucky. K-fold cross-validation splits the training data into k parts, trains on k minus 1 of them and tests on the remaining one, k times, then averages the scores. It gives a more reliable estimate and shows how much scores vary. Use it to compare models and tune settings; keep a final test set for the end.

Data leakage

Leakage happens when a feature contains information about the answer that would not be available at prediction time. Typical causes:

• Features recorded after the event, such as “visited the cancellation page” when predicting cancellation.
• Random splits on time-based data, so the model trains on the future.
• Preprocessing fitted on all data, such as scaling or encoding before splitting.
• The same customer in training and test data through duplicate or related rows.

See it happen

The dataset used in every lesson of this course is a made-up churn table. Run this block first:

import numpy as np
import pandas as pd

def make_customers(n=5000, seed=7):
    """A made-up churn dataset: who cancels their subscription in the next 60 days?"""
    rng = np.random.default_rng(seed)
    tenure = rng.integers(1, 60, n)                       # months as a customer
    spend = np.round(rng.normal(40, 15, n).clip(5), 2)    # monthly spend
    tickets = rng.poisson(1.2, n)                         # support tickets last quarter
    last_login = rng.integers(0, 45, n)                   # days since last login
    logit = -1.2 - 0.05 * tenure + 0.35 * tickets + 0.06 * last_login - 0.01 * spend
    churned = rng.random(n) < 1 / (1 + np.exp(-logit))
    return pd.DataFrame({"tenure_months": tenure, "monthly_spend": spend, "support_tickets": tickets,
                         "days_since_login": last_login, "churned": churned.astype(int)})

Then add a tempting feature, visits to the cancellation page, which is recorded after the prediction date:

from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score
import numpy as np

df = make_customers()
# A tempting but leaky feature: visits to the "cancel subscription" page, recorded AFTER the prediction date.
rng = np.random.default_rng(1)
df["cancel_page_visits"] = df["churned"] * rng.integers(1, 4, len(df)) + (rng.random(len(df)) < 0.05)

y = df["churned"]
for cols, label in [(["tenure_months", "monthly_spend", "support_tickets", "days_since_login"], "honest features"),
                    (["tenure_months", "monthly_spend", "support_tickets", "days_since_login", "cancel_page_visits"],
                     "with leaky feature")]:
    X = df[cols]
    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=0, stratify=y)
    m = LogisticRegression(max_iter=1000).fit(X_train, y_train)
    cv = cross_val_score(LogisticRegression(max_iter=1000), X_train, y_train, cv=5, scoring="roc_auc")
    print(f"{label:19} test AUC {roc_auc_score(y_test, m.predict_proba(X_test)[:, 1]):.3f}   "
          f"5-fold CV AUC {cv.mean():.3f} (+/- {cv.std():.3f})")

honest features     test AUC 0.832   5-fold CV AUC 0.787 (+/- 0.014)
with leaky feature  test AUC 0.997   5-fold CV AUC 0.997 (+/- 0.001)

Two lessons:

• The leaky feature makes the model look almost perfect (0.997). In real use, on the prediction date, nobody has visited the cancellation page yet, so the model would perform far worse than the score suggests. A suspiciously good score is the most common sign of leakage; check the most important features first.
• The honest model’s single test split (0.832) is more optimistic than cross-validation (0.787). One lucky split can mislead; cross-validation gives the steadier estimate to report.

How to prevent leakage

1. For every feature, ask: “Would I know this at the moment of prediction?”
2. Build features as of a prediction date, and split data by time for time-based problems.
3. Put preprocessing and the model in one pipeline fitted on training data only.
4. Remove duplicates and related rows across the split.
5. Be suspicious of near-perfect scores.

Key takeaway

Overfitting shows as a gap between training and test scores; data leakage shows as scores that are too good to be true. Use cross-validation for reliable estimates, build features only from information available at prediction time, and investigate any surprisingly strong result before trusting it. That completes the Machine Learning course; test yourself with the machine learning interview questions.