Decision Trees, Random Forests and Gradient Boosting

Tree-based models are the most widely used models for tabular business data. They capture non-linear patterns and interactions between features without manual feature engineering, and need no feature scaling.

How they work

• A decision tree asks a series of yes/no questions about features, such as “days since login above 20?”, choosing at each step the split that best separates the outcomes. Each leaf gives a prediction. Easy to explain, but a deep tree memorises its training data.
• A random forest trains many trees, each on a random sample of rows and features, and averages them. The averaging cancels much of each tree’s overfitting.
• Gradient boosting builds trees one after another, each correcting the errors of the trees before it. It is often the most accurate model for tabular data. XGBoost, LightGBM, CatBoost and scikit-learn’s HistGradientBoostingClassifier are common implementations.

Compare them on the same data

The dataset used in every lesson of this course is a made-up churn table. Run this block first:

import numpy as np
import pandas as pd

def make_customers(n=5000, seed=7):
    """A made-up churn dataset: who cancels their subscription in the next 60 days?"""
    rng = np.random.default_rng(seed)
    tenure = rng.integers(1, 60, n)                       # months as a customer
    spend = np.round(rng.normal(40, 15, n).clip(5), 2)    # monthly spend
    tickets = rng.poisson(1.2, n)                         # support tickets last quarter
    last_login = rng.integers(0, 45, n)                   # days since last login
    logit = -1.2 - 0.05 * tenure + 0.35 * tickets + 0.06 * last_login - 0.01 * spend
    churned = rng.random(n) < 1 / (1 + np.exp(-logit))
    return pd.DataFrame({"tenure_months": tenure, "monthly_spend": spend, "support_tickets": tickets,
                         "days_since_login": last_login, "churned": churned.astype(int)})

Then train five models and compare their ROC AUC (a ranking score from 0.5, random, to 1, perfect; see evaluation metrics) on training and test data:

from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import RandomForestClassifier, HistGradientBoostingClassifier
from sklearn.metrics import roc_auc_score

df = make_customers()
X, y = df.drop(columns="churned"), df["churned"]
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=0, stratify=y)

models = {
    "logistic regression":      LogisticRegression(max_iter=1000),
    "decision tree (no limit)": DecisionTreeClassifier(random_state=0),
    "decision tree (depth 4)":  DecisionTreeClassifier(max_depth=4, random_state=0),
    "random forest":            RandomForestClassifier(n_estimators=300, min_samples_leaf=20, random_state=0),
    "gradient boosting":        HistGradientBoostingClassifier(max_depth=3, learning_rate=0.05, random_state=0),
}
for name, m in models.items():
    m.fit(X_train, y_train)
    train_auc = roc_auc_score(y_train, m.predict_proba(X_train)[:, 1])
    test_auc = roc_auc_score(y_test, m.predict_proba(X_test)[:, 1])
    print(f"{name:25} train AUC {train_auc:.3f}   test AUC {test_auc:.3f}")

logistic regression       train AUC 0.788   test AUC 0.832
decision tree (no limit)  train AUC 1.000   test AUC 0.592
decision tree (depth 4)   train AUC 0.774   test AUC 0.808
random forest             train AUC 0.843   test AUC 0.818
gradient boosting         train AUC 0.812   test AUC 0.824

Three lessons from this table:

• The unlimited tree is perfect on training data (1.000) and poor on new data (0.592). That gap is overfitting: it memorised individual customers instead of learning general patterns.
• Limiting depth, or averaging many trees, fixes most of it. The depth-4 tree, the forest and the boosted model all generalise well.
• The simple logistic regression scores best here, because this made-up data was generated from a mostly linear pattern. On real data with interactions and non-linear effects, boosted trees often win. Always compare on your own data instead of assuming the complex model is better.

Choosing a model

• Start with logistic regression as a baseline.
• Try gradient boosting, tuned with early stopping, and a random forest.
• Prefer the simpler model when scores are close, because it is easier to explain, maintain and monitor.

Common mistakes

1. Deep single trees without limits.
2. Judging by training scores instead of test or cross-validation scores.
3. Tuning boosting without early stopping, which leads to overfitting.
4. Assuming the most complex model wins.

Key takeaway

Decision trees split data with simple questions but overfit when deep; random forests average many trees, and gradient boosting builds them in sequence to correct errors. Compare training and test scores to spot overfitting, and keep the simpler model when results are close. Next: measuring models properly with evaluation metrics.