The Machine Learning Workflow: From Data to a Working Model

Machine learning builds a model from past examples so it can make predictions on new cases. In supervised learning, each past example has a known answer, such as whether a customer cancelled, and the model learns which patterns lead to which answer. Most data science work follows the same workflow, whatever the algorithm.

The workflow

1. Define the problem: what exactly will be predicted, when, and what decision it supports. “Which customers will cancel in the next 60 days, so the retention team can contact them?”
2. Get and prepare the data: one row per example, with features known at prediction time and the target.
3. Split the data: train the model on one part, and test it on a part it has never seen.
4. Start with a baseline: the simplest possible prediction, to know what “better” means.
5. Train models and compare them on the same test data.
6. Evaluate with metrics that match the decision, not just accuracy.
7. Deploy and monitor: use the predictions, measure the real effect, and retrain as data changes.

A first model

The dataset used in every lesson of this course is a made-up churn table. Run this block first:

import numpy as np
import pandas as pd

def make_customers(n=5000, seed=7):
    """A made-up churn dataset: who cancels their subscription in the next 60 days?"""
    rng = np.random.default_rng(seed)
    tenure = rng.integers(1, 60, n)                       # months as a customer
    spend = np.round(rng.normal(40, 15, n).clip(5), 2)    # monthly spend
    tickets = rng.poisson(1.2, n)                         # support tickets last quarter
    last_login = rng.integers(0, 45, n)                   # days since last login
    logit = -1.2 - 0.05 * tenure + 0.35 * tickets + 0.06 * last_login - 0.01 * spend
    churned = rng.random(n) < 1 / (1 + np.exp(-logit))
    return pd.DataFrame({"tenure_months": tenure, "monthly_spend": spend, "support_tickets": tickets,
                         "days_since_login": last_login, "churned": churned.astype(int)})

Now split the data, train a baseline and a model, and compare them:

from sklearn.model_selection import train_test_split
from sklearn.dummy import DummyClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score, recall_score

df = make_customers()
X, y = df.drop(columns="churned"), df["churned"]
print(f"{len(df)} customers, churn rate {y.mean():.1%}")

# Hold back 25% of customers as a test set the model never sees during training
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=0, stratify=y)

baseline = DummyClassifier(strategy="most_frequent").fit(X_train, y_train)
model = LogisticRegression(max_iter=1000).fit(X_train, y_train)

for name, m in [("baseline (always 'stays')", baseline), ("logistic regression", model)]:
    pred = m.predict(X_test)
    print(f"{name:26} accuracy {accuracy_score(y_test, pred):.3f}   "
          f"churners caught {recall_score(y_test, pred, zero_division=0):.0%}")

5000 customers, churn rate 24.7%
baseline (always 'stays')  accuracy 0.753   churners caught 0%
logistic regression        accuracy 0.802   churners caught 36%

The baseline that predicts “stays” for everyone is already 75% accurate, because three quarters of customers stay. But it catches no churners, so it is useless for the retention team. The logistic regression model is only a few points more accurate, yet it catches 36% of churners. This is why you always compare against a baseline, and why accuracy alone can hide a useless model. Later lessons improve on both the model and the way it is measured.

Why the test set matters

A model can memorise its training data. Testing on separate data shows how it performs on customers it has not seen, which is what matters in practice. Keep the test set aside until the end; if you keep tuning against it, it stops being a fair test. The stratify=y option keeps the churn rate the same in both parts.

Common mistakes

1. No clear decision: a model nobody acts on has no value.
2. No baseline, so a weak model looks good.
3. Testing on training data, which always looks better than reality.
4. Using features that are not known at prediction time (see overfitting and data leakage).

Key takeaway

Define the prediction and the decision, split the data, compare against a baseline, train models and evaluate them on unseen data with metrics that match the decision. Next: the two classic starting models, linear and logistic regression.