Model Serving: Batch vs Online Predictions

A trained model is only useful when its predictions reach the people or systems that act on them. There are two main ways to serve predictions. Batch scores many records on a schedule and stores the results. Online scores one request at a time, on demand, usually through an API. Choosing the right one saves a lot of complexity.

Batch vs online

	Batch	Online
When predictions are made	On a schedule, such as nightly	When a request arrives
Speed needed	Minutes or hours for the whole run	Milliseconds per request
Typical use	Churn scores, demand forecasts, lead scoring	Fraud checks at payment, recommendations on a page
Complexity	Low: a scheduled job writing a table	Higher: a service, scaling, fresh features
Uses latest data	As of the last run	Yes, at request time

Choose batch unless the decision needs data from the last few seconds or minutes. Many teams start with batch and add online serving only for the decisions that need it.

Both in one example

This example scores 100,000 customers in a batch, then serves the same model behind a small HTTP API that validates its input:

import json, threading
from http.server import BaseHTTPRequestHandler, HTTPServer
from urllib.request import Request, urlopen
import numpy as np
from sklearn.linear_model import LogisticRegression

rng = np.random.default_rng(3)
X = rng.normal(size=(2000, 4)); y = (X @ [0.8, -0.5, 0.3, 0.0] + rng.normal(0, 1, 2000) > 0).astype(int)
model = LogisticRegression().fit(X, y)
FEATURES = ["tenure", "spend", "tickets", "logins"]

# Batch: score every customer once a night and save the results for other systems to read
customers = rng.normal(size=(100_000, 4))
scores = model.predict_proba(customers)[:, 1]
print(f"batch: scored {len(scores):,} customers; "
      f"{(scores > 0.7).sum():,} above 0.7")

# Online: a small HTTP service that scores one customer per request
class Score(BaseHTTPRequestHandler):
    def do_POST(self):
        body = json.loads(self.rfile.read(int(self.headers["Content-Length"])))
        missing = [f for f in FEATURES if f not in body]
        if missing:                                         # validate input before the model sees it
            code, out = 400, {"error": f"missing features: {missing}"}
        else:
            p = model.predict_proba([[body[f] for f in FEATURES]])[0, 1]
            code, out = 200, {"churn_probability": round(float(p), 3), "model_version": "v1"}
        self.send_response(code); self.end_headers(); self.wfile.write(json.dumps(out).encode())
    def log_message(self, *a):
        pass

server = HTTPServer(("127.0.0.1", 0), Score)
threading.Thread(target=server.serve_forever, daemon=True).start()
url = f"http://127.0.0.1:{server.server_port}/score"
def call(payload):
    req = Request(url, data=json.dumps(payload).encode(), headers={"Content-Type": "application/json"})
    try:
        return json.load(urlopen(req))
    except Exception as e:
        return json.load(e)
print("online:", call({"tenure": 1.2, "spend": -0.3, "tickets": 0.5, "logins": 0.1}))
print("online:", call({"tenure": 1.2}))
server.shutdown()

batch: scored 100,000 customers; 31,523 above 0.7
online: {'churn_probability': 0.902, 'model_version': 'v1'}
online: {'error': "missing features: ['spend', 'tickets', 'logins']"}

The batch run scores everyone in one go; the results would be written to a table for the retention team. The online service returns a probability and the model version for a valid request, and rejects an incomplete request with a clear error instead of guessing missing values. Production services use a web framework or a model-serving platform, with authentication, logging and scaling, but the shape is the same.

Serving checklist

• Same features in training and serving: share the feature code, or use a feature store, to avoid training-serving skew.
• Validate inputs: types, ranges and missing fields, before the model sees them.
• Return the model version with every prediction, and log both.
• Roll out gradually: shadow mode (score without using the result) or a small share of traffic first.
• Have a fallback: a previous model version or a simple rule if the service fails.
• Set latency budgets and measure them, including feature lookups.

Common mistakes

1. Building an online API for a decision that happens once a day.
2. Recomputing features differently in serving than in training.
3. No input validation, so bad requests produce confident nonsense.
4. No version in the response, so nobody knows which model made a prediction.

Key takeaway

Serve predictions in batch when the decision can wait, and online when it needs fresh data within seconds. Whichever you choose, use the same features as training, validate inputs, return the model version, roll out gradually and keep a fallback. Next: model monitoring and drift.