What is Batch Processing?

How it works

A batch job knows its input before it starts: yesterday’s partition, a set of files, the rows changed in the last hour. Because the input is bounded, the job can read all of it, compute a complete answer and write it in one go. An orchestrator such as Airflow schedules the runs and passes each one its data interval.

The choice between batch and streaming starts from one question: what decision depends on this data, and how fresh must it be? A daily finance report needed by 07:00 is a batch job; streaming would add cost and moving parts for no benefit. A dashboard “refreshed every 15 minutes” is usually served well by frequent batch or micro-batch, which processes a stream in small, regular batches and keeps batch’s simplicity. Only requirements such as scoring a payment before it is approved really need continuous streaming.

Batch has practical strengths:

• Simple reruns: fix a bug, then rerun the affected date range.
• Easier debugging: the input is fixed, so a failing run can be reproduced.
• Lower cost: compute runs only during the window instead of all day.

Its weaknesses are latency and late data. An event that arrives after its day was processed needs the day to be rebuilt, which is why good batch jobs are idempotent per partition.

Example

A daily batch run reads exactly its half-open interval [start, end) in UTC, so a retry or backfill of 5 October always reads the same rows.

CREATE TABLE clicks (user_id int, clicked_at timestamptz);
INSERT INTO clicks VALUES
  (1, '2026-10-04 23:59:59+00'), (1, '2026-10-05 00:00:00+00'),
  (2, '2026-10-05 13:30:00+00'), (3, '2026-10-06 00:00:00+00');

-- Run parameters supplied by the scheduler, never computed from now().
SELECT DATE '2026-10-05'                              AS run_date,
       count(*)                                       AS clicks,
       count(DISTINCT user_id)                        AS users
FROM clicks
WHERE clicked_at >= TIMESTAMPTZ '2026-10-05 00:00+00'
  AND clicked_at <  TIMESTAMPTZ '2026-10-06 00:00+00';

  run_date  | clicks | users
------------+--------+-------
 2026-10-05 |      2 |     2

Common mistakes

• Choosing streaming because it sounds modern when nobody needs the data faster than hourly.
• Windows defined with BETWEEN, so a row at exactly midnight is counted on two days.
• Computing the window from the clock at run time, which makes retries and backfills read the wrong data.
• Appending each run’s output instead of overwriting its partition, so a rerun duplicates the day.