Batch vs Streaming Data Pipelines

Batch pipelines process a bounded set of data on a schedule: every night, every hour. Streaming pipelines process an unbounded flow of events continuously, with results updated within seconds or minutes. Between them sits micro-batch, which processes a stream in small, frequent batches. The choice is one of the first decisions in any pipeline design, and in interviews it is a reliable way to show judgement.

Compare them on what matters

	Batch	Streaming
Latency	Minutes to hours	Seconds to minutes
Complexity	Lower: bounded input, simple reruns	Higher: state, late data, ordering, checkpoints
Correctness with late data	Rerun the affected period	Watermarks and update logic required
Reprocessing history	Natural (rerun a date range)	Possible, but needs a replayable source and care
Cost	Compute runs only during the window	Compute runs continuously
Debugging	Easier: inspect a fixed input	Harder: the input keeps changing
Failure impact	A late report	Growing lag; recovery from checkpoints

Start from the latency requirement

Ask: what decision depends on this data, and how fresh must it be for that decision?

• A daily finance report must be correct by 07:00: batch. Streaming would add cost and complexity for no benefit.
• Card fraud must be scored before a payment is approved: streaming, with latency in the tens or hundreds of milliseconds (usually in a dedicated service rather than an analytics pipeline).
• A dashboard “refreshed every 15 minutes” is usually served well by frequent batch or micro-batch, which keeps batch’s simplicity.
• Inventory sync between a warehouse system and a web shop, where overselling costs money: streaming or CDC with seconds of latency.

Ask too what happens if data is late: if the answer is “someone looks at the dashboard a bit later”, batch is fine. Streaming is not automatically “better”; it trades simplicity and cost for latency, so it should be justified by a requirement someone will pay for.

Event time vs processing time

A streaming system sees events in processing time (when they arrive), but most questions are about event time (when they happened). Mobile apps buffer events while offline, producers retry, and partitions are consumed at different speeds, so events arrive late and out of order.

The simulation below counts clicks per one-minute window. Event e4 happened at 10:00:50 but arrived after events from 10:01.

from collections import Counter

events = [  # (event_id, event_time, arrival_time) as "HH:MM:SS" strings
    ("e1", "10:00:10", "10:00:11"),
    ("e2", "10:00:40", "10:00:41"),
    ("e3", "10:01:05", "10:01:06"),
    ("e4", "10:00:50", "10:01:20"),   # late: happened in the 10:00 window
    ("e5", "10:01:30", "10:01:31"),
    ("e6", "10:00:30", "10:03:10"),   # very late
]

minute = lambda ts: ts[:5]
by_processing_time = Counter(minute(arrival) for _, _, arrival in events)
by_event_time = Counter(minute(event_time) for _, event_time, _ in events)
print("processing time:", dict(sorted(by_processing_time.items())))
print("event time:     ", dict(sorted(by_event_time.items())))

processing time: {'10:00': 2, '10:01': 3, '10:03': 1}
event time:      {'10:00': 4, '10:01': 2}

Counting by arrival puts clicks in the wrong minute and invents activity at 10:03. Counting by event time is correct, but raises a new question: when is the 10:00 window finished? If you wait forever for late events, you can never emit a final result or free the window’s memory.

Watermarks

A watermark is the engine’s estimate of how far event time has progressed: “I do not expect events older than T any more”. A common rule is watermark = latest event time seen minus an allowed lateness. When the watermark passes the end of a window, the window is finalised and its state can be dropped. Events that arrive for a window after that are too late: dropped, sent to a side output, or handled by a batch correction.

from datetime import datetime, timedelta

def to_dt(ts):
    return datetime.strptime(ts, "%H:%M:%S")

def windowed_counts(events, allowed_lateness=timedelta(seconds=30), window=timedelta(minutes=1)):
    open_windows, finalised, too_late = Counter(), {}, []
    max_event_time = None
    for event_id, event_time, _arrival in sorted(events, key=lambda e: e[2]):   # process in arrival order
        et = to_dt(event_time)
        max_event_time = et if max_event_time is None else max(max_event_time, et)
        watermark = max_event_time - allowed_lateness
        start = et.replace(second=0)
        if start + window <= watermark and start.strftime("%H:%M") in finalised:
            too_late.append(event_id)
            continue
        open_windows[start] += 1
        for w in [w for w in open_windows if w + window <= watermark]:          # close finished windows
            finalised[w.strftime("%H:%M")] = open_windows.pop(w)
    finalised.update({w.strftime("%H:%M"): c for w, c in open_windows.items()})  # end of input
    return finalised, too_late

print(windowed_counts(events))

({'10:00': 3, '10:01': 2}, ['e6'])

With 30 seconds of allowed lateness, e4 (15 seconds behind the newest event time when it arrives) still counts for 10:00, but e6 arrives after the 10:00 window was finalised and is reported as too late. More allowed lateness means more correct counts, but results are emitted later and the engine holds more state. That trade-off is the core tuning decision in event-time streaming.

In Spark Structured Streaming the same idea is one call:

from pyspark.sql import SparkSession, functions as F

spark = SparkSession.builder.appName("clicks").getOrCreate()
clicks = (spark.readStream.format("rate").option("rowsPerSecond", 5).load()
          .withColumnRenamed("timestamp", "event_time"))

per_minute = (clicks
    .withWatermark("event_time", "30 seconds")                 # allowed lateness
    .groupBy(F.window("event_time", "1 minute"))
    .count())

query = (per_minute.writeStream
    .outputMode("append")                                       # emit each window once, when final
    .format("console")
    .option("checkpointLocation", "/tmp/checkpoints/clicks")    # state and progress survive restarts
    .trigger(processingTime="10 seconds")                       # micro-batch interval
    .start())

Flink and Kafka Streams use the same concepts with different APIs (Flink calls the per-source rule a watermark strategy; Kafka Streams uses a grace period on windows).

Other things a streaming job must handle

• State and checkpoints. Windows, joins and deduplication keep state. Checkpoints store state and source offsets so a restarted job continues without losing or double-counting data. A changed query can be incompatible with an old checkpoint, which makes upgrades a planned operation.
• Delivery semantics. Most pipelines are at-least-once end to end; make sinks idempotent or use transactional sinks. See Kafka delivery semantics.
• Backpressure and lag. When input outpaces processing, lag grows. Monitor consumer lag in time, not just in messages.
• Reprocessing. To fix a bug you replay the source from an earlier offset, which needs enough retention in Kafka or a copy of the raw events in the lake.
• Always-on operations. Someone owns the job at 03:00 on Sunday.

The micro-batch middle ground

Spark Structured Streaming runs by default as a sequence of small batches. You get continuous ingestion with batch-style processing, and the trigger interval tunes latency against cost: every 10 seconds, every 5 minutes, or availableNow, which processes everything new and then stops. availableNow is a useful pattern: the job is written as a stream (incremental, checkpointed, exactly-once into Delta Lake) but scheduled like a batch job, so you pay only while it runs and can move to continuous running later without rewriting it.

Warehouses offer similar options: Snowpipe and Snowpipe Streaming for continuous loading into Snowflake, and streaming ingestion into BigQuery, often combined with scheduled transformations.

Lambda and kappa

• Lambda architecture runs a batch layer and a streaming layer in parallel and merges them: the stream gives fresh approximate results, the batch recomputes accurate ones. It works, but the logic exists twice and the two versions drift.
• Kappa architecture uses a single streaming pipeline and replays the log to reprocess history. It is simpler if the log (or a lake copy of it) retains enough history.

Most modern designs avoid maintaining logic twice: one code path, run continuously or in micro-batches, with batch reprocessing from the raw layer when needed. The lambda and kappa case study goes deeper.

Common mistakes

1. Choosing streaming without a latency requirement that justifies its cost.
2. Aggregating by processing time and reporting wrong counts when events arrive late.
3. No watermark, so state grows until the job runs out of memory.
4. Ignoring replay: if the source cannot be replayed, recovering from a bug is hard.
5. Running a streaming job with no monitoring of lag, and no owner.
6. Changing a stateful query and restarting it on an incompatible checkpoint.

Interview angle

“Batch or streaming for this use case?” appears in most system-design rounds. Start from the freshness requirement and what late data costs, propose the simplest option that meets it (often micro-batch), and then show you know what streaming brings: event time, watermarks and late data, state and checkpoints, at-least-once delivery and idempotent sinks, and the cost of always-on compute. Practise with batch or streaming: how do you choose?.

Key takeaway

Choose by the freshness the decision needs. Batch is simpler and cheaper; streaming is justified when decisions depend on near-real-time data, and it requires deliberate handling of event time, watermarks, state and duplicates. Micro-batch often gives most of the freshness with most of batch’s simplicity.