What is a Backfill?

How it works

A pipeline that is parameterised by its data interval and overwrites (or merges) its own slice can be backfilled by simply running it once per past interval. That is the whole trick: the backfill is the production code path, not a second, hand-written script that drifts from it.

Before starting, check that the raw data for the range still exists, that the source or lake copy can take the read load, and which downstream tables will need to pick up the corrected rows. Downstream incremental models only process rows that look new to them, so they often need their own backfill.

Run it in chunks, typically one day or one partition per run, with a concurrency limit so the backfill does not starve the regular schedule. Order matters: newest first gives consumers the most relevant corrections sooner, while oldest first is required when each day depends on the previous one.

Orchestrators help. In Airflow 3, backfills are created through the CLI, UI or REST API and run by the scheduler, with options for a dry run, a concurrency limit and whether to rerun dates that already succeeded. For a large logic change, write to a shadow table, compare it with production partition by partition, then swap, and tell consumers which dates will change.

Example

A chunked backfill plan in plain Python: each day runs the same idempotent job, a failure affects only that day, and the failed day can be rerun later without touching the others.

from datetime import date, timedelta

def days(start: date, end: date):
    d = start
    while d <= end:
        yield d
        d += timedelta(days=1)

def load_day(day: date) -> None:
    """Stand-in for the production job: overwrite one partition."""
    if day == date(2026, 9, 3):
        raise RuntimeError("source file missing")

results = {}
for day in days(date(2026, 9, 1), date(2026, 9, 5)):
    try:
        load_day(day)
        results[day.isoformat()] = "ok"
    except RuntimeError as exc:
        results[day.isoformat()] = f"failed: {exc}"

for day, status in results.items():
    print(day, status)

2026-09-01 ok
2026-09-02 ok
2026-09-03 failed: source file missing
2026-09-04 ok
2026-09-05 ok

The equivalent request in Airflow 3 (not run here, it needs a scheduler) is airflow backfill create --dag-id orders_daily --from-date 2026-09-01 --to-date 2026-09-05 --max-active-runs 2.

Common mistakes

• Backfilling a job that appends, which duplicates every day in the range.
• Resetting the incremental watermark to the start of the backfill, so the next daily run reprocesses or skips data.
• Unlimited parallelism that overloads the source database or the warehouse.
• Rerunning old dates with today’s dimension values when the report should use the values that were true at the time.