What is Orchestration?

How it works

An orchestrator holds pipeline definitions as DAGs of tasks and runs a loop: when a schedule fires or an event arrives (a file lands, an upstream dataset updates), it creates a run, starts every task whose dependencies have succeeded, and records each task’s state. Its core features are:

• Scheduling: cron-like schedules or data-aware triggers, with a data interval per run so each run processes one slice.
• Dependencies: downstream tasks wait for upstream success; failed branches stop, independent branches carry on.
• Retries and timeouts: transient failures (an API timeout) are retried with a delay; real failures stop the run and alert.
• Backfills and reruns: re-running past intervals, which is safe only when tasks are idempotent.
• Observability: run history, logs, durations and SLAs in one place.

A good orchestrator coordinates work rather than doing it. Tasks trigger Spark jobs, dbt builds, warehouse queries or Databricks jobs, and the orchestrator waits for them. Airflow is task-centric and the most widely used; Dagster is asset-centric (it models the tables a pipeline produces); Prefect focuses on dynamic Python workflows. Cloud options include Azure Data Factory, AWS Step Functions and Databricks Workflows.

Example

A tiny orchestrator: tasks run in dependency order, a flaky API task succeeds on its retry, a failing quality check exhausts its retries, and publish is never run on bad data (Python 3.11).

from graphlib import TopologicalSorter

calls = {"load_api": 0}
def extract_db():  return "ok"
def load_api():                       # fails once, like a flaky API, then succeeds
    calls["load_api"] += 1
    if calls["load_api"] == 1:
        raise TimeoutError("API timed out")
    return "ok"
def transform():   return "ok"
def broken_check(): raise ValueError("row count is 0")
def publish():     return "ok"

tasks = {"extract_db": extract_db, "load_api": load_api, "transform": transform,
         "quality_check": broken_check, "publish": publish}
deps = {"transform": {"extract_db", "load_api"}, "quality_check": {"transform"}, "publish": {"quality_check"}}

state = {}
for name in TopologicalSorter(deps).static_order():
    if any(state.get(d) != "success" for d in deps.get(name, ())):
        state[name] = "upstream_failed"          # never run on top of a failure
    else:
        for attempt in range(1, 3 + 1):          # up to 3 tries
            try:
                tasks[name]()
                state[name] = "success"
                break
            except Exception as e:
                state[name] = "failed"
                print(f"{name}: attempt {attempt} failed ({e})")
    print(f"{name:<14} {state[name]}")

extract_db     success
load_api: attempt 1 failed (API timed out)
load_api       success
transform      success
quality_check: attempt 1 failed (row count is 0)
quality_check: attempt 2 failed (row count is 0)
quality_check: attempt 3 failed (row count is 0)
quality_check  failed
publish        upstream_failed

Common mistakes

• Processing large data inside the orchestrator’s workers instead of pushing the work to Spark, dbt or the warehouse.
• Retrying tasks that are not idempotent, so every retry adds duplicate rows.
• Retrying deterministic failures, such as a failed data-quality check, which only delays the alert.
• Using wall-clock time (“today”) inside tasks instead of the run’s data interval, which breaks backfills.