What is a DAG?

How it works

• Directed: every edge points one way, from upstream task to downstream task.
• Acyclic: following the edges never leads back to where you started, so there is always at least one valid run order (a topological order).
• Graph: tasks are nodes, dependencies are edges.

An orchestrator repeatedly finds the tasks whose upstream tasks have all succeeded and runs them, often in parallel. Tasks with no path between them, such as two independent extracts, run at the same time; a join waits for both. If a task fails, everything downstream of it is held back, while unrelated branches carry on.

The same idea appears inside engines: Spark turns a query into a DAG of stages split at shuffles, and dbt builds a DAG of models from ref() calls. In Airflow, a DAG also carries a schedule, and each scheduled run (a DAG run) processes one data interval, which is what makes backfills possible.

Example

Python’s standard library can order a pipeline in waves of tasks that may run in parallel, and it refuses a graph with a cycle (Python 3.11, graphlib).

from graphlib import TopologicalSorter

# Each task maps to the tasks it depends on.
dag = {
    "extract_orders": set(),
    "extract_customers": set(),
    "clean_orders": {"extract_orders"},
    "join": {"clean_orders", "extract_customers"},
    "publish_report": {"join"},
}
ts = TopologicalSorter(dag)
ts.prepare()
wave = 1
while ts.is_active():
    ready = sorted(ts.get_ready())
    print(f"wave {wave}: {ready}")
    ts.done(*ready)
    wave += 1

dag["extract_orders"] = {"publish_report"}  # a cycle: no longer a DAG
try:
    list(TopologicalSorter(dag).static_order())
except Exception as e:
    print(type(e).__name__)

wave 1: ['extract_customers', 'extract_orders']
wave 2: ['clean_orders']
wave 3: ['join']
wave 4: ['publish_report']
CycleError

Common mistakes

• Putting heavy work in the top level of an Airflow DAG file, which runs every time the scheduler parses it.
• Making tasks depend on run order instead of declared dependencies, so parallel runs break.
• Writing tasks that are not idempotent, so retries and backfills create duplicates.
• Chaining many DAGs through fragile time-based waits instead of explicit dependencies such as datasets or sensors.