What Is Data Orchestration? Orchestration vs Scheduling vs ETL

Data orchestration is the job of running the steps of data pipelines in the right order, at the right time, and dealing with whatever goes wrong. An orchestrator is the tool that does it. It knows that the revenue table can only be built after orders and payments have loaded, retries the payments load when the API times out, stops the dashboard refresh if the revenue build fails, and shows you all of this on one screen.

A good comparison is the conductor of an orchestra: the conductor does not play any instrument, but makes sure each one comes in at the right moment. An orchestrator usually does not move or transform the data itself. It tells other tools (SQL in a warehouse, a Spark job, a Python script, dbt) when to run, and watches what happens.

Orchestration vs scheduling vs ETL

	Scheduling (cron)	ETL / ELT tool or script	Orchestration
Answers	“When does this run?”	“How is the data moved and changed?”	“In what order do all the steps run, and what happens when one fails?”
Knows dependencies	No, only times	Within its own job	Across all jobs and tools
Retries and alerts	No, you add them yourself	Sometimes, for its own steps	Built in, for every step
History and backfills	No	Rarely	Yes: past runs, reruns for past dates
Example	0 2 * * * load.sh	A Python loader, a dbt project, a Spark job	Airflow, Dagster, Prefect, Step Functions

A common failure story shows the difference. With cron, the load runs at 02:00 and the report at 03:00. One night the load takes 70 minutes, so the report runs on half-loaded data and nobody notices until the morning. An orchestrator starts the report only after the load has succeeded, and alerts someone if it fails.

What an orchestrator handles for you

1. Dependencies. Steps are arranged in a DAG (directed acyclic graph): each task lists what it depends on, and there are no loops.
2. Scheduling and triggers. Runs start on a timetable, when a file arrives, or when another pipeline finishes.
3. Retries. Temporary failures, such as timeouts, are retried with a delay.
4. Failure handling. When a task fails, the tasks that depend on it do not run, and independent branches carry on.
5. Backfills. You can rerun a pipeline for past dates, for example after fixing a bug. This only works safely if tasks are idempotent.
6. Visibility. A UI and logs show what ran, how long it took and what failed.
7. Alerts. Messages to chat or email when a run fails or is late.
8. Parameters and secrets. Run dates, connections and credentials passed to tasks in one controlled way.

See it in code: a tiny orchestrator

This short example has the core ideas: a DAG of tasks, a run order worked out from dependencies, retries for temporary errors, and skipping tasks whose inputs failed. The payments API times out once; the ads API key has expired:

from graphlib import TopologicalSorter

# A tiny orchestrator: tasks, the tasks each one depends on, retries, and skipping what cannot run.
attempts = {}

def extract_orders():   return "orders loaded"
def extract_payments():
    attempts["extract_payments"] = attempts.get("extract_payments", 0) + 1
    if attempts["extract_payments"] < 2:             # the payments API fails once, then works
        raise ConnectionError("payments API timed out")
    return "payments loaded"
def extract_ads():      raise PermissionError("ads API key expired")
def build_revenue():    return "revenue table built"
def build_marketing():  return "marketing table built"
def refresh_dashboard(): return "dashboard refreshed"

tasks = {extract_orders: [], extract_payments: [], extract_ads: [],
         build_revenue: [extract_orders, extract_payments],
         build_marketing: [extract_orders, extract_ads],
         refresh_dashboard: [build_revenue]}

state = {}
for task in TopologicalSorter(tasks).static_order():   # every task comes after the tasks it depends on
    name = task.__name__
    if any(state[up.__name__] != "success" for up in tasks[task]):
        state[name] = "upstream_failed"
        print(f"{name:18} skipped: an upstream task did not succeed")
        continue
    for attempt in range(1, 4):                         # up to 3 tries
        try:
            print(f"{name:18} {task()} (attempt {attempt})")
            state[name] = "success"
            break
        except ConnectionError as e:                    # a temporary error: try again
            print(f"{name:18} attempt {attempt} failed: {e}, retrying")
        except Exception as e:                          # a permanent error: retrying will not help
            print(f"{name:18} failed: {e}")
            state[name] = "failed"
            break
    else:                                               # all tries used up
        state[name] = "failed"

extract_orders     orders loaded (attempt 1)
extract_payments   attempt 1 failed: payments API timed out, retrying
extract_payments   payments loaded (attempt 2)
extract_ads        failed: ads API key expired
build_revenue      revenue table built (attempt 1)
build_marketing    skipped: an upstream task did not succeed
refresh_dashboard  dashboard refreshed (attempt 1)

What happened:

• The payments timeout was retried and the second attempt worked, so nobody had to wake up.
• The expired ads key is not a temporary problem, so retrying would only waste time. The task failed at once.
• The marketing table was skipped because one of its inputs failed. The revenue branch did not depend on ads, so it carried on and the dashboard still refreshed.

Airflow and other orchestrators do the same things at scale. They add a scheduler, a database of past runs, a UI, workers on many machines, and alerts. In Airflow, the dictionary above becomes a DAG, each function a task, the retry loop the retries setting, and upstream_failed is a real task state with the same name. Continue with Airflow DAG fundamentals.

Common orchestrators

• Apache Airflow: the most widely used open-source orchestrator; pipelines are Python code. Offered as managed services by the major clouds and by vendors.
• Dagster and Prefect: newer open-source tools, with a stronger focus on data assets and on developer experience. See Airflow vs Dagster vs Prefect.
• Cloud services: AWS Step Functions, Azure Data Factory pipelines, Google Cloud Composer (managed Airflow) and Workflows.
• Built into platforms: Databricks Jobs (Lakeflow Jobs), dbt Cloud jobs and Snowflake Tasks orchestrate work inside their own platform.

Choose by what your team already runs, how many different tools the pipelines touch, and whether you need open-source portability.

Common mistakes

1. Doing heavy data processing inside the orchestrator’s workers. Let the warehouse or Spark do the work; the orchestrator should start it and watch it.
2. Chaining jobs by time instead of by dependency, as in the cron story above.
3. Retrying every error. Retry timeouts and rate limits; fail fast on bad credentials or bad data.
4. Tasks that are not idempotent, so retries and backfills create duplicates.
5. No alerts, so failures are found by the people reading the dashboard.

What interviewers look for

• A one-sentence definition, and the difference from scheduling and from ETL.
• The list of what an orchestrator handles: dependencies, retries, failure handling, backfills, visibility and alerts.
• What a DAG is and why it cannot contain loops.
• An example of a temporary vs a permanent failure and how you would handle each.

Practise with the data engineering interview questions and the Airflow interview questions.

Key takeaway

Data orchestration runs pipeline steps in dependency order, on a schedule or a trigger, and handles retries, failures, backfills and alerts. Scheduling only decides when something starts; ETL tools decide how data is moved and changed; the orchestrator coordinates all of them. Next: Airflow DAG fundamentals.