Course · Streaming & orchestration
Airflow for Data Engineers
Airflow schedules and orchestrates pipelines as DAGs. Learn scheduling, task dependencies, retries and idempotent task design.
- Lessons
- 8
- Interview questions
- 17
- Projects & case studies
- 7
- Reading time
- ~3 h
Your progress
Saved in this browser onlyCourse structure
- 2 lessonsBeginnerCore concepts you will use every day.
- 5 lessonsIntermediatePatterns used in production pipelines.
- 1 lessonAdvancedPerformance, internals and edge cases.
Practise
- InterviewAirflow interview questionsThe full list with difficulty, type and a box to tick off each one.
- Cheat sheetAirflow Cheat SheetA quick Airflow 3 reference: TaskFlow DAGs, schedules and data intervals, retries, sensor modes, trigger rules, the CLI and what changed from Airflow 2.
- InterviewAll interview questionsEvery question across all topics in one filterable list.
Practice
12 problems in 1 topic, from basic to advanced. 0 of 12 done.
Questions0/12
- How do you handle task failures and alerting in Airflow?MediumDataDank only
- How do you make one Airflow DAG depend on another?MediumDataDank only
- How do you test Airflow DAGs before they reach production?MediumDataDank only
- How should Airflow retries and idempotency work together?MediumDataDank only
- How would you process a variable number of files each day in Airflow?MediumDataDank only
- Poke, reschedule or deferrableMediumDataDank only
- What are the logical date and data interval in Airflow, and what changed in Airflow 3?MediumDataDank only
- What is the difference between catchup and backfill in Airflow 3?MediumDataDank only
- When should Airflow tasks pass data through XCom, and when through external storage?MediumDataDank only
- Airflow is slow and tasks sit in scheduled or queued. How do you find the bottleneck?HardDataDank only
- How do you choose an Airflow executorHardDataDank only
- What changed in Airflow 3, and how would you migrate Airflow 2 DAGs?HardDataDank only
Ticks are saved in this browser only and match the ticks on the interview pages.
Lessons
Work through the lessons in order. Completed lessons show a tick; lessons you have opened are outlined.
Beginner
Core concepts you will use every day.
- Airflow DAG Fundamentals, TaskFlow and Dynamic DAGsBuild Airflow 3 DAGs from first principles: tasks and dependencies, the TaskFlow API, params and Jinja templating, dynamic DAGs and dynamic task mapping.
- Airflow Operators, Hooks, Providers, Branching and Trigger RulesUse BashOperator and PythonOperator well, write custom operators and hooks, pick provider packages, run pods, and control flow with branching and trigger rules.
Intermediate
Patterns used in production pipelines.
- Airflow Sensors and Deferrable OperatorsAirflow sensors and deferrable operators: sensor modes, timeouts, ExternalTaskSensor, FileSensor and triggers that wait without tying up workers.
- Airflow Scheduling: Intervals, Logical Dates, Catchup, Timetables and AssetsAirflow 3 scheduling: cron and presets, logical dates and data intervals, catchup pitfalls, timetables, asset-aware scheduling and deadline alerts.
- Airflow XCom, Variables, Connections and SecretsAirflow XCom, Variables and Connections: pass small data between tasks, avoid large payloads with custom backends, and manage secrets safely.
- Operating Airflow: Backfills, Retries, Alerting, Logging and the REST APIRun Airflow day to day: backfill history, clear tasks safely, set retries and alerts, configure logs and StatsD metrics, and use the REST API v2.
- Airflow Best Practices: Idempotent Tasks, Testing and CI/CDAirflow best practices: write DAGs that are safe to rerun and cheap to parse, test with DagBag, unit tests and dag.test(), and ship through CI/CD.
Projects and case studies
Apply what you learned and prepare material to discuss in interviews.
Projects
System design case studies
- AdvancedDesign a Backfill and Late-Data Handling SystemDesign the platform capabilities that let a data team handle late-arriving data automatically in daily and hourly pipelines, and run large backfills (after bug fixes, new columns or new pipelines) across months of history and dozens of dependent tables, safely, cheaply and without disturbing production loads or consumers.
- AdvancedDesign a Data Catalog and Lineage SystemAnalysts at a large company spend days finding the right table, nobody knows who owns half the datasets, and engineers cannot tell what will break if they change a column. Design a data catalog and lineage system that automatically harvests metadata from warehouses, lakehouses, pipelines and BI tools, makes data discoverable and trustworthy, shows table- and column-level lineage, and supports governance workflows such as ownership, classification and access requests.
- AdvancedDesign a Data Observability SystemA data platform runs 3,000 tables across a warehouse and a lakehouse, fed by Airflow, dbt, Spark and streaming jobs. Problems are usually found by business users hours later. Design a data observability system that monitors pipelines and data automatically, detects freshness, volume, schema and distribution anomalies, finds the likely root cause through lineage, and drives incidents to resolution against defined SLAs.
- AdvancedDesign a Data SLA and Freshness Monitoring SystemDesign a system that tells a company, for each of its important datasets, whether the data is fresh, complete and correct enough to use right now; alerts the right owner before consumers notice a problem; shows which downstream dashboards and models are affected; and reports SLA attainment over time.
- AdvancedDesign an Idempotent Reprocessing SystemA bug in the revenue logic went unnoticed for three weeks; a source re-sent a month of corrected data; a new metric needs two years of history. Today each of these takes a week of manual work and risks duplicates. Design a reprocessing system that lets engineers re-run any pipeline over any range of history, safely and repeatably, while daily runs continue, and that propagates corrections downstream.
- IntermediateDesign a Batch Ingestion FrameworkA data team writes a new pipeline by hand for every source, and now runs 150 slightly different jobs pulling from databases, SFTP drops, object storage and REST APIs. Design a reusable, metadata-driven batch ingestion framework that onboards a new source through configuration, lands data reliably and idempotently in the lakehouse, and is easy to operate, backfill and monitor.
Resources
Cheat sheets
Related courses
- PythonPython glues pipelines together: ingestion, validation, orchestration and PySpark jobs. Focus on functions, generators, error handling and testable code.
- KafkaKafka is a distributed log used for streaming data. Learn topics, partitions, consumer groups and delivery semantics before building streaming pipelines.
- Apache SparkHow Spark works inside: driver and executors, RDDs, jobs and stages, shuffles and skew, caching, Catalyst, AQE, memory tuning, Kubernetes, History Server.