Course · Languages & query
Python for Data Engineers
Python glues pipelines together: ingestion, validation, orchestration and PySpark jobs. Focus on functions, generators, error handling and testable code.
- Lessons
- 10
- Interview questions
- 11
- Projects & case studies
- 4
- Reading time
- ~1 h
Your progress
Saved in this browser onlyCourse structure
- 1 lessonStart hereThe complete overview of the course in one read.
- 3 lessonsBeginnerCore concepts you will use every day.
- 6 lessonsIntermediatePatterns used in production pipelines.
Practise
- InterviewPython interview questionsThe full list with difficulty, type and a box to tick off each one.
- Cheat sheetPython for Data Engineers Cheat SheetA quick Python reference for pipelines: files and CSV, JSON, dates, collections, generators and batching, error handling, logging and testing with pytest.
- InterviewAll interview questionsEvery question across all topics in one filterable list.
Lessons
Work through the lessons in order. Completed lessons show a tick; lessons you have opened are outlined.
Start here
The complete overview of the course in one read.
Beginner
Core concepts you will use every day.
- Python Functions, Modules and Reusable Data Pipeline CodeStructure Python pipeline code as small pure functions, clear modules and a thin entry point so it is testable, reusable and safe to run from any scheduler.
- Python Data Structures for Data Engineering InterviewsChoose between list, tuple, set, dict, deque and Counter by their operations and costs, with the patterns that come up in data engineering coding interviews.
- Python File I/O for Data Engineers: CSV, JSON, JSON Lines and GzipPython file I/O for pipelines: read and write CSV and JSON safely, stream JSON Lines and gzip files, handle encodings, quoting and Decimals, and write files atomically.
Intermediate
Patterns used in production pipelines.
- Python Iterators, Generators and Memory-Efficient ProcessingUse Python iterators and generators to process files and API pages lazily, in batches, with flat memory use, and avoid the mistakes that silently exhaust them.
- Python Error Handling and Logging for Data PipelinesPython error handling and logging for pipelines: catch specific exceptions, add context with raise from, clean up with context managers, and log with run ids and tracebacks.
- Working with APIs in Python: Pagination, Retries and Rate LimitsPython API ingestion for data pipelines: requests with timeouts, cursor pagination as a generator, retries with backoff and Retry-After, rate limits and incremental pulls.
- pandas for Data Engineers: Essentials and When Not to Use Itpandas for data engineers: read with explicit dtypes, clean, deduplicate, join safely with validate, aggregate and write Parquet, plus when DuckDB, Polars or Spark fit better.
- Testing Python Data Pipelines with pytestpytest for data pipelines: test transforms with parametrize, fixtures and tmp_path, fake APIs with monkeypatch, test database loads and idempotency, and compare DataFrames.
- Build an Idempotent CSV Loader in PythonBuild a small Python loader that reads CSV files, skips and counts bad rows, and loads into a database so running it twice gives the same result.
Projects and case studies
Apply what you learned and prepare material to discuss in interviews.
Projects
- BeginnerCSV to Data Warehouse PipelineA small online shop exports orders as daily CSV files. Build a pipeline that loads them into a star schema so that sales can be reported reliably, even when files are resent, corrected or contain bad rows.
- AdvancedFraud Detection Data PipelineBuild the data side of a fraud-detection system: compute per-card behavioural features from a transaction stream using only past data, flag suspicious transactions with transparent rules, keep a point-in-time-correct feature table that a model could train on, and backtest the rules honestly.
System design case studies
- AdvancedDesign an LLM RAG Data Ingestion PipelineDesign the ingestion pipeline behind an internal assistant that answers employees' questions using company documents (wiki pages, shared drives, PDFs, support tickets, policies), so that retrieved passages are relevant, current, and never shown to someone who could not open the original document.
- AdvancedDesign a Vector Embeddings PipelineDesign a platform that computes, stores and serves vector embeddings for several use cases (product search, recommendations, document retrieval) over hundreds of millions of items: keep embeddings in sync with changing source data, let teams upgrade embedding models safely, and serve low-latency nearest-neighbour queries with measured recall.
Resources
Cheat sheets
Related courses
- SQLSQL is the core language of data work: querying, transforming and modelling data in warehouses, lakehouses and Spark. Start here before any other tool.
- PySparkPySpark is the Python API for Apache Spark. Learn DataFrames, joins, window functions and how partitions and shuffles decide performance.
- AirflowAirflow schedules and orchestrates pipelines as DAGs. Learn scheduling, task dependencies, retries and idempotent task design.