Course · Distributed processing
PySpark for Data Engineers
PySpark is the Python API for Apache Spark. Learn DataFrames, joins, window functions and how partitions and shuffles decide performance.
- Lessons
- 10
- Interview questions
- 7
- Projects & case studies
- 2
- Reading time
- ~3 h
Your progress
Saved in this browser onlyCourse structure
- 1 lessonStart hereThe complete overview of the course in one read.
- 4 lessonsBeginnerCore concepts you will use every day.
- 5 lessonsIntermediatePatterns used in production pipelines.
Practise
- InterviewPySpark interview questionsThe full list with difficulty, type and a box to tick off each one.
- Cheat sheetPySpark Cheat SheetA quick PySpark reference: reading and writing data, column expressions, joins, aggregations, window functions and the settings that matter for performance.
- InterviewAll interview questionsEvery question across all topics in one filterable list.
Lessons
Work through the lessons in order. Completed lessons show a tick; lessons you have opened are outlined.
Start here
The complete overview of the course in one read.
Beginner
Core concepts you will use every day.
- PySpark DataFrames, Columns and SchemasPySpark DataFrames and schemas: explicit StructType schemas, select, withColumn and filter, null handling, and safe type casting under Spark 4 ANSI mode.
- PySpark Transformations vs ActionsUnderstand lazy evaluation in PySpark: transformations build a plan, actions run it, and narrow versus wide transformations decide where Spark shuffles data.
- PySpark Aggregations, Pivot and Temporary ViewsPySpark aggregations: groupBy and agg, rollup and cube subtotals, pivot and unpivot, and sharing DataFrames with SQL through temporary views.
- Reading and Writing Data in PySpark: Parquet, CSV, JSON, ORC, Avro and JDBCRead and write Parquet, CSV, JSON, ORC and Avro in PySpark, capture corrupt records, load JDBC tables in parallel and use partition discovery to prune folders.
Intermediate
Patterns used in production pipelines.
- PySpark Joins and Join StrategyWrite correct PySpark joins, then control how Spark executes them: broadcast hash, sort-merge and shuffle hash joins, hints, AQE, bucketing and bucketed joins.
- PySpark Window Functions: Ranking, Lag and Running TotalsPySpark window functions for top-N per group, deduplication, lag and lead, running totals, gap filling and sessions, and avoiding single-partition windows.
- PySpark UDFs, pandas UDFs and the Pandas API on SparkWhen to write Python UDFs in PySpark, how Arrow and pandas UDFs reduce their cost, how applyInPandas and UDTFs work, and when the pandas API on Spark fits.
- Nested and Complex Types in PySpark: Structs, Arrays, Maps and explodeWork with nested JSON in PySpark: query structs, arrays and maps, use higher-order functions, flatten with explode and posexplode, and rebuild nested output.
- Writing Efficient Output: File Sizes, Small Files and Partitioned WritesPySpark write performance: repartition vs coalesce before a write, partitioned output, the small files problem, fewer shuffles and custom partitioners.
Projects and case studies
Apply what you learned and prepare material to discuss in interviews.
Projects
- IntermediateLarge-Scale Batch Processing PipelineProcess a large public dataset (several gigabytes or more) with PySpark into partitioned, query-ready tables, and document, with evidence from the Spark UI and query plans, how you found and fixed the main performance bottleneck.
- IntermediateS3 → PySpark → Snowflake Data PipelineRaw event files arrive in object storage every day. Clean, deduplicate and aggregate them with PySpark, write curated Parquet partitioned by date, and load curated tables into a cloud warehouse for analysts, so that any day can be reprocessed on demand without duplicates.
Resources
Cheat sheets
- Cheat sheetPySpark Cheat SheetA quick PySpark reference: reading and writing data, column expressions, joins, aggregations, window functions and the settings that matter for performance.
- Cheat sheetApache Spark Interview Cheat SheetThe Spark concepts interviewers ask about most, in one page: lazy evaluation, stages and shuffles, joins, partitions, skew, AQE, caching and Spark 4 defaults.
Related courses
- Apache SparkHow Spark works inside: driver and executors, RDDs, jobs and stages, shuffles and skew, caching, Catalyst, AQE, memory tuning, Kubernetes, History Server.
- SQLSQL is the core language of data work: querying, transforming and modelling data in warehouses, lakehouses and Spark. Start here before any other tool.
- PythonPython glues pipelines together: ingestion, validation, orchestration and PySpark jobs. Focus on functions, generators, error handling and testable code.
- Delta LakeDelta Lake adds ACID transactions, schema enforcement and time travel to files in a data lake, which is the foundation of the lakehouse pattern.