Projects
Data Engineering projects
Projects that you can build, run and explain. Each one ends with the interview questions it prepares you for.
Beginner
Intermediate
- IntermediateE-commerce Analytics Data PlatformAn online store has orders, customers, products, refunds and web sessions in separate systems. Build an ELT platform that models them into trusted marts for revenue, conversion and repeat customers, with every metric defined once and tested.
- IntermediateLarge-Scale Batch Processing PipelineProcess a large public dataset (several gigabytes or more) with PySpark into partitioned, query-ready tables, and document, with evidence from the Spark UI and query plans, how you found and fixed the main performance bottleneck.
- IntermediateS3 → PySpark → Snowflake Data PipelineRaw event files arrive in object storage every day. Clean, deduplicate and aggregate them with PySpark, write curated Parquet partitioned by date, and load curated tables into a cloud warehouse for analysts, so that any day can be reprocessed on demand without duplicates.
Advanced
- AdvancedChange Data Capture PipelineReplicate an operational PostgreSQL table into a lakehouse table within minutes, including updates and deletes, so analysts query current data without touching the production database, and prove that replays, duplicates and out-of-order delivery cannot corrupt the copy.
- AdvancedFraud Detection Data PipelineBuild the data side of a fraud-detection system: compute per-card behavioural features from a transaction stream using only past data, flag suspicious transactions with transparent rules, keep a point-in-time-correct feature table that a model could train on, and backtest the rules honestly.
- AdvancedKafka → Spark → Delta Lake Streaming PipelineAn application emits user events to Kafka. Build a streaming pipeline that lands them in Delta Lake within a minute, deduplicates replays, produces per-minute aggregates that tolerate late events, and recovers from restarts without losing or double-counting data.
- AdvancedReal-Time Analytics PipelineBuild a pipeline that turns a stream of order events into per-minute revenue and order counts by category, visible on a dashboard within a minute, correct when events arrive late or twice, and reconciled with a batch recount.