Course · Interview foundations
Dev tools for Data Engineers
The command line, shell scripts, Git, Python environments, Docker and repo automation: the day-one toolkit every data pipeline job assumes you already know.
- Lessons
- 7
- Interview questions
- 0
- Projects & case studies
- 0
- Reading time
- ~2 h
Your progress
Saved in this browser onlyCourse structure
- 1 lessonStart hereThe complete overview of the course in one read.
- 2 lessonsBeginnerCore concepts you will use every day.
- 3 lessonsIntermediatePatterns used in production pipelines.
- 1 lessonAdvancedPerformance, internals and edge cases.
Practise
Lessons
Work through the lessons in order. Completed lessons show a tick; lessons you have opened are outlined.
Start here
The complete overview of the course in one read.
Beginner
Core concepts you will use every day.
- The Command Line for Data Work: Pipes, grep, awk, jq and Large FilesCommand line for data engineers: pipes, redirection, grep, awk, sort and jq to inspect large CSV and JSON files, plus exit codes, set -euo pipefail and SSH.
- Git for Data Engineers: Branches, Merge vs Rebase, Pull Requests and SecretsGit for data engineers: branches, merge vs rebase, conflicts, pull requests, .gitignore for data repos, tags, and what to do when you commit a secret.
Intermediate
Patterns used in production pipelines.
- Shell Scripting for Data Pipelines: Robust Bash, Retries, Traps and CronBash scripting for data pipelines: a robust script with arguments, logging, traps, retries, locking and idempotent loads, plus cron and its limits.
- Python Environments and Packaging for Data Engineers: venv, pip, Lock Files and pyproject.tomlPython virtual environments and packaging for data pipelines: venv, pip, lock files, pyproject.toml, uv and pip-tools, and why it works on my machine.
- Docker for Data Engineers: Images, Dockerfiles, Volumes and Docker ComposeDocker for data engineers: images vs containers, a multi-stage non-root Dockerfile for a Python pipeline, volumes, config, and Docker Compose with Postgres.
Resources
Related courses
- PythonPython glues pipelines together: ingestion, validation, orchestration and PySpark jobs. Focus on functions, generators, error handling and testable code.
- SQLSQL is the core language of data work: querying, transforming and modelling data in warehouses, lakehouses and Spark. Start here before any other tool.
- Apache AirflowApache Airflow schedules and orchestrates pipelines as DAGs. Learn scheduling, task dependencies, retries and idempotent task design.
