Course · Cloud
AWS for Data Engineers
AWS for Data Engineers: S3, IAM, Lambda, Glue, Athena, Redshift, EMR, Kinesis, Step Functions, Lake Formation and how they fit into one data platform.
- Lessons
- 14
- Interview questions
- 1
- Projects & case studies
- 2
- Reading time
- ~4 h
Your progress
Saved in this browser onlyCourse structure
- 1 lessonStart hereThe complete overview of the course in one read.
- 2 lessonsBeginnerCore concepts you will use every day.
- 8 lessonsIntermediatePatterns used in production pipelines.
- 3 lessonsAdvancedPerformance, internals and edge cases.
Practise
Lessons
Work through the lessons in order. Completed lessons show a tick; lessons you have opened are outlined.
Start here
The complete overview of the course in one read.
Beginner
Core concepts you will use every day.
- Amazon S3 for Data EngineersS3 for data lakes: buckets and keys, storage classes, lifecycle rules, versioning, partitioned layouts, events, encryption, access points and multipart upload.
- AWS IAM for Data Engineers: Roles, Policies and Cross-Account AccessIAM for data pipelines: users, groups and roles, identity and resource policies, least privilege, STS, cross-account access, boundaries and policy evaluation.
Intermediate
Patterns used in production pipelines.
- AWS Lambda for Data PipelinesAWS Lambda in data pipelines: triggers, S3 and Kinesis processing, concurrency, cold starts, layers, memory and timeout tuning, and DLQ error handling.
- AWS Glue: Data Catalog, Crawlers and Spark ETL JobsAWS Glue for Data Engineers: Data Catalog, crawlers, partitions, Spark ETL jobs, DynamicFrames, Glue Studio, job bookmarks, workflows and cost tuning.
- Amazon Athena: Serverless SQL on S3Query S3 with Amazon Athena: external tables, CTAS, partition projection, performance tuning, workgroups, federated queries and what drives the cost per query.
- Amazon Redshift for Data EngineersRedshift for Data Engineers: architecture, RA3, distribution and sort keys, COPY and UNLOAD, vacuum, WLM, concurrency scaling, Spectrum and Serverless.
- Amazon Kinesis: Data Streams, Firehose and Stream ProcessingAmazon Kinesis for Data Engineers: Data Streams, shard throughput, KCL and KPL, enhanced fan-out, Firehose, Managed Flink, and how it compares with Kafka.
- AWS Step Functions for Data Pipeline OrchestrationOrchestrate AWS data pipelines with Step Functions: state machines, Standard vs Express, Map and Parallel, retries and catches, Glue and Lambda integrations.
- Monitoring Data Pipelines on AWS: CloudWatch and EventBridgeMonitor AWS data pipelines with CloudWatch metrics, alarms, custom metrics, Logs Insights and dashboards, plus EventBridge rules that react to job failures.
- AWS Data Services Landscape: Networking, Databases, Messaging, CDC and CostAWS data platform services: VPC networking, RDS and Aurora, DynamoDB, SQS and SNS, DMS for CDC, Amazon MSK, Cost Explorer and Well-Architected.
Advanced
Performance, internals and edge cases.
- Amazon EMR: Spark Clusters, EMR on EKS and EMR ServerlessSpark on Amazon EMR: cluster node types, Spark tuning, instance fleets and Spot, bootstrap actions, EMR on EKS, EMR Serverless and cost control.
- AWS Lake Formation: Data Lake Governance and Fine-Grained AccessAWS Lake Formation for S3 data lakes: permissions over the Glue Data Catalog, column, row and cell filters, LF-Tags, blueprints and cross-account sharing.
- Modern AWS Data Services: Iceberg, Data Quality, Serverless and IaCModern AWS data services: S3 Tables and Iceberg, Glue Data Quality, DataBrew, AppFlow, OpenSearch, QuickSight, Secrets Manager, CDK and cost tags.
Projects and case studies
Apply what you learned and prepare material to discuss in interviews.
Projects
System design case studies
Resources
Related courses
- Apache SparkHow Spark works inside: driver and executors, RDDs, jobs and stages, shuffles and skew, caching, Catalyst, AQE, memory tuning, Kubernetes, History Server.
- KafkaKafka is a distributed log used for streaming data. Learn topics, partitions, consumer groups and delivery semantics before building streaming pipelines.
- AirflowAirflow schedules and orchestrates pipelines as DAGs. Learn scheduling, task dependencies, retries and idempotent task design.
- SnowflakeSnowflake is a cloud data warehouse that separates storage from compute. Learn virtual warehouses, micro-partitions, pruning, caching and cost control.