Resources
Data Engineering glossary
Short, plain-English definitions of the terms you meet in Data Engineering lessons and interviews. Each term has its own page with how it works, an example, common mistakes and the lessons that teach it.
A
- ACID TransactionsACID transactions explained: atomicity, consistency, isolation and durability, why pipelines need them, and how lakehouse tables add them. Runnable SQL example.
- Apache IcebergApache Iceberg explained: an open table format with snapshots, hidden partitioning and safe schema evolution, shared by Spark, Trino and Snowflake. Example.
- At-least-onceAt-least-once delivery explained: process first, commit the offset after, so no record is lost but retries create duplicates. How to dedupe, with an example.
- AvroApache Avro explained: a compact, row-based binary format with a JSON schema, used for Kafka messages and schema evolution. How it compares to Parquet. Example.
B
- BackfillBackfill in data engineering: rerunning a pipeline for a past date range after a bug fix or new source, done safely with idempotent chunks. With an example.
- Batch ProcessingBatch processing explained: running pipelines on bounded data on a schedule, when it beats streaming, how micro-batch fits in, with a runnable SQL example.
- Broadcast JoinBroadcast join in Spark explained: copying a small table to every executor so the large table is joined without a shuffle, its limits, and a PySpark example.
- BucketingBucketing in Spark explained: hashing a high-cardinality key into a fixed number of files at write time so later joins skip the shuffle. With a PySpark example.
C
- Change Data Capture (CDC)Change data capture (CDC) explained: how log-based CDC streams inserts, updates and deletes from a database, and how to apply change events with MERGE.
- Columnar StorageColumnar storage explained: why analytics engines store each column together, how that cuts I/O and improves compression, and when row storage wins. Example.
- Consumer GroupKafka consumer groups explained: how consumers share a topic's partitions, why parallelism is capped by partition count, and what triggers a rebalance. Example.
- Consumer LagKafka consumer lag explained: how far a consumer group trails the newest messages per partition, how to measure it, and what causes and fixes growing lag.
- CTECTE (common table expression) explained: naming a subquery with WITH to make SQL readable, chaining steps, and recursive CTEs for hierarchies. SQL example.
D
- DAGDAG (directed acyclic graph) explained for data pipelines: tasks as nodes, dependencies as edges, why cycles are forbidden, and how a scheduler orders tasks.
- Data CatalogData catalog explained: a searchable inventory of datasets with schemas, owners, descriptions, lineage and access rules, like Unity Catalog. SQL example.
- Data ContractData contracts explained: an agreed, versioned spec of a dataset's schema, meaning, quality and owner, checked in CI so changes never surprise users.
- Data LakeData lake explained: files of any format in cheap object storage such as S3, read with schema-on-read, why lakes turn into swamps, with a PyArrow example.
- Data LineageData lineage explained: tracing where data comes from, how it is transformed and what depends on it, for impact analysis, debugging and audits. Python example.
- Data QualityData quality explained: completeness, uniqueness, validity and freshness, the pipeline checks that enforce them, and what to do when one fails. SQL example.
- Data SkewData skew in Spark explained: why one hot key makes one task run far longer than the rest, how to detect it, and fixes from AQE to salting. PySpark example.
- Data VaultData Vault explained: hubs, links and satellites, hash keys and insert-only loading for an auditable integration layer, and when it is worth the overhead.
- Data WarehouseData warehouse explained: a database modelled for analytical questions across the business, how it differs from an operational database, with a SQL example.
- Dead-letter QueueDead-letter queue (DLQ) explained: parking records that keep failing so a pipeline keeps running, what to store with them and how to replay them. Example.
- Delta LakeDelta Lake explained: an open table format that adds a transaction log to Parquet files for ACID writes, MERGE, schema enforcement and time travel. Example.
- Dimension TableDimension table explained: the who, what, where and when of business events, why dimensions are wide and readable, and Unknown members. With SQL example.
E
- ELT (Extract, Load, Transform)ELT (extract, load, transform) explained: load raw data first and transform it with SQL inside the warehouse, as dbt does. Includes a runnable SQL example.
- ETL (Extract, Transform, Load)ETL (extract, transform, load) explained: when transforming data before it lands still beats ELT, how an ETL step works, and a runnable Python example.
- Exactly-onceExactly-once processing explained: why retries cause duplicates, how idempotent writes and atomic offset commits give exactly-once results, with an example.
F
G
I
- IdempotencyIdempotency in data pipelines: what it means for a job to be safe to rerun, how overwrite and MERGE patterns achieve it, with a runnable SQL example.
- Incremental ModelIncremental models explained: processing only new or changed rows each run instead of rebuilding the table, with watermarks, merges and lookbacks. SQL example.
K
L
- LakehouseLakehouse explained: table formats like Delta Lake and Iceberg add ACID commits, MERGE and time travel to files in object storage. With a PySpark example.
- Late-Arriving DataLate-arriving data explained: events or facts that land after their period was processed, and the batch and streaming patterns that keep totals correct.
M
- Materialized ViewMaterialized views explained: a stored query result that makes repeated aggregations fast, how refresh works, and how it differs from a view. SQL example.
- Medallion ArchitectureMedallion architecture explained: bronze, silver and gold lakehouse layers, what belongs in each, and how data moves between them. With a runnable SQL example.
O
- OLAP (Online Analytical Processing)OLAP (online analytical processing) explained: large scans and aggregations over history on columnar, parallel engines, and how it differs from OLTP, with SQL.
- OLTP (Online Transaction Processing)OLTP (online transaction processing) explained: many small, concurrent reads and writes by key with ACID guarantees, and how it differs from OLAP. SQL example.
- OrchestrationData orchestration explained: scheduling pipeline tasks, running them in dependency order, retrying failures and stopping bad data. Tools and an example.
P
- ParquetApache Parquet explained: a columnar file format with row groups, per-column compression and min/max statistics that let engines skip data. PyArrow example.
- PartitioningData partitioning explained: splitting tables into folders or segments by a column such as date so queries skip what they do not need. PySpark example included.
- Predicate PushdownPredicate pushdown explained: how engines push WHERE filters into the scan so partitions, files and Parquet row groups are skipped unread. PySpark example.
R
S
- SCD Type 2SCD Type 2 explained: keeping every version of a dimension row with valid_from, valid_to and is_current, loading it idempotently and point-in-time joins.
- Schema EvolutionSchema evolution explained: changing a table or message schema as sources change, which changes are safe, and how Delta, Iceberg and Avro handle them. Example.
- Schema RegistrySchema registry explained: a service that stores versioned Avro, Protobuf or JSON schemas for Kafka topics and blocks incompatible changes. With an example.
- Semantic LayerSemantic layer explained: defining metrics such as revenue and average order value once so BI tools, notebooks and AI assistants all get the same numbers.
- ShuffleShuffle in Spark explained: how groupBy, joins and distinct move rows between executors, why shuffles are expensive, and how to make them cheaper, in PySpark.
- Slowly Changing Dimension (SCD)Slowly changing dimension (SCD) explained: how Types 0, 1, 2, 3 and 6 handle changed attributes like a customer's city, and how to choose. With SQL example.
- Small Files ProblemThe small files problem explained: why thousands of tiny files slow Spark, Athena and lakehouse reads, what creates them, and how compaction fixes it. Example.
- Snowflake SchemaSnowflake schema explained: a star schema whose dimensions are normalised into further tables, the trade-off in joins and readability, and a SQL example.
- Star SchemaStar schema explained: one fact table surrounded by denormalised dimension tables, why warehouses use it, and the rules that keep it correct. With SQL example.
- Stream ProcessingStream processing explained: handling unbounded events continuously with state, event time and checkpoints, and when it is worth the cost. PySpark example.
- Surrogate KeySurrogate key explained: a warehouse-generated key with no business meaning, why dimensions need one for SCD Type 2, sequence vs hash keys, with a SQL example.
T
U
W
- WatermarkWatermark in stream processing: how an event-time watermark decides when a window is complete and late events are dropped, with a runnable example.
- Window FunctionSQL window functions explained: ROW_NUMBER, RANK, LAG and running totals computed over a partition of rows without collapsing them like GROUP BY. SQL example.