Data Lakes, Lakehouse Architecture and Delta Lake

A data lake stores files cheaply in object storage. A lakehouse adds the reliability of a warehouse on top of those files through an open table format such as Delta Lake. This guide connects the pieces.

1. Lakes, warehouses and lakehouses

Lakes are flexible and cheap but lack transactions and enforced schemas; warehouses are reliable but keep data in their own storage; lakehouses keep data in open formats on object storage while adding transactions and governance.

Read: Lake vs warehouse vs lakehouse · Practise: When to choose each

2. File formats

Use columnar Parquet for analytical data, row-based Avro for messages, and treat JSON and CSV as landing formats only.

Read: JSON vs Parquet · Parquet vs Avro vs ORC · Practise: Parquet internals

3. Table formats and the transaction log

Delta Lake writes Parquet data files plus a _delta_log of commits. That log provides atomic writes, consistent snapshots, optimistic concurrency, time travel and row-level MERGE, UPDATE and DELETE. Apache Iceberg and Apache Hudi solve the same problem with different metadata designs.

Read: Delta Lake transactions, schema evolution and time travel · Delta vs plain Parquet tables, Iceberg and Hudi · Practise: What Delta Lake solves · Delta vs Iceberg vs Hudi · Time travel uses and limits

4. MERGE, CDC and streaming

MERGE applies inserts, updates and deletes from a change feed in one commit and can maintain a Type 2 dimension. The change data feed exposes row-level changes to downstream jobs, and Delta tables work as exactly-once streaming sinks and as streaming sources.

Read: MERGE, change data feed and streaming · Practise: How MERGE works and why concurrent MERGEs fail

5. Schema enforcement and evolution

Reject mismatched writes by default; evolve schemas deliberately, preferring additive changes and expand-and-contract migrations for breaking ones.

Read: Schema evolution patterns · Practise: When schema evolution is safe

6. Maintenance and layout

Compact small files with OPTIMIZE, cluster data on common filters (liquid clustering on new tables, Z-order on older ones), use deletion vectors for cheap row-level changes, and VACUUM with a retention that respects time travel and long readers.

Read: OPTIMIZE, Z-ordering, VACUUM and liquid clustering · Partitioning, clustering and data layout · Practise: Z-order vs liquid clustering · VACUUM risks

7. Medallion layers

Organise tables into bronze (raw), silver (cleaned and conformed) and gold (business models). Each layer reads only the previous one and can be rebuilt.

Read: Databricks workspace, jobs and lakehouse · Modelling methodologies: medallion

8. Governance

A catalog provides names, permissions, masking and lineage across engines.

Read: Unity Catalog and governance

Design it

Put it together in the scalable lakehouse case study, then build the Kafka → Spark → Delta streaming project.