What is a Lakehouse?

How it works

A lakehouse keeps the data in Parquet files in your own storage and adds metadata that records exactly which files make up each version of a table. Delta Lake writes an ordered transaction log of JSON commits next to the data; Iceberg keeps a tree of metadata and manifest files and swaps a pointer in a catalog; Hudi keeps a timeline of commits. In all three, a reader looks at the metadata, not at whatever files happen to be in the folder.

That design gives lake files the properties a warehouse table has:

• ACID commits: a write becomes visible all at once or not at all.
• Schema enforcement and evolution: a mismatched write fails unless evolution is requested.
• Row-level changes: MERGE, UPDATE and DELETE.
• Time travel: query an earlier version for audits, debugging or reproducible ML.
• File statistics for data skipping, plus compaction and clustering.

Several engines can share one copy of the data: Spark, Trino, Flink, DuckDB and major warehouses can read (and some write) Delta or Iceberg tables. A catalog such as Unity Catalog, AWS Glue or Apache Polaris maps table names to metadata and enforces access. Teams usually organise lakehouse tables in medallion layers (bronze, silver, gold).

The cost is operational: someone must run compaction and cleanup, choose layouts, keep engines and catalogs on compatible versions, and decide which engine owns writes to each table.

Example

A Delta table in a local folder: each write is a numbered commit in _delta_log, and an earlier version stays readable (PySpark 4.2.0 with delta-spark 4.4.0).

import os, tempfile
from pyspark.sql import SparkSession
from delta import configure_spark_with_delta_pip

builder = (SparkSession.builder.master("local[2]").appName("glossary-lakehouse")
           .config("spark.sql.extensions", "io.delta.sql.DeltaSparkSessionExtension")
           .config("spark.sql.catalog.spark_catalog", "org.apache.spark.sql.delta.catalog.DeltaCatalog")
           .config("spark.sql.shuffle.partitions", "2"))
spark = configure_spark_with_delta_pip(builder).getOrCreate()
spark.sparkContext.setLogLevel("ERROR")
path = os.path.join(tempfile.mkdtemp(), "orders")

spark.createDataFrame([(1, 40.0), (2, 25.0)], "id INT, amount DOUBLE").write.format("delta").save(path)
spark.createDataFrame([(3, 10.0)], "id INT, amount DOUBLE").write.format("delta").mode("append").save(path)

print(sorted(f for f in os.listdir(f"{path}/_delta_log") if f.endswith(".json")))
print("now:", spark.read.format("delta").load(path).count(), "rows")
print("version 0:", spark.read.format("delta").option("versionAsOf", 0).load(path).count(), "rows")
spark.stop()

['00000000000000000000.json', '00000000000000000001.json']
now: 3 rows
version 0: 2 rows

Common mistakes

• Treating a lakehouse as maintenance-free; without compaction and cleanup, small files and old versions pile up.
• Letting two engines write the same table without checking they support the same protocol features.
• Reading table folders directly as plain Parquet, which ignores the log and returns removed or uncommitted files.
• Choosing a lakehouse only for fashion when a managed warehouse would serve a SQL-only team with less work.