Delta Lake vs Apache Iceberg vs Apache Hudi: Which Table Format to Choose

Delta Lake, Apache Iceberg and Apache Hudi are the three open table formats. All three take a folder of Parquet files and add a layer of metadata, so the folder behaves like a database table: changes are all-or-nothing, readers see a consistent snapshot, you can go back in time, and you can update or delete single rows. They differ in how they store that metadata, how they make a change safe, and which engines and tools support them best. This lesson explains those differences, shows two of the formats side by side in code, and ends with a simple way to choose.

If you need a short answer for an interview, read Delta vs Iceberg vs Hudi: interview answer. This lesson is the longer version, with the reasons behind each point.

What all three give you

Before comparing them, it helps to see what they share. Each format adds:

• Atomic commits. A write either fully appears or does not appear at all. A failed job never leaves half a table behind.
• Snapshot isolation. A reader sees one consistent version of the table, even while a writer is adding files.
• Time travel. Older versions stay readable until you clean them up.
• Schema enforcement and evolution. Bad writes are rejected, and columns can be added or changed safely.
• Row-level changes. UPDATE, DELETE and MERGE on files that cannot be edited in place.
• File statistics. Minimum and maximum values per file, so engines can skip files that cannot match a filter.

So “which one supports ACID?” is the wrong question. They all do. The useful questions are about design and ecosystem.

How each format stores its metadata

This is the biggest design difference, and most other differences follow from it.

Delta Lake keeps an ordered transaction log in a _delta_log/ folder. Each commit is one JSON file, numbered 00000000000000000000.json, 00000000000000000001.json and so on. Each file lists actions such as “add this data file” and “remove that one”. Every few commits, Delta writes a Parquet checkpoint that sums up the log so far, so readers do not have to replay thousands of JSON files. The current table is “replay the log from the last checkpoint”. The full detail is in Delta Lake transactions and schema evolution.

Apache Iceberg keeps a tree of metadata files. At the top is a table metadata file (.metadata.json) that holds the schema, partition spec and list of snapshots. Each snapshot points to a manifest list, which points to manifests, which list the data files with their statistics. A catalog (a REST catalog, Hive Metastore, AWS Glue, a JDBC database, Nessie and others) stores one pointer: the location of the current metadata file.

Apache Hudi keeps a timeline in a .hoodie/ folder: a record of every action on the table (commits, delta commits, compactions, cleans), each moving through requested, inflight and completed states. Hudi also keeps an internal metadata table with file listings, statistics and indexes. Hudi is built around record keys, which it uses to find the existing version of a row quickly during upserts.

How a commit is made safe

Object storage such as S3 has no transactions, so each format needs one step that cannot be done twice.

	How the commit becomes visible	What it needs
Delta Lake	Write the next numbered log file, but only if no one else wrote it first	Storage that can “create only if absent”, or a commit coordinator for stores that cannot
Iceberg	Swap the catalog’s pointer from the old metadata file to the new one, if it has not changed	A catalog that supports compare-and-swap
Hudi	Move an action on the timeline to “completed”	A lock provider when several writers run at once

In all three, two writers that change different files can usually both succeed: the second one checks for conflicts and retries. Two writers that change the same files conflict, and one fails.

See the difference on disk

The example below creates the same small orders table as a Delta table (with PySpark) and as an Iceberg table (with PyIceberg, the Python library for Iceberg, using a local SQLite catalog). Then it changes one row in each and counts the files each format wrote. You need pyspark, delta-spark and pyiceberg[sql-sqlite,pyarrow].

import json, os, tempfile
from collections import Counter
import pyarrow as pa
from pyspark.sql import SparkSession
from delta import configure_spark_with_delta_pip
from pyiceberg.catalog.sql import SqlCatalog
from pyiceberg.expressions import EqualTo

builder = (
    SparkSession.builder.master("local[2]").appName("table-formats")
    .config("spark.sql.extensions", "io.delta.sql.DeltaSparkSessionExtension")
    .config("spark.sql.catalog.spark_catalog", "org.apache.spark.sql.delta.catalog.DeltaCatalog")
    .config("spark.sql.shuffle.partitions", "2")
    .config("spark.ui.showConsoleProgress", "false")
)
spark = configure_spark_with_delta_pip(builder).getOrCreate()
spark.sparkContext.setLogLevel("ERROR")
base = tempfile.mkdtemp()

def metadata_files(root):
    """Sort a table folder's files into kinds, so the layout is easy to compare."""
    kinds = Counter()
    for d, _, files in os.walk(root):
        for f in files:
            if f.endswith(".crc"):
                continue
            if f.endswith(".metadata.json"):
                kinds["Iceberg table metadata (.metadata.json)"] += 1
            elif f.startswith("snap-") and f.endswith(".avro"):
                kinds["Iceberg manifest list (snap-*.avro)"] += 1
            elif f.endswith(".avro"):
                kinds["Iceberg manifest (*.avro)"] += 1
            elif os.path.basename(d) == "_delta_log":
                kinds["Delta log commit (_delta_log/*.json)"] += 1
            elif f.endswith(".parquet"):
                kinds["Parquet data file"] += 1
    for kind, n in sorted(kinds.items()):
        print(f"  {n}  {kind}")

Delta Lake. Write three orders as one file, then change the amount of order 3.

rows = [(1, "IN", 100.0), (2, "US", 250.0), (3, "IN", 75.0)]

# Delta Lake: create the table, then change one row
delta_path = f"{base}/orders_delta"
(spark.createDataFrame(rows, "order_id INT, country STRING, amount DOUBLE")
      .coalesce(1).write.format("delta").save(delta_path))
spark.sql(f"UPDATE delta.`{delta_path}` SET amount = 80.0 WHERE order_id = 3")

print("Delta:")
metadata_files(delta_path)
with open(f"{delta_path}/_delta_log/00000000000000000001.json") as f:
    print("  actions in commit 1:", [next(iter(json.loads(line))) for line in f])

Delta:
  2  Delta log commit (_delta_log/*.json)
  2  Parquet data file
  actions in commit 1: ['commitInfo', 'add', 'remove']

Two commits, one per write. The UPDATE wrote a new Parquet file with all three rows (one of them changed) and logged add for the new file and remove for the old one. The old file is still on disk for time travel, until VACUUM deletes it. This is copy-on-write: changing one row rewrote the whole file.

Apache Iceberg. The same rows and the same change.

# Apache Iceberg: the same table and the same change, through PyIceberg and a local SQLite catalog
warehouse = f"{base}/iceberg"
os.makedirs(warehouse)
catalog = SqlCatalog("local", uri=f"sqlite:///{warehouse}/catalog.db", warehouse=f"file://{warehouse}")
catalog.create_namespace("shop")

data = pa.table({"order_id": pa.array([r[0] for r in rows], pa.int32()),
                 "country": [r[1] for r in rows], "amount": [r[2] for r in rows]})
orders = catalog.create_table("shop.orders", schema=data.schema)
orders.append(data)
orders.overwrite(pa.table({"order_id": pa.array([3], pa.int32()), "country": ["IN"], "amount": [80.0]}),
                 overwrite_filter=EqualTo("order_id", 3))

print("Iceberg:")
metadata_files(orders.location().removeprefix("file://"))
print("  snapshot operations:", [s.summary.operation.value for s in orders.snapshots()])

Iceberg:
  4  Iceberg manifest (*.avro)
  3  Iceberg manifest list (snap-*.avro)
  3  Iceberg table metadata (.metadata.json)
  3  Parquet data file
  snapshot operations: ['append', 'overwrite', 'append']

Iceberg wrote a new metadata file for each change (the empty table, the first append, and the update), and each snapshot has its own manifest list. PyIceberg made the change in two snapshots: an overwrite that rewrote the old file without order 3, then an append with the new row. The catalog now points to the newest metadata file. That pointer swap is the commit.

Both tables return the same data:

print(spark.read.format("delta").load(delta_path).orderBy("order_id").collect())
print(orders.scan().to_arrow().sort_by("order_id").to_pylist())

[Row(order_id=1, country='IN', amount=100.0), Row(order_id=2, country='US', amount=250.0), Row(order_id=3, country='IN', amount=80.0)]
[{'order_id': 1, 'country': 'IN', 'amount': 100.0}, {'order_id': 2, 'country': 'US', 'amount': 250.0}, {'order_id': 3, 'country': 'IN', 'amount': 80.0}]

Hudi is not shown here because it has no lightweight Python writer; it is normally written from Spark or Flink with the Hudi bundle jar. On disk, you would see a .hoodie/ timeline folder instead of _delta_log/ or metadata/.

Partitioning

• Delta Lake uses ordinary partition columns: you choose a column such as order_date, and data is split into one folder per value. Queries must filter on that exact column to skip folders. Newer tables can use liquid clustering instead of partitions, which lets you change the clustering columns later without rewriting the table (see OPTIMIZE, Z-ordering, VACUUM and liquid clustering).
• Iceberg uses hidden partitioning. The table declares a transform such as days(event_ts) or bucket(16, user_id). Users filter on event_ts as normal, and the engine works out which partitions to skip. Users never see or write a partition column. Iceberg also supports partition evolution: you can change the partition spec, for example from daily to hourly, and only new data uses the new layout. Old files do not need rewriting.
• Hudi uses partition paths, plus clustering to rewrite data into a better layout in the background.

Updates and deletes

Files in a lake cannot be changed in place, so there are two basic ways to change a row:

• Copy-on-write (COW). Rewrite every file that contains a changed row. Reads stay fast, but writes are expensive when you change a few rows in many large files.
• Merge-on-read (MOR). Write the change separately (a small log file, a delete file or a deletion vector), and combine it with the original file when reading. Writes are cheap, reads pay a little more, and a background compaction later folds the changes into new files.

	Copy-on-write	Merge-on-read
Delta Lake	Default	Deletion vectors, which mark deleted rows without rewriting the file
Iceberg	Supported	Delete files in spec v2 (position or equality deletes), and deletion vectors in spec v3
Hudi	COW table type	MOR table type, with row changes written to log files and compacted later

Hudi was designed from the start for heavy upserts, for example from CDC, so its MOR tables, record keys and indexes (to find which file holds a key without scanning) are its strongest feature. Delta and Iceberg have caught up for many workloads, so the gap is smaller than older blog posts suggest.

Reading only what changed

Many pipelines need “what changed since my last run” rather than the whole table.

• Delta Lake has the change data feed: when enabled, it records inserted, updated and deleted rows per commit (see MERGE, upserts and the change data feed).
• Iceberg supports incremental reads between two snapshots. Appends are simple to read this way. Engines such as Spark also offer change-log views that work out updates and deletes between snapshots.
• Hudi has incremental queries as a core feature: “give me every record that changed after this instant on the timeline”.

Engines and catalogs

In practice, this section decides most choices.

• Delta Lake has its deepest support in Apache Spark and Databricks. Many other engines can read it, and some can write it, often through Delta Kernel, a library for building connectors.
• Iceberg was designed for many engines sharing one table. Spark, Flink, Trino, Presto, Hive and many cloud warehouses and query services can read it, and many can write it. The REST catalog specification gives all of them one standard way to find tables and commit changes.
• Hudi is used most with Spark and Flink, especially in streaming ingestion, and many query engines can read it.

Two features reduce the cost of choosing:

• Delta UniForm writes Iceberg metadata next to the Delta log, so Iceberg readers can query a Delta table.
• Apache XTable (incubating) translates metadata between all three formats.

Both make a table readable in another format. There is still one format that is the writer of record, and that is the one you design for.

Side-by-side summary

	Delta Lake	Apache Iceberg	Apache Hudi
Metadata	Ordered JSON commit log plus Parquet checkpoints	Tree: metadata file, manifest lists, manifests	Timeline of actions plus an internal metadata table
Commit	Create the next log file only if it does not exist	Compare-and-swap of the catalog pointer	Complete the action on the timeline, with a lock for many writers
Partitioning	Partition columns, or liquid clustering	Hidden partitioning and partition evolution	Partition paths plus clustering
Row changes	Copy-on-write, or deletion vectors	Copy-on-write, delete files (v2), deletion vectors (v3)	COW or MOR tables, with record keys and indexes
Changed-rows reads	Change data feed	Incremental snapshot reads	Incremental queries
Strongest fit	Spark and Databricks platforms	Many engines sharing tables through a catalog	Upsert-heavy and streaming ingestion

How to choose

Ask these questions in order:

1. Which engines must write to the tables? If it is only Spark, or only Databricks, Delta is the simplest choice. If Spark, Flink and a warehouse all write, Iceberg with a shared catalog is usually easier.
2. Which engines must read them? If readers need Iceberg but writers use Delta, UniForm may be enough. Check that every reader supports the features you plan to use.
3. What does your platform already use? Your cloud platform, catalog and team skills usually matter more than small feature differences. Moving formats later is a migration, not a setting.
4. What is the write pattern? Mostly appends with occasional fixes: any format works. Constant upserts from CDC with strict latency: test Hudi MOR tables, Delta with deletion vectors and Iceberg with merge-on-read against your own data.

Whichever you pick, plan for maintenance: compacting small files, expiring old snapshots or log entries, and removing orphan files (OPTIMIZE and VACUUM in Delta, rewrite_data_files, expire_snapshots and remove_orphan_files in Iceberg, compaction, clustering and cleaning services in Hudi). A table format without maintenance slowly gets slower and more expensive, whichever one it is.

Common mistakes

1. Calling them storage engines or databases. They are specifications for files plus metadata. The engine (Spark, Trino, Flink, a warehouse) does the work.
2. Saying only one of them is ACID. All three give atomic commits and snapshot isolation per table.
3. Choosing from benchmarks. Speed depends far more on file sizes, clustering, statistics and the engine than on the format.
4. Using a feature your readers cannot read. Turning on deletion vectors, column mapping or Iceberg v3 features can make a table unreadable by an older connector. Upgrade readers first.
5. Treating UniForm or XTable as “use both”. They make tables readable in another format; they do not give you two independent writers.

Interview angle

Interviewers usually ask this as “Delta vs Iceberg: which would you pick and why?” A strong answer names the shared features in one sentence, then compares the metadata design and commit, then chooses based on engines, catalog and write pattern, not on hype. For a ready-made answer and follow-up questions, see Delta vs Iceberg vs Hudi: interview answer and the Databricks interview questions.