What is Delta Lake?

How it works

A Delta table is a folder of Parquet data files and a _delta_log folder of numbered JSON commits (00000000000000000000.json, …01.json, …). Each commit lists the files it added and removed. A reader replays the log (helped by periodic Parquet checkpoints) to learn which files make up the current version, so files that a failed or in-flight write left behind are never read.

Writers commit with optimistic concurrency: they write new files, then try to create the next log entry. If another writer took that number first, Delta checks whether the two changes conflict and either retries or fails. That is what turns plain object storage into a table with:

• ACID transactions: a write is all or nothing, and readers see a consistent snapshot.
• DML: UPDATE, DELETE and MERGE, done by rewriting the affected files (or marking rows with deletion vectors).
• Schema enforcement and evolution: writes with mismatched columns are rejected unless you opt in to schema evolution.
• Time travel: older versions stay readable until VACUUM removes their files.

Maintenance commands such as OPTIMIZE (compaction, with Z-ordering or liquid clustering) keep reads fast. Delta Lake is the default table format on Databricks and is also readable from Spark, Trino, Flink and other engines.

Example

An upsert with merge, then an append with a wrong column type that Delta rejects. The history shows two commits (PySpark 4.2.0 with delta-spark 4.4.0).

import os, tempfile
from pyspark.sql import SparkSession
from delta import configure_spark_with_delta_pip
from delta.tables import DeltaTable

builder = (SparkSession.builder.master("local[2]").appName("glossary-delta")
           .config("spark.sql.extensions", "io.delta.sql.DeltaSparkSessionExtension")
           .config("spark.sql.catalog.spark_catalog", "org.apache.spark.sql.delta.catalog.DeltaCatalog")
           .config("spark.sql.shuffle.partitions", "2"))
spark = configure_spark_with_delta_pip(builder).getOrCreate()
spark.sparkContext.setLogLevel("ERROR")
path = os.path.join(tempfile.mkdtemp(), "customers")

spark.createDataFrame([(1, "Pune"), (2, "Delhi")], "id INT, city STRING").write.format("delta").save(path)

# Upsert: id 2 changes city, id 3 is new.
updates = spark.createDataFrame([(2, "Chennai"), (3, "Kochi")], "id INT, city STRING")
(DeltaTable.forPath(spark, path).alias("t")
   .merge(updates.alias("s"), "t.id = s.id")
   .whenMatchedUpdateAll().whenNotMatchedInsertAll().execute())

# Schema enforcement: a write with a wrong column type is rejected.
try:
    spark.createDataFrame([("4", "Goa")], "id STRING, city STRING").write.format("delta").mode("append").save(path)
except Exception as e:
    print("rejected:", type(e).__name__)

spark.read.format("delta").load(path).orderBy("id").show()
DeltaTable.forPath(spark, path).history().select("version", "operation").orderBy("version").show()
spark.stop()

rejected: AnalysisException
+---+-------+
| id|   city|
+---+-------+
|  1|   Pune|
|  2|Chennai|
|  3|  Kochi|
+---+-------+

+-------+---------+
|version|operation|
+-------+---------+
|      0|    WRITE|
|      1|    MERGE|
+-------+---------+

Common mistakes

• Reading a Delta folder as plain Parquet, which includes files the log has removed.
• Running VACUUM with a short retention while long queries or time-travel reads still need old files.
• Letting streaming writes create many small files and never running OPTIMIZE.
• Turning on automatic schema evolution everywhere, so a typo in an upstream column silently adds a new column.