What is Schema Evolution?

How it works

Sources change: an API adds a field, an application team renames a column, a type grows from int to bigint. Unplanned change is called schema drift; handling it deliberately is schema evolution. Two mechanisms work together:

• Schema enforcement rejects writes that do not match the table’s schema, so drift cannot slip in silently.
• Schema evolution lets you accept a change on purpose, such as Delta Lake’s mergeSchema option or ALTER TABLE ... ADD COLUMNS.

Changes fall into rough risk groups:

Change	Usually
Add a nullable column	Safe: old rows read as NULL
Widen a type (int → bigint)	Safe in most formats
Rename or drop a column	Breaking for readers; needs column mapping (Delta) or ID-based columns (Iceberg)
Narrow or change a type	Breaking: needs a rewrite or a new column

Formats differ. Plain Parquet folders have no table schema, so readers may merge or fail. Delta Lake and Iceberg store the schema in table metadata, and Iceberg tracks columns by ID, so renames are metadata-only. Kafka pipelines use Avro or Protobuf with a schema registry that checks compatibility before a producer can register a new version.

Example

An upstream system adds a tier column. Delta rejects the append until the write opts in to evolution; old rows then read tier as NULL (PySpark 4.2.0 with delta-spark 4.4.0).

import os, tempfile
from pyspark.sql import SparkSession
from delta import configure_spark_with_delta_pip

builder = (SparkSession.builder.master("local[2]").appName("glossary-schema-evolution")
           .config("spark.sql.extensions", "io.delta.sql.DeltaSparkSessionExtension")
           .config("spark.sql.catalog.spark_catalog", "org.apache.spark.sql.delta.catalog.DeltaCatalog")
           .config("spark.sql.shuffle.partitions", "2"))
spark = configure_spark_with_delta_pip(builder).getOrCreate()
spark.sparkContext.setLogLevel("ERROR")
path = os.path.join(tempfile.mkdtemp(), "customers")

spark.createDataFrame([(1, "Pune")], "id INT, city STRING").write.format("delta").save(path)
new = spark.createDataFrame([(2, "Delhi", "gold")], "id INT, city STRING, tier STRING")  # upstream added `tier`

try:
    new.write.format("delta").mode("append").save(path)            # enforcement: rejected
except Exception as e:
    print("without mergeSchema:", type(e).__name__)

new.write.format("delta").mode("append").option("mergeSchema", "true").save(path)  # evolution: opt in
df = spark.read.format("delta").load(path)
print(df.schema.simpleString())
df.orderBy("id").show()
spark.stop()

without mergeSchema: AnalysisException
struct<id:int,city:string,tier:string>
+---+-----+----+
| id| city|tier|
+---+-----+----+
|  1| Pune|NULL|
|  2|Delhi|gold|
+---+-----+----+

Common mistakes

• Turning on automatic schema merging everywhere, so a misspelled column becomes a permanent new column.
• Renaming a column in the source without telling consumers, which silently turns their column into NULLs.
• Reading a folder of Parquet files with mixed schemas and no table format to reconcile them.
• Treating a type change as an evolution when it really needs a backfill of history.