What is the Small Files Problem?

How it works

Every file costs something no matter how small it is: an object-store LIST and GET request, a footer to read, an entry in the table’s metadata, and often its own Spark task. Ten thousand 100 KB files hold the same data as one 1 GB file, but reading them can take many times longer and cost more in S3 or ADLS requests.

Small files usually come from:

• Streaming and micro-batches: every trigger writes new files per partition.
• Over-partitioning: partitioning by a high-cardinality column, or by hour and country and type, so each folder gets a sliver of data.
• Too many write tasks: 200 shuffle partitions writing a 50 MB result produce 200 tiny files.
• Frequent small MERGEs and appends into lakehouse tables.

Fixes work at write time and afterwards. At write time: coarser partitioning, coalesce/repartition before writing, and features such as Delta’s optimized writes or Spark AQE’s partition coalescing. Afterwards: compaction, which rewrites many small files into a few large ones in one transaction, such as OPTIMIZE in Delta Lake, rewrite_data_files in Iceberg, or Databricks auto-compaction. Old files stay until VACUUM or snapshot expiry removes them.

Example

Thirty micro-batch appends leave 60 small files; one OPTIMIZE compacts them into a single file with the same rows (PySpark 4.2.0 with delta-spark 4.4.0).

import os, tempfile
from pyspark.sql import SparkSession
from delta import configure_spark_with_delta_pip
from delta.tables import DeltaTable

builder = (SparkSession.builder.master("local[2]").appName("glossary-small-files")
           .config("spark.sql.extensions", "io.delta.sql.DeltaSparkSessionExtension")
           .config("spark.sql.catalog.spark_catalog", "org.apache.spark.sql.delta.catalog.DeltaCatalog"))
spark = configure_spark_with_delta_pip(builder).getOrCreate()
spark.sparkContext.setLogLevel("ERROR")
path = os.path.join(tempfile.mkdtemp(), "events")

for batch in range(30):   # 30 micro-batches, like a streaming job writing every few seconds
    spark.range(batch * 100, batch * 100 + 100).repartition(2).write.format("delta").mode("append").save(path)

table = DeltaTable.forPath(spark, path)
print("files before:", table.detail().select("numFiles").first()[0])
metrics = table.optimize().executeCompaction().select("metrics.numFilesAdded", "metrics.numFilesRemoved").first()
print("OPTIMIZE added", metrics[0], "file, removed", metrics[1])
print("files after:", DeltaTable.forPath(spark, path).detail().select("numFiles").first()[0])
print("rows:", spark.read.format("delta").load(path).count())
spark.stop()

files before: 60
OPTIMIZE added 1 file, removed 60
files after: 1
rows: 3000

Common mistakes

• Partitioning by a column with thousands of values, such as user_id or a timestamp.
• Never compacting a table written by a streaming job.
• Calling repartition(1) on large outputs to avoid small files, which creates one huge, slow task.
• Compacting but never vacuuming, so storage keeps every old small file.