What does VACUUM do in Delta Lake, and what can go wrong?

Detailed explanation

DELETE, UPDATE, MERGE, OPTIMIZE and overwrites never delete data files; they add remove actions to the log, so older versions can still read the files. VACUUM is the step that actually frees the storage. It deletes files that are:

1. not referenced by the current version, and
2. older than the retention threshold.

It is not automatic in open-source Delta; schedule it. VACUUM ... DRY RUN lists what would be deleted. VACUUM LITE (Delta 3.3 and later) finds candidates from the log rather than listing the whole directory, which is much cheaper on large object-store tables.

Example

In a local test, t.vacuum(0) failed with DELTA_VACUUM_RETENTION_PERIOD_TOO_SHORT. After disabling spark.databricks.delta.retentionDurationCheck.enabled and running VACUUM ... RETAIN 0 HOURS, the current version still read correctly, but reading version 0 failed with FAILED_READ_FILE.FILE_NOT_EXIST.

VACUUM events DRY RUN;                       -- inspect first
VACUUM events;                               -- default 7-day retention
ALTER TABLE events SET TBLPROPERTIES ('delta.deletedFileRetentionDuration' = 'interval 14 days');

Trade-offs and pitfalls

• Long readers and streams. A job that started before a compaction still reads the pre-compaction files; a lagging stream may need files from days ago. Retention must cover both.
• Restore window. If you promise “we can roll back a week”, keep at least a week, and keep delta.logRetentionDuration (30 days by default) at least as long.
• Cost. Longer retention keeps more removed files in storage, especially on tables rewritten often by MERGE or OPTIMIZE.
• Erasure. With deletion vectors, deleted rows remain in the Parquet files until they are rewritten (REORG TABLE ... APPLY (PURGE) or compaction); only then can VACUUM remove the old files after the retention period.

Common mistakes

1. Disabling the retention check to “save storage” while jobs are running.
2. Assuming a count() on an old version proves its files exist; Delta can answer it from log statistics without opening files.
3. Treating time travel as long-term backup.