What is Apache Iceberg?

How it works

An Iceberg table is data files (usually Parquet) plus three layers of metadata:

1. A catalog (Hive Metastore, AWS Glue, a REST catalog such as Polaris or Unity Catalog, or a JDBC database) stores one pointer: the table’s current metadata file.
2. The metadata file holds the schema, partition spec and the list of snapshots.
3. Each snapshot points to manifest lists and manifests, which list data files with per-column statistics used for pruning.

A commit writes new data and metadata files, then atomically swaps the catalog pointer. Readers always see one complete snapshot, and old snapshots stay readable for time travel until they are expired.

Two features set Iceberg apart. Hidden partitioning lets you partition by a transform such as days(ts) or bucket(16, id); queries simply filter on ts, and you can change the partition scheme later without rewriting old data. Schema evolution tracks columns by ID rather than name, so adding, dropping, renaming or reordering columns is a metadata-only change that never resurrects old data. Iceberg’s broad engine support is why AWS (S3 Tables, Athena), Snowflake and Databricks all read and write it.

Example

Two appends with a schema change in between. Old rows show None for the new column, and the first snapshot is still readable (Python 3.11, PyIceberg 0.12 with a local SQLite catalog).

import tempfile
import pyarrow as pa
from pyiceberg.catalog.sql import SqlCatalog
from pyiceberg.types import StringType

wh = tempfile.mkdtemp()
catalog = SqlCatalog("local", uri=f"sqlite:///{wh}/catalog.db", warehouse=f"file://{wh}")
catalog.create_namespace("shop")

orders = pa.table({"id": pa.array([1, 2], pa.int32()), "amount": [40.0, 25.0]})
table = catalog.create_table("shop.orders", schema=orders.schema)
table.append(orders)                                   # snapshot 1

with table.update_schema() as update:                  # schema evolution: metadata only
    update.add_column("country", StringType())
table.append(pa.table({"id": pa.array([3], pa.int32()), "amount": [10.0], "country": ["IN"]}))

print(table.scan().to_arrow().sort_by("id").to_pylist())
first = table.history()[0].snapshot_id
print("rows at first snapshot:", table.scan(snapshot_id=first).to_arrow().num_rows)
print("snapshots:", len(table.history()), "| schema fields:", [f.name for f in table.schema().fields])

[{'id': 1, 'amount': 40.0, 'country': None}, {'id': 2, 'amount': 25.0, 'country': None}, {'id': 3, 'amount': 10.0, 'country': 'IN'}]
rows at first snapshot: 2
snapshots: 2 | schema fields: ['id', 'amount', 'country']

Common mistakes

• Never expiring snapshots or removing orphan files, so metadata and storage grow without limit.
• Adding a derived event_date column for partitioning instead of using a hidden days(ts) transform.
• Pointing two engines at different catalogs for the same table, so their commits overwrite each other’s pointer.
• Streaming many small commits without compaction (rewrite_data_files), which slows planning and reads.