What is Parquet?

How it works

A Parquet file is split into row groups (typically 128 MB to 1 GB of rows). Inside each row group, every column is stored as a separate column chunk made of pages. The footer at the end of the file holds the schema and, for every column chunk, its location, encoding, compression and statistics (min, max, null count).

That layout gives analytics engines three savings:

1. Column pruning: SELECT amount reads only the amount chunks, not the whole row.
2. Row-group skipping: a filter like order_id > 90000 is checked against each row group’s min and max, and row groups that cannot match are never read. This is predicate pushdown.
3. Compression: values of one type sit next to each other, so dictionary and run-length encoding plus a codec such as Snappy or Zstandard compress them far better than row formats.

Parquet files are immutable: you cannot update one row in place. Table formats such as Delta Lake, Apache Iceberg and Apache Hudi add a transaction log on top of plain Parquet files to provide updates, deletes and ACID commits.

Example

100,000 rows written in four row groups. The read asks for one column and a range of order_id, so only one column of the last row group is decoded (PyArrow 25).

import pyarrow as pa, pyarrow.parquet as pq

table = pa.table({
    "order_id": list(range(1, 100_001)),
    "country": ["IN", "US", "UK", "DE"] * 25_000,
    "amount": [float(i % 500) for i in range(100_000)],
})
pq.write_table(table, "orders.parquet", row_group_size=25_000, compression="zstd")

meta = pq.ParquetFile("orders.parquet").metadata
print("rows:", meta.num_rows, "row groups:", meta.num_row_groups)
stats = meta.row_group(0).column(0).statistics
print("row group 0 order_id min/max:", stats.min, stats.max)

# Read one column only, and only rows that can match the filter.
t = pq.read_table("orders.parquet", columns=["amount"], filters=[("order_id", ">", 90_000)])
print("rows read:", t.num_rows, "columns read:", t.column_names)

rows: 100000 row groups: 4
row group 0 order_id min/max: 1 25000
rows read: 10000 columns read: ['amount']

Common mistakes

• Writing thousands of tiny Parquet files from a streaming or highly partitioned job; per-file overhead then dominates reads.
• Expecting min/max skipping to work on a column whose values are randomly spread across row groups; sort or cluster on filter columns first.
• Using SELECT * on wide tables, which throws away the main benefit of a columnar format.
• Changing a column’s type between files written to the same folder, so readers fail or silently cast.