How is a Parquet file organised, and how does that speed up queries?

Detailed explanation

Level	What it is	Why it matters
File	Data plus a footer; the magic bytes PAR1 at both ends	The footer is read first, so planning needs no full read
Row group	A horizontal slice of rows	Unit of parallelism (one task can read one group) and of skipping
Column chunk	One column’s values within a row group	Read only the columns a query uses
Page	Sub-unit of a chunk, around 1 MB	Unit of encoding and compression; page-level indexes allow finer skipping

Encodings come before compression: a low-cardinality column (country, status) becomes a small dictionary plus bit-packed indexes, and repeated values become runs. Compression codecs (Snappy by default in Spark, zstd as a common alternative) then work on already compact data. Because compression is per page, a file stays splittable at row-group boundaries whatever the codec.

Example

Reading the footer with PyArrow for a file sorted by order_id:

import pyarrow.parquet as pq
meta = pq.ParquetFile("orders.parquet").metadata
for g in range(meta.num_row_groups):
    s = meta.row_group(g).column(0).statistics
    print(g, s.min, s.max)
# 0 0 75378
# 1 75379 150757
# 2 150758 199999

WHERE order_id >= 180000 reads only group 2. In Spark’s plan this shows as PushedFilters: [GreaterThanOrEqual(order_id,180000)] and a ReadSchema listing only the selected columns. Spark still applies the filter to the rows it reads, because statistics only rule groups out.

Trade-offs and pitfalls

• Small files. Each file has a footer to fetch and tiny row groups that compress and skip poorly. Compact them.
• Unsorted data. If every row group spans the full range of a column, statistics skip nothing. Sort, Z-order or cluster on common filter columns.
• Wide row groups vs parallelism. Very large row groups reduce parallelism and skipping granularity; very small ones add overhead.
• Table formats add a layer. Delta stores per-file min/max in its log, so it can skip whole files before opening any footer.

Common mistakes

1. Saying Parquet “stores data by column” without mentioning row groups, which are what make it splittable and skippable.
2. Expecting a filter on any column to skip data regardless of layout.
3. Comparing Parquet and ORC sizes without noticing different default codecs.