Data File Formats Explained: Row vs Columnar, CSV, JSON, Avro, Parquet, ORC and Arrow

A file format decides how data is laid out on disk: as text or binary, with or without a schema, and, most importantly, row by row or column by column. The same table can be 20 MB as JSON and 1.3 MB as Parquet, and reading one column can be many times faster in a columnar format. Choosing the right format is one of the cheapest performance wins in data engineering.

Row-based vs columnar, in one picture

Take three orders:

order_id	city	amount
1	Pune	25.00
2	Leeds	40.00
3	Pune	12.50

A row-based format writes each row together:

1, Pune, 25.00 | 2, Leeds, 40.00 | 3, Pune, 12.50

A columnar format writes each column together:

1, 2, 3 | Pune, Leeds, Pune | 25.00, 40.00, 12.50

What follows from that:

• Row-based is natural for writing one record at a time and for reading whole records. Appending a new order is just adding to the end. This suits logs, messages and data being moved between systems.
• Columnar is natural for analytics. To total amount, a reader can jump to that column and skip the rest. Values of the same type sit next to each other, so they compress well (Pune, Pune, Pune... compresses to almost nothing). The cost is that writing single rows is awkward, so columnar files are written in batches.

The same split explains OLTP vs OLAP databases: row stores for transactions, column stores for analysis.

The formats you will meet

Format	Layout	Text or binary	Schema	Typical use
CSV	Row	Text	None (header names only)	Exports, spreadsheets, simple hand-offs
JSON / JSON Lines	Row	Text	None, each record describes itself	APIs, logs, semi-structured events
Avro	Row	Binary	Stored in the file header	Kafka messages, raw ingestion with schema evolution
Protobuf	Row (one message)	Binary	In a separate .proto file	Service-to-service messages, gRPC, some Kafka topics
Parquet	Columnar	Binary	Stored in the file footer	Data lake and lakehouse tables, analytics
ORC	Columnar	Binary	Stored in the file footer	Hive-based platforms, some lakehouse tables
Arrow (IPC / Feather)	Columnar	Binary	Stored in the file	Fast exchange between tools, in-memory processing

CSV

Plain text, one line per row, values separated by commas. Every tool can read it, which is its biggest strength. Its weaknesses are serious for pipelines: there are no data types (is 01234 a number or a code?), no standard for dates or nulls, and quoting rules differ between tools, so a comma or line break inside a value can shift columns. Use CSV at the edges, when a person or an old system needs it, and convert it early.

JSON and JSON Lines

JSON stores each record with its field names, so it handles nested and changing data well. JSON Lines (one JSON object per line) is the version used for large files, because it can be split and streamed line by line. Repeating every field name on every row makes it the largest format here, and every value has to be parsed from text. It is the usual format for API responses and raw event logs. See JSON vs Parquet.

Avro

A compact binary row format. The schema, written in JSON, is stored once in the file header, and each record is stored without field names. Avro has clear rules for schema evolution: a reader with a newer schema can read old data if new fields have defaults. That makes it the classic choice for Kafka messages with a schema registry, and for landing raw data before it is turned into tables.

Protobuf

Protocol Buffers is also a binary row format, designed for messages between services. The schema lives in a .proto file that both sides compile into code, and the data does not carry the schema. It is small and fast, and common with gRPC and in some Kafka setups. It is rarely used for files in a data lake.

Parquet

The default columnar format for data lakes and lakehouses. A Parquet file is split into row groups; inside each, every column is stored as its own chunk, encoded and compressed. The footer holds the schema and statistics such as the minimum and maximum of each column per row group, so engines can skip whole row groups that cannot match a filter. Spark, Trino, DuckDB, Snowflake, BigQuery and most other engines read it. The internals are covered in Parquet vs Avro vs ORC.

ORC

A columnar format from the Hive ecosystem, with a similar design to Parquet (stripes instead of row groups, with indexes and statistics). It performs similarly; Parquet has wider support outside Hive-based platforms, which is why most new projects choose Parquet.

Arrow

Apache Arrow is first an in-memory columnar format: a standard way to lay out columns in memory so that tools such as pandas, Polars, DuckDB and Spark can pass data to each other without converting it. Arrow IPC files (also called Feather) store that layout on disk. They are very fast to read, but are meant for exchange and caching rather than long-term storage, where Parquet compresses better.

Table formats are a different layer

Delta Lake, Apache Iceberg and Apache Hudi are table formats, not file formats. They store data in Parquet files and add a transaction log or metadata layer on top, which brings ACID commits, time travel and schema evolution for the whole table. See Delta vs Iceberg vs Hudi.

Measure it yourself

This experiment writes the same 200,000 orders in six formats, then reads back only the amount column from each file and totals it:

import os, time, tempfile, json, random
import pyarrow as pa, pyarrow.csv as pcsv, pyarrow.json as pjson, pyarrow.parquet as pq, pyarrow.orc as orc, pyarrow.feather as feather
import fastavro

# 200,000 synthetic orders: an id, a date, a city, a status and an amount.
rng = random.Random(42)
n = 200_000
cities = ["Pune", "Leeds", "Austin", "Lagos", "Osaka", "Lima"]
statuses = ["placed", "paid", "shipped", "cancelled"]
table = pa.table({
    "order_id": pa.array(range(1, n + 1), pa.int64()),
    "order_date": pa.array([f"2026-{rng.randint(1, 12):02d}-{rng.randint(1, 28):02d}" for _ in range(n)]),
    "city": pa.array([rng.choice(cities) for _ in range(n)]),
    "status": pa.array([rng.choice(statuses) for _ in range(n)]),
    "amount": pa.array([round(rng.uniform(5, 500), 2) for _ in range(n)]),
})
rows = table.to_pylist()
d = tempfile.mkdtemp()
path = lambda name: os.path.join(d, name)

pcsv.write_csv(table, path("orders.csv"))
with open(path("orders.jsonl"), "w") as f:
    for r in rows:
        f.write(json.dumps(r) + "\n")
schema = {"type": "record", "name": "Order", "fields": [
    {"name": "order_id", "type": "long"}, {"name": "order_date", "type": "string"},
    {"name": "city", "type": "string"}, {"name": "status", "type": "string"},
    {"name": "amount", "type": "double"}]}
with open(path("orders.avro"), "wb") as f:
    fastavro.writer(f, schema, rows, codec="deflate")
pq.write_table(table, path("orders.parquet"), compression="zstd")
orc.write_table(table, path("orders.orc"), compression="zstd")
feather.write_feather(table, path("orders.arrow"), compression="zstd")

def sum_amount(name):
    """Total of one column, reading the file the way each format allows."""
    p = path(name)
    if name.endswith(".csv"):
        return pcsv.read_csv(p, convert_options=pcsv.ConvertOptions(include_columns=["amount"]))["amount"].to_pylist()
    if name.endswith(".jsonl"):
        return pjson.read_json(p)["amount"].to_pylist()   # JSON has no way to skip the other fields
    if name.endswith(".avro"):
        with open(p, "rb") as f:
            return [r["amount"] for r in fastavro.reader(f)]
    if name.endswith(".parquet"):
        return pq.read_table(p, columns=["amount"])["amount"].to_pylist()
    if name.endswith(".orc"):
        return orc.read_table(p, columns=["amount"])["amount"].to_pylist()
    return feather.read_table(p, columns=["amount"])["amount"].to_pylist()

totals = set()
print(f"{'file':16}{'size KB':>9}{'read 1 column (ms)':>21}")
for name in ["orders.csv", "orders.jsonl", "orders.avro", "orders.parquet", "orders.orc", "orders.arrow"]:
    times = []
    for _ in range(3):                       # best of three, so one-off start-up costs do not count
        start = time.perf_counter()
        totals.add(round(sum(sum_amount(name)), 2))
        times.append((time.perf_counter() - start) * 1000)
    print(f"{name:16}{os.path.getsize(path(name)) // 1024:>9}{min(times):>21.0f}")
print("every file gives the same total:", len(totals) == 1)

file              size KB   read 1 column (ms)
orders.csv           8488                   21
orders.jsonl        20211                   71
orders.avro          2115                  351
orders.parquet       1340                   10
orders.orc           1527                    9
orders.arrow         3250                   10

How to read the results:

• Size. JSON Lines is the biggest because every row repeats the field names. CSV drops the names but stores numbers as text. Avro is binary, so it is much smaller. Parquet and ORC are the smallest: similar values stored together compress far better.
• Speed. The three columnar formats read one column in about 10 ms, because they skip the other four columns entirely. CSV and JSON have to parse every byte of every row to find the amount values.
• Avro looks slow here, but that is mostly Python: fastavro builds a Python dictionary for every one of the 200,000 rows, while the other readers stay in compiled code. In Java or Spark, Avro reads are much faster. Even so, a row format always decodes every field, so for one-column analytics it cannot match a columnar format.
• Arrow is larger than Parquet because Arrow’s layout is built for fast access in memory, not for maximum compression.

Your numbers will differ with your data, codec and machine. The pattern does not: text formats are big, row formats read everything, columnar formats read only what the query needs.

Which format should you use?

Situation	Good choice	Why
Analytics tables in a lake or lakehouse	Parquet (inside Delta or Iceberg)	Columnar, compressed, statistics for skipping, read by every engine
Kafka messages and event streams	Avro or Protobuf with a schema registry	Compact rows, written one at a time, safe schema changes
Raw landing zone from APIs	JSON Lines (keep the original), then Parquet	Keeps the source exactly; convert for querying
Hand-off to a person or spreadsheet	CSV	Everyone can open it
Passing data between Python tools, caching	Arrow / Feather	No conversion cost, very fast reads
An existing Hive platform built on ORC	ORC	Consistency with what is there

A common lakehouse pattern: land the source as JSON or Avro in the bronze layer, then write cleaned Parquet-based tables in silver and gold. See medallion architecture.

Common mistakes

1. Running analytics on CSV or JSON in the lake. Every query parses everything. Convert to Parquet once.
2. Comparing sizes without fixing the codec. Writers use different default compression, so a table of sizes often compares codecs, not formats.
3. Writing many tiny Parquet files. Columnar formats need decent-sized files (often 128 MB to 1 GB) to compress and skip well.
4. Calling Delta Lake or Iceberg a file format. They are table formats on top of Parquet.
5. Trusting CSV types. Leading zeros, dates and empty strings get changed by tools that guess types.

What interviewers look for

For “row-based vs columnar formats” or “Parquet vs Avro vs CSV”:

• The layout difference in one sentence, and why columnar suits analytics.
• One good use for each format, especially Avro for streaming and Parquet for analytics.
• Why columnar files are smaller: same-type values together, encoding and compression.
• Bonus: Parquet row groups and footer statistics, Avro schema evolution, and the difference between file formats and table formats.

Practise with the data engineering interview questions.

Key takeaway

Row-based formats (CSV, JSON, Avro, Protobuf) store whole records together and suit writing, messaging and moving data. Columnar formats (Parquet, ORC, Arrow) store each column together, so analytical queries read less and files compress far better. Land data in the format the source gives you, store analytics tables as Parquet, use Avro or Protobuf for streams, and keep CSV for people. Next: JSON vs Parquet in a real pipeline.