What is Avro?

How it works

An Avro schema lists a record’s fields and types (long, double, string, enum, arrays, maps, nested records and unions such as ["null", "string"] for optional fields). The binary encoding contains only the values, in schema order, with no field names, so records are small and fast to write. That is why the schema must be available to read the data:

• Container files (.avro) store the schema once in the header, followed by compressed blocks of records.
• Kafka messages store only a small schema ID; the full schema lives in a schema registry.

Avro’s strength is schema resolution: a reader supplies its own schema, and Avro maps the writer’s data onto it by field name, filling added fields from their defaults and ignoring removed ones. That is what makes compatible schema evolution possible.

Avro is row-oriented, so it suits writing and reading whole records: event streams, CDC from Debezium, and raw landing zones. Parquet is columnar and suits analytical scans. A common pattern is Avro on the wire and in the bronze layer, Parquet or Delta for analytics.

Example

One order is 72 bytes as JSON and 12 bytes as Avro binary. A container file carries its schema, so it is self-describing (Python 3.11, fastavro 1.13).

import io, json
from fastavro import parse_schema, reader, schemaless_writer, writer

schema = parse_schema({
    "type": "record", "name": "Order", "namespace": "shop",
    "fields": [
        {"name": "order_id", "type": "long"},
        {"name": "status", "type": {"type": "enum", "name": "Status", "symbols": ["PLACED", "SHIPPED"]}},
        {"name": "amount", "type": "double"},
        {"name": "coupon", "type": ["null", "string"], "default": None},
    ],
})
order = {"order_id": 1001, "status": "SHIPPED", "amount": 499.0, "coupon": None}

one = io.BytesIO()
schemaless_writer(one, schema, order)                 # how a Kafka message body is encoded
print("JSON bytes:", len(json.dumps(order).encode()), "| Avro bytes:", len(one.getvalue()))

f = io.BytesIO()
writer(f, schema, [order, {**order, "order_id": 1002, "coupon": "DIWALI"}])  # container file: schema in header
f.seek(0)
r = reader(f)
print("schema stored in file:", r.writer_schema["name"])
print([rec["coupon"] for rec in r])

JSON bytes: 72 | Avro bytes: 12
schema stored in file: shop.Order
[None, 'DIWALI']

Common mistakes

• Adding a field without a default, which breaks readers of older data.
• Using Avro files for analytical tables queried by a few columns, where Parquet reads far less.
• Sending Avro to Kafka without a schema registry, then losing track of which schema wrote which messages.
• Changing an enum’s symbols carelessly; readers fail on symbols they do not know unless a default is set.