What is a Schema Registry?

How it works

Schemas are registered under a subject, usually <topic>-value and <topic>-key. When a producer serializes a message, its serializer registers or looks up the schema and writes a 5-byte header (a magic byte and the schema ID) before the Avro, Protobuf or JSON payload. A consumer reads the ID, fetches and caches that schema, and decodes the message, resolving it against its own reader schema.

The registry enforces a compatibility mode per subject:

Mode	Guarantee	Safe changes (Avro)
BACKWARD (default)	New consumers can read old data	Delete fields; add fields with defaults
FORWARD	Old consumers can read new data	Add fields; delete fields with defaults
FULL	Both directions	Add or delete fields with defaults
_TRANSITIVE variants	Against all versions, not just the last	Same, checked against history

BACKWARD lets you upgrade consumers first; FORWARD lets you upgrade producers first. A registration that breaks the mode is rejected with HTTP 409, which stops a breaking change at deploy time instead of in production. Common implementations are Confluent Schema Registry, AWS Glue Schema Registry, Apicurio and the registries built into managed Kafka services.

Example

A BACKWARD compatibility check: data written with version 1 is read with a candidate version 2. Adding tier with a default works; adding it without a default fails, which is the kind of change a registry rejects (Python 3.11, fastavro 1.13).

import io
from fastavro import parse_schema, schemaless_reader, schemaless_writer

v1 = {"type": "record", "name": "Customer", "fields": [
    {"name": "id", "type": "long"},
    {"name": "city", "type": "string"}]}
v2_ok = {"type": "record", "name": "Customer", "fields": [
    {"name": "id", "type": "long"},
    {"name": "city", "type": "string"},
    {"name": "tier", "type": "string", "default": "standard"}]}   # new field WITH a default
v2_bad = {"type": "record", "name": "Customer", "fields": [
    {"name": "id", "type": "long"},
    {"name": "city", "type": "string"},
    {"name": "tier", "type": "string"}]}                          # new field, no default

def backward_compatible(new, old):
    """BACKWARD check: can a consumer on `new` read a message written with `old`?"""
    buf = io.BytesIO()
    schemaless_writer(buf, parse_schema(old), {"id": 7, "city": "Pune"})
    buf.seek(0)
    try:
        return schemaless_reader(buf, parse_schema(old), parse_schema(new))
    except Exception as e:
        return f"incompatible ({type(e).__name__})"

print("v2 with default:", backward_compatible(v2_ok, v1))
print("v2 without default:", backward_compatible(v2_bad, v1))

v2 with default: {'id': 7, 'city': 'Pune', 'tier': 'standard'}
v2 without default: incompatible (SchemaResolutionError)

Common mistakes

• Setting compatibility to NONE to get a deploy through, which lets breaking changes reach consumers.
• Using BACKWARD while upgrading producers before consumers, the order that mode does not protect.
• Auto-registering schemas from every producer build in production instead of registering through CI.
• Deleting old schema versions that messages still on the topic refer to.