What is a Kafka Topic?

How it works

A topic such as orders is split into partitions, numbered from 0. Each partition is an ordered, append-only log on disk, and every record in it gets a sequential offset. Partitions are the unit of parallelism and ordering:

• Writes: a record with a key goes to hash(key) % partitions, so every event for customer-17 lands in the same partition and stays in order. A record with no key is spread across partitions (the sticky partitioner fills one batch at a time).
• Reads: within a consumer group, each partition is read by exactly one consumer, so a topic with 12 partitions supports at most 12 parallel consumers in that group.
• Durability: each partition has a leader and follower replicas on other brokers (replication.factor, typically 3). With acks=all and min.insync.replicas=2, a write survives the loss of one broker.

Reading does not delete anything. Records are removed by retention (retention.ms or retention.bytes) or, on compacted topics, only older values per key are removed, so many consumer groups can read the same topic independently and replay it from any offset.

Example

Kafka’s default partitioner for keyed records, reproduced in Python (it matches Kafka’s own murmur2 test values). The same key always maps to the same partition, but changing the partition count moves some keys (Python 3.11).

def murmur2(data: bytes) -> int:
    """Kafka's murmur2 hash (org.apache.kafka.common.utils.Utils.murmur2)."""
    m, r, length = 0x5BD1E995, 24, len(data)
    h = (0x9747B28C ^ length) & 0xFFFFFFFF
    for i in range(0, length - length % 4, 4):
        k = int.from_bytes(data[i:i + 4], "little")
        k = (k * m) & 0xFFFFFFFF
        k ^= k >> r
        k = (k * m) & 0xFFFFFFFF
        h = ((h * m) & 0xFFFFFFFF) ^ k
    tail = data[length - length % 4:]
    if len(tail) == 3: h ^= tail[2] << 16
    if len(tail) >= 2: h ^= tail[1] << 8
    if len(tail) >= 1:
        h ^= tail[0]
        h = (h * m) & 0xFFFFFFFF
    h ^= h >> 13
    h = (h * m) & 0xFFFFFFFF
    h ^= h >> 15
    return h

def partition_for(key: str, partitions: int) -> int:
    return (murmur2(key.encode()) & 0x7FFFFFFF) % partitions   # Kafka's default for keyed records

for key in ["customer-17", "customer-42", "customer-17"]:
    print(key, "-> partition", partition_for(key, 6))

# Adding partitions changes where existing keys go, which breaks per-key ordering.
for key in ["customer-42", "customer-99"]:
    print(f"{key}: 6 partitions -> {partition_for(key, 6)}, 12 partitions -> {partition_for(key, 12)}")

customer-17 -> partition 4
customer-42 -> partition 3
customer-17 -> partition 4
customer-42: 6 partitions -> 3, 12 partitions -> 9
customer-99: 6 partitions -> 4, 12 partitions -> 4

Common mistakes

• Expecting ordering across a whole topic; Kafka only orders records within one partition.
• Adding partitions to a keyed topic in production, which remaps keys and breaks per-key ordering during the change.
• Choosing a low-cardinality or skewed key, so one partition gets most of the traffic.
• Treating a topic like a queue that empties once read, then being surprised by disk use or by replays after a reset.