Explain Kafka partitions and consumer groups.

Detailed explanation

• Partitions are the unit of both ordering and parallelism. Each has its own sequence of offsets.
• Keys route records: the same key always maps to the same partition while the partition count is unchanged.
• Consumer groups split partitions among members. Two different groups each receive every record independently.

Example

A topic orders has 6 partitions.

Consumers in group billing	Partitions per consumer
1	6
3	2 each
6	1 each
8	6 consumers get 1, 2 are idle

A second group analytics reads all 6 partitions on its own, unaffected by billing.

Failure and rebalancing

If a consumer crashes or stops sending heartbeats, the group rebalances and its partitions move to other members, which resume from the last committed offsets. Records processed but not yet committed are processed again, which is why at-least-once consumers need idempotent processing.

With the classic eager protocol every member pauses during a rebalance; cooperative-sticky assignment and the KIP-848 consumer protocol (group.protocol=consumer, GA since Kafka 4.0) move only the partitions that change owner. Kafka 4.2 also added share groups, where members share partitions and acknowledge records individually like a queue; they are a different tool from ordinary consumer groups and give up ordering.

Trade-offs

• More partitions allow more parallel consumers but cost broker resources and, on keyed topics, cannot be added later without moving keys.
• A narrow key (order ID) spreads load; a broad key (country) creates hot partitions but orders more events together.
• Separate groups give independent reads and offsets; sharing a group ID between two applications splits the data between them.

Common mistakes

1. Claiming Kafka guarantees ordering across a whole topic.
2. Adding consumers beyond the partition count to increase throughput.
3. Picking a skewed key so one partition receives most of the traffic.