What is a Consumer Group?

How it works

Consumers that use the same group.id form one group. A broker acting as group coordinator tracks the members and assigns partitions among them. The rules:

• Each partition goes to exactly one consumer in the group, which keeps per-partition ordering.
• One consumer can own several partitions.
• A group can use at most as many consumers as there are partitions; extra consumers sit idle as standbys.
• Each group commits its own offsets, so a billing group and an analytics group can read the same topic at different speeds.

When a member joins, leaves, crashes or stops polling for longer than max.poll.interval.ms, the group rebalances: partitions are reassigned. Older “eager” rebalancing stopped every consumer during the move. Cooperative rebalancing (CooperativeStickyAssignor) and, since Kafka 4.0, the new consumer group protocol (KIP-848) move only the partitions that need to change. Static membership (group.instance.id) avoids rebalances when a consumer restarts quickly.

Example

Kafka’s range assignment of a 6-partition topic: two consumers get three partitions each, three get two each, and a seventh consumer has nothing to do (Python 3.11).

def range_assign(partitions, consumers):
    """Kafka's RangeAssignor for one topic: contiguous blocks, the first consumers get one extra."""
    consumers = sorted(consumers)
    per, extra = divmod(partitions, len(consumers))
    result, start = {}, 0
    for i, c in enumerate(consumers):
        n = per + (1 if i < extra else 0)
        result[c] = list(range(start, start + n))
        start += n
    return result

for group in [["c1", "c2"], ["c1", "c2", "c3"], ["c1", "c2", "c3", "c4", "c5", "c6", "c7"]]:
    print(f"{len(group)} consumers:", range_assign(6, group))

2 consumers: {'c1': [0, 1, 2], 'c2': [3, 4, 5]}
3 consumers: {'c1': [0, 1], 'c2': [2, 3], 'c3': [4, 5]}
7 consumers: {'c1': [0], 'c2': [1], 'c3': [2], 'c4': [3], 'c5': [4], 'c6': [5], 'c7': []}

Common mistakes

• Scaling consumers beyond the partition count and expecting more throughput.
• Doing slow work between polls, so consumers exceed max.poll.interval.ms and trigger rebalance storms.
• Reusing one group.id for two different applications, which splits the partitions between them.
• Committing offsets before processing during a rebalance, which can skip records.