What is Consumer Lag?

How it works

Each Kafka partition is an ordered log. The broker knows its log-end offset (where the next message will go), and each consumer group stores a committed offset per partition (where it will resume). Lag is the gap between them, per partition. kafka-consumer-groups.sh --describe --group <name> prints it, and tools such as Burrow, Kafka exporters for Prometheus or cloud consoles track it over time. Some teams also convert it to time lag, the age of the oldest unprocessed message, which is easier to alert on.

Lag grows when the consume rate falls below the produce rate. Usual causes:

• Slow processing: a heavy transform, a slow sink or synchronous calls per message.
• Too few consumers: parallelism is capped by partition count, so extra consumers beyond it sit idle.
• Skewed partitions: one hot key sends most traffic to one partition, so one consumer falls behind while others are idle.
• Rebalances: consumers keep joining and leaving (often because processing exceeds max.poll.interval.ms), and nothing is consumed during each rebalance.

Fixes match the cause: batch writes to the sink, add consumers up to the partition count, add partitions or a better key, and tune poll settings.

Example

Lag per partition from log-end and committed offsets. The total looks large, but nearly all of it is on partition 2, which points to a hot partition or a stuck consumer rather than too few consumers (Python 3.11).

# Log-end offsets on the broker and the group's committed offsets, per partition.
end_offsets = {0: 1_204_330, 1: 1_198_870, 2: 1_310_442}
committed   = {0: 1_204_301, 1: 1_198_870, 2: 1_052_118}

for p in sorted(end_offsets):
    print(f"partition {p}: lag {end_offsets[p] - committed[p]:>7,}")
total = sum(end_offsets[p] - committed[p] for p in end_offsets)
print(f"total lag: {total:,} records")

# At 2,000 records/s of spare consumer capacity, how long to catch up?
print(f"catch-up time: {total / 2_000 / 60:.1f} min")

partition 0: lag      29
partition 1: lag       0
partition 2: lag 258,324
total lag: 258,353 records
catch-up time: 2.2 min

Common mistakes

• Alerting on total lag only, which hides one stuck partition behind healthy ones.
• Adding consumers beyond the partition count and expecting lag to fall.
• Reading lag from an idle group whose offsets are not committed, which shows huge false lag.
• Raising max.poll.records without raising max.poll.interval.ms, which causes rebalance loops.