What is a Dead-letter Queue?

How it works

A poison message is a record that fails every time it is processed. Without a DLQ, a streaming consumer has two bad choices: retry it forever, which blocks the partition and grows lag, or skip it silently, which loses data. A DLQ adds a third choice: move it aside and continue.

A good DLQ design separates errors by type:

• Transient errors (a timeout, a throttled API): retry with backoff, possibly through retry topics with increasing delays, before giving up.
• Permanent errors (bad schema, invalid values, unparseable bytes): send straight to the DLQ.

Each dead letter should carry enough context to act on: the original payload unchanged, source topic, partition and offset, the error message, a timestamp and the pipeline version. Kafka Connect supports this with errors.tolerance=all and errors.deadletterqueue.topic.name; AWS SQS and SNS have native DLQs with a max-receive count; in batch pipelines the same idea is a quarantine table for rows that fail checks.

A DLQ is only useful if someone watches it: alert on its size, assign an owner, and have a replay path that sends fixed records back through the normal, idempotent pipeline.

Example

Two of four messages are bad. They are parked with their offset and error, and the good ones load without stopping the stream (Python 3.11).

import json

messages = [
    (0, '{"order_id": 1, "amount": 40}'),
    (1, '{"order_id": 2, "amount": "twenty"}'),    # bad type
    (2, '{"order_id": 3, "amount": 10'),            # broken JSON
    (3, '{"order_id": 4, "amount": 25}'),
]
loaded, dead_letters = [], []

def process(raw):
    event = json.loads(raw)
    if not isinstance(event["amount"], (int, float)):
        raise TypeError(f"amount must be a number, got {event['amount']!r}")
    return event

for offset, raw in messages:
    try:
        loaded.append(process(raw))
    except Exception as e:                       # park it with context, keep the stream moving
        dead_letters.append({"offset": offset, "payload": raw,
                             "error": f"{type(e).__name__}: {e}"})

print("loaded:", [e["order_id"] for e in loaded])
for d in dead_letters:
    print("DLQ:", d["offset"], d["error"])

loaded: [1, 4]
DLQ: 1 TypeError: amount must be a number, got 'twenty'
DLQ: 2 JSONDecodeError: Expecting ',' delimiter: line 1 column 29 (char 28)

Common mistakes

• Sending transient failures to the DLQ, so a short outage fills it with good records.
• Storing only the error message and not the original payload and offset, so records cannot be replayed.
• Having no alert or owner, so the DLQ silently grows for months.
• Replaying into a non-idempotent sink, which duplicates records that had partly succeeded.