How do acks, replication factor and min.insync.replicas decide Kafka durability?

Detailed explanation

Three settings work together:

Setting	Where	Meaning
replication.factor	Topic	Number of replicas per partition, on different brokers
acks	Producer	0: do not wait; 1: leader has it; all (-1): every current ISR member has it
min.insync.replicas	Topic or broker (default 1)	Minimum ISR size for the leader to accept an acks=all write

The ISR (in-sync replicas) is the set of replicas caught up with the leader. A follower that falls behind for replica.lag.time.max.ms (30 seconds by default) leaves it. acks=all waits for the ISR as it is now, so on its own it can degrade to “the leader alone has it”. min.insync.replicas puts a floor under that: if the ISR is smaller, the leader refuses acks=all writes, and the producer retries until delivery.timeout.ms expires. That turns silent loss into a visible error.

Two more pieces complete the picture:

• Unclean leader election (unclean.leader.election.enable, off by default) would let an out-of-sync replica become leader if every ISR member is gone, restoring availability by discarding data. Keep it off for data you must not lose.
• High watermark: consumers only read records replicated to the whole ISR, so a consumer never sees a record that a clean leader change could remove. A producer using acks=1 can, however, be told “success” for a record that is later truncated.

Example

This model lists, for RF 3, what happens to writes when brokers fail:

def outcome(rf, min_isr, acks, alive):
    """alive: replicas still in sync (including the leader)."""
    if alive == 0:
        return "partition offline"
    if acks == "all":
        if alive < min_isr:
            return "REJECTED (NOT_ENOUGH_REPLICAS)"
        return f"ok, on {alive} {'copy' if alive == 1 else 'copies'}"
    if acks == "1":
        return "ok, on 1 copy until followers catch up (loss if the leader dies now)"
    return "fire and forget: no confirmation at all"

for min_isr in (1, 2, 3):
    print(f"RF=3, min.insync.replicas={min_isr}, acks=all")
    for alive in (3, 2, 1):
        print(f"   {alive} in sync -> {outcome(3, min_isr, 'all', alive)}")
print("RF=3, acks=1 with 3 in sync ->", outcome(3, 2, "1", 3))

RF=3, min.insync.replicas=1, acks=all
   3 in sync -> ok, on 3 copies
   2 in sync -> ok, on 2 copies
   1 in sync -> ok, on 1 copy
RF=3, min.insync.replicas=2, acks=all
   3 in sync -> ok, on 3 copies
   2 in sync -> ok, on 2 copies
   1 in sync -> REJECTED (NOT_ENOUGH_REPLICAS)
RF=3, min.insync.replicas=3, acks=all
   3 in sync -> ok, on 3 copies
   2 in sync -> REJECTED (NOT_ENOUGH_REPLICAS)
   1 in sync -> REJECTED (NOT_ENOUGH_REPLICAS)
RF=3, acks=1 with 3 in sync -> ok, on 1 copy until followers catch up (loss if the leader dies now)

The middle block is the standard configuration: one broker down still accepts writes on two copies; two down rejects writes instead of accepting them on one. The first block accepts a write on a single copy, which a second failure would lose. The last block stops writes during every rolling restart.

The matching topic creation:

kafka-topics.sh --bootstrap-server localhost:9092 --create --topic payments \
  --partitions 12 --replication-factor 3 --config min.insync.replicas=2

Trade-offs and pitfalls

• Durability costs availability: with min ISR 2, two broker failures stop writes to affected partitions. That is the intended behaviour for financial or CDC data; metrics topics may accept min.insync.replicas=1.
• Latency: acks=all waits for the slowest ISR follower, so a degraded follower slows every write until it is removed from the ISR.
• Rack awareness: three replicas in one availability zone do not survive a zone outage. Set broker.rack and use three zones.
• Do not “fix” NotEnoughReplicasException by lowering min.insync.replicas or switching to acks=1; fix the brokers.
• Kafka 4.x detail: the high watermark does not advance while the ISR is below min.insync.replicas, and eligible leader replicas (on by default for new clusters from 4.1) let the controller pick a safe leader in more failure cases without an unclean election.

Common mistakes

1. Saying “acks=all means all replicas”. It means all in-sync replicas.
2. Leaving the broker default min.insync.replicas=1 and assuming acks=all is durable.
3. Setting min.insync.replicas equal to the replication factor.