Kafka and Real-Time Data Engineering

Real-time data engineering means processing events continuously instead of in scheduled batches. Kafka is the most common backbone: a durable, partitioned log that many systems can write to and read from independently. This guide is the map of the Kafka course. Each step names the idea, the lesson that teaches it and the interview questions that test it.

1. Kafka’s model

A topic is a partitioned, replicated, append-only log. Partitions provide ordering and parallelism; keys route related events to the same partition; consumer groups divide partitions among consumers. Kafka 4.x runs without ZooKeeper: cluster metadata lives in a Raft log on KRaft controllers.

Read: Topics, partitions and consumer groups · Practise: Partitions and consumer groups · How Kafka guarantees ordering · Choosing a partition count

2. Storage, retention and compaction

Each partition is a set of segment files. Retention deletes whole segments by time or size; compaction keeps the latest value per key and uses tombstones for deletes.

Read: Log storage, retention and compaction · Practise: Log compaction use cases · Sizing retention and disk

3. Producers, consumers and groups

Producers batch, compress and retry; acks=all plus the idempotent producer are the defaults. Consumers poll, process and commit offsets; the order of “process” and “commit” decides whether you lose or duplicate data. Rebalances move partitions between group members, and the KIP-848 consumer protocol makes them incremental.

Read: Producers · Consumers, offsets and lag · Consumer groups and rebalancing · Practise: Consumer lag causes and fixes · Rebalancing problems

4. Durability and delivery semantics

Replication factor 3, min.insync.replicas=2 and acks=all survive a broker loss without losing acknowledged writes. At-least-once is the practical default; make processing and sinks idempotent. Kafka’s exactly-once covers read-process-write within Kafka.

Read: Replication and ISR · Delivery semantics and exactly-once · Practise: acks and min.insync.replicas · At-least-once delivery · Exactly-once semantics

5. Integration and change data capture

Kafka Connect runs reusable source and sink connectors; Debezium streams database changes from the transaction log, keyed by primary key, so they can be applied with ordered, idempotent merges. Bad records go to a dead-letter topic instead of blocking the pipeline.

Read: Kafka Connect and Debezium · CDC patterns and failure modes · Practise: CDC with Debezium · Poison messages and DLQs · Case study: CDC platform

6. Schemas and contracts

Use a schema registry with a compatibility mode so producers cannot break consumers, register schemas from CI, and ship real breaking changes as new topics.

Read: Schema Registry and schema evolution · Practise: Schema evolution with Schema Registry

7. Stream processing

Aggregate by event time, use windows and grace periods (or watermarks) for late data, deduplicate by event ID, and checkpoint state together with offsets. Kafka Streams does this inside a JVM service; Spark and Flink run it on their own clusters.

Read: Kafka Streams basics · Batch vs streaming · Case study: Real-time analytics pipeline

8. Log or queue?

Logs retain events for replay by many consumers; queues distribute units of work. Most data pipelines want the log model; Kafka 4.2’s share groups add queue-style consumption on Kafka topics.

Read: Kafka vs message queues · Practise: Kafka vs Kinesis vs a queue

9. Operations

Monitor consumer lag (in time, not only records), under-replicated and under-min-ISR partitions, and end-to-end latency; size partitions for peak throughput and consumer parallelism; set retention longer than your longest outage.

Read: Kafka ingestion system design

Build it

Start with the Kafka → Spark → Delta streaming project, then the real-time analytics project. Revise with the Kafka cheat sheet.