Unboxing Kafka

What you'll learn

• The Origin Story — LinkedIn's Activity Stream Problem

  In 2010, LinkedIn faced a scaling crisis. They were tracking user activity (page views, profile updates, messages, searches) and needed to: 1. Reliably capture billions of events per day. 2. Route events to multiple downstream systems (H…

• The Log Abstraction

  A commit log (also called a write-ahead log or transaction log) is an append-only, ordered sequence of records. Databases use commit logs internally for durability and replication. LinkedIn's insight was to expose the log as the primary…

• Why "Kafka"?

  The name comes from Franz Kafka, the author. Jay Kreps (one of the creators) later said: "We thought the name was cool. We didn't overthink it." The system was originally called "LinkedIn Data Bus" internally before being renamed Kafka f…

• What Made Kafka Different

  By 2010, distributed message queues existed (RabbitMQ, ActiveMQ, Amazon SQS). But Kafka's design choices were unusual: 1. Messages stored on disk (not memory) — optimized for sequential I/O, which is as fast as memory for writes and near…

• Core Abstractions — Topics, Partitions, Offsets, Brokers

• Topics

  A topic is a logical category or feed name. Producers publish records to topics; consumers subscribe to topics. Example topics: - user-logins - purchase-events - page-views A topic is analogous to a table in a database, but with append-o…

• Partitions

  A topic is divided into partitions — numbered shards (0, 1, 2, ...). Each partition is an ordered, immutable log of records. Why partitions? 1. Parallelism: Multiple producers can write to different partitions in parallel. Multiple consu…

• Offsets

  Each record in a partition is assigned a sequential offset (0, 1, 2, ...). Offsets are immutable and unique per partition (not globally unique — partition 0 and partition 1 each have an offset 0). Consumers track their current offset per…

• Brokers

  A broker is a Kafka server. A Kafka cluster consists of multiple brokers (typically 3, 5, or more for production). Each broker: - Stores partitions (on disk). - Handles read/write requests from producers and consumers. - Replicates parti…

• Controller Broker

  One broker in the cluster is elected as the controller (via ZooKeeper in pre-2.8 versions, via Raft in KRaft mode since 2.8). The controller is responsible for: - Leader election — when a partition leader fails, the controller picks a ne…

• ZooKeeper (Pre-2.8) vs KRaft (2.8+)

  Pre-2.8: Kafka relied on ZooKeeper for: - Storing cluster metadata (topics, partitions, broker list). - Controller election. - Tracking ISR (in-sync replicas) for each partition. ZooKeeper added operational complexity (separate ZooKeeper…

• Storage Model — Disk as the Source of Truth

  Kafka's performance relies on sequential disk I/O being fast. Here's how it works.

• Log Segments on Disk

  Each partition is stored as a directory on disk. Inside the directory are log segments — immutable files containing batches of records. Example: Each .log file is an append-only sequence of records. New records are appended to the active…

• Why Sequential I/O is Fast

  Modern disks (even SSDs) are much faster for sequential reads/writes than random access: - Random I/O: ~100-200 IOPS (seeks + rotational latency on HDDs, random-write amplification on SSDs). - Sequential I/O: ~500 MB/s for HDDs, 1-5 GB/s…

• Zero-Copy via sendfile

  When a consumer fetches records, Kafka uses the sendfile system call (on Linux): This zero-copy optimization: 1. Reads data from the log file into the kernel's page cache (OS-managed memory cache). 2. Transfers data from the page cache d…

• Log Retention Policy

  Kafka retains records for a configurable time (default 7 days, but often set to 30+ days or even forever). Two retention modes: 1. Time-based: Delete segments older than N days (e.g., log.retention.hours=168 for 7 days). 2. Size-based: D…

• Log Compaction (Optional)

  For topics used as state stores (e.g., a changelog of database updates), time-based retention doesn't work — you want to keep the latest value for each key, even if old. Log compaction retains the most recent record for each key and dele…

• Partitioning Strategy — Ordering vs Parallelism

  Partitioning is the key to Kafka's scalability, but it introduces an ordering trade-off.

• Ordering Guarantee

  Kafka guarantees order within a partition, but not across partitions. Example: Topic page-views with 3 partitions. - Partition 0: offsets 0, 1, 2, ... (ordered) - Partition 1: offsets 0, 1, 2, ... (ordered) - Partition 2: offsets 0, 1, 2…

• Key-Based Partitioning

  If you need ordering for a subset of records, use a key. Kafka hashes the key to pick a partition: Same key → same partition → order preserved. Example: Topic user-events with key=userid. All events for user:123 go to the same partition,…

• Partitioning Strategies

  1. No key (round-robin): Kafka cycles through partitions. Maximum parallelism, no ordering. 2. Key-based hashing: Same key → same partition. Per-key ordering, parallelism limited by key cardinality. 3. Custom partitioner: Producer can im…

• Hot Partitions

  If your key distribution is skewed (e.g., 90% of events have key=popularuser), one partition becomes a hot spot (overloaded). Fixes: - Add a random suffix to the key (e.g., user:123:0, user:123:1) to spread load. - Use more partitions (b…

• Producer Path — Batching, Compression, Acks

• Producer Workflow

  1. Producer sends a record (key, value, optional headers) to a topic. 2. Producer picks a partition (via key hash or round-robin). 3. Producer batches records in memory (per partition). 4. Producer compresses the batch (gzip, snappy, lz4…

• Batching

  Producers don't send every record immediately. Instead, they buffer records in memory (per partition) for up to: - linger.ms milliseconds (default 0 — send immediately, but usually set to 10-100 ms for batching). - batch.size bytes (defa…

• Compression

  Each batch is compressed before sending. Compression levels: - gzip: High compression ratio, slow (CPU-intensive). - snappy: Moderate compression, fast (Google's compression library). - lz4: Low compression, very fast (default). - zstd:…

• Acks Levels (Durability vs Latency)

  The acks config controls when the leader responds: | acks | Leader waits for... | Durability | Latency | |----------|--------------------------------------------------------------|-----------------------------------------|-------------|…

• Idempotent Producer

  Without idempotency, retries can cause duplicates. Example: 1. Producer sends batch. 2. Leader writes batch and replicates to followers. 3. Leader's ACK is lost (network glitch). 4. Producer retries → leader writes the same batch again (…

• Exactly-Once Semantics with Transactions

  For multi-partition writes (e.g., writing to topic A and topic B atomically), Kafka supports transactions: Kafka uses a transaction coordinator (a special broker role) to ensure atomicity. The transaction log is stored in a special topic…

• Consumer Path — Pull-Based, Groups, Offset Tracking

• Consumer Workflow

  1. Consumer subscribes to a topic (or multiple topics). 2. Consumer joins a consumer group (or runs solo). 3. Kafka assigns partitions to consumers in the group (one partition → one consumer at a time). 4. Consumer polls for records (bat…

• Pull-Based Model

  Unlike RabbitMQ (where the broker pushes messages to consumers), Kafka consumers pull records: Advantages: - Consumer controls the rate (slow consumers don't get overwhelmed). - Consumer controls batching (fetch 100 records at a time, or…

• Consumer Groups

  A consumer group is a set of consumers that share the workload of reading a topic. Example: Topic page-views with 3 partitions, consumer group analytics with 2 consumers. - Consumer 1: assigned partitions 0 and 1. - Consumer 2: assigned…

• Partition Assignment Strategies

  When consumers join a group, Kafka rebalances partitions across consumers. Assignment strategies: 1. RangeAssignor (default): Sort partitions by number, divide evenly. If 3 partitions and 2 consumers → C1 gets [0, 1], C2 gets [2]. 2. Rou…

• Rebalance Process

  When a consumer joins or leaves, Kafka triggers a rebalance: 1. Stop the world (pre-3.0): All consumers stop processing, return partitions, wait for new assignments. This can take seconds (throughput drops to zero). 2. Incremental rebala…

• Offset Tracking in consumeroffsets

  Consumers commit their current offset per partition to a special Kafka topic consumeroffsets (a compacted log with key=(group, topic, partition), value=offset). When a consumer crashes and restarts: 1. It rejoins the group. 2. Kafka read…

• Consumer Group Example — Rebalance During Scale-Up

  ---

• Replication — Leader, Followers, ISR

  Each partition has: - 1 leader (handles all reads and writes). - N-1 followers (replicate the leader's log).

• Replication Factor (RF)

  Configured per topic (e.g., replication.factor=3). Each partition has RF replicas spread across different brokers. Example: Topic orders with 1 partition, RF=3. - Leader: Broker 1 (offset 0..999). - Follower: Broker 2 (offset 0..999, rep…

• In-Sync Replicas (ISR)

  A follower is in-sync if it's caught up to the leader (within replica.lag.time.max.ms, default 10 seconds). The ISR is the set of replicas (leader + followers) that are in-sync. Example: - Leader: Broker 1 (offset 1000). - Follower: Brok…

• High-Water Mark (HWM)

  The high-water mark is the highest offset that all ISRs have replicated. Consumers can only read up to the HWM (not beyond). Why? To prevent non-repeatable reads. If a consumer reads offset 1001 from the leader, and then the leader crash…

• Leader Election

  When the leader crashes: 1. Controller (via ZooKeeper or Raft) picks a new leader from the ISR. 2. The new leader becomes writable. 3. Followers start replicating from the new leader. If all ISRs are down (catastrophic failure), Kafka ca…

• Delivery Semantics — At-Most-Once, At-Least-Once, Exactly-Once

• Producer Delivery Semantics

  | acks | Idempotence | Delivery Guarantee | Notes | |----------|-----------------|------------------------------|------------------------------------------------| | 0 | Off | At-most-once (fire-and-forget) | Data loss if broker crashes b…

• Consumer Delivery Semantics

  | Commit timing | Delivery Guarantee | Notes | |---------------------------------------|------------------------------|------------------------------------------------| | Commit before processing | At-most-once | If consumer crashes mid-…

• Exactly-Once End-to-End (Kafka Streams / Transactions)

  For multi-stage pipelines (read from topic A, write to topic B), Kafka provides exactly-once semantics via transactions: The transaction coordinator ensures: - Either both the output write and the offset commit succeed, or both fail (no…

• Kafka Workflow — Pub-Sub vs Queue

  Kafka supports both pub-sub and queue semantics via consumer groups.

• Pub-Sub (Multiple Groups)

  Multiple consumer groups subscribe to the same topic. Each group gets a full copy of the data. Example: - Topic: user-signups - Consumer group analytics: reads all records → sends to data warehouse. - Consumer group email-service: reads…

• Queue (Single Group)

  A single consumer group reads a topic. Partitions are load-balanced across consumers in the group. Example: - Topic: order-processing (10 partitions) - Consumer group workers (5 consumers) - Each consumer gets 2 partitions (load balanced…

• Performance — Why Kafka is Fast

  Kafka achieves millions of messages/second on commodity hardware. Here's how.

• 1. Sequential Disk I/O

  Kafka writes are sequential appends (no random seeks). Reads are sequential scans (consumers read from the tail). Modern disks are very fast for sequential I/O: - HDD: ~500 MB/s sequential (vs ~100 IOPS random). - SSD: 1-5 GB/s sequentia…

• 2. OS Page Cache

  Kafka relies on the OS page cache (free RAM used by the OS to cache disk reads). When a consumer reads, Kafka: 1. Reads from disk → OS page cache (if not already cached). 2. Subsequent reads hit the page cache (memory speed). The page ca…

• 3. Zero-Copy (sendfile)

  As described earlier, sendfile transfers data from disk → page cache → network socket without copying to user-space. This saves CPU and memory bandwidth.

• 4. Batching

  Producers batch records (10-100 ms), reducing network round-trips. Consumers fetch batches (not one record at a time), reducing overhead.

• 5. Compression

  Compression (lz4, snappy) reduces network bandwidth by 50-80%, critical for cross-datacenter replication.

• 6. Partitioning

  Partitions enable parallelism. A topic with 100 partitions can: - Be written by 100 producers in parallel (no contention). - Be read by 100 consumers in parallel (in different groups). - Spread across 100 brokers (no single bottleneck). ---

• KRaft Mode — Removing ZooKeeper

• The ZooKeeper Problem

  Pre-2.8, Kafka used ZooKeeper for: - Storing cluster metadata (topics, partitions, ISRs). - Controller election. - Tracking consumer group membership (pre-0.9; later moved to Kafka). ZooKeeper added: - Operational complexity: Separate en…

• KRaft Mode (Kafka Raft)

  Kafka 2.8 (2021) introduced KRaft mode — a ZooKeeper-free architecture where metadata is stored in a special Kafka topic (clustermetadata) and the controller is elected via Raft (the consensus protocol from etcd, Consul). How it works: 1…

• Kafka vs RabbitMQ vs Amazon SQS — Comparison

  | Feature | Kafka | RabbitMQ | Amazon SQS | |--------------------------|-----------------------------------------------|---------------------------------------------|---------------------------------------------| | Model | Distributed co…

• Interview Takeaways

  When discussing Kafka in system design interviews, focus on these points: 1. Kafka is a log, not a queue Messages are appended to a log and retained for days/weeks (configurable). Consumers track their own offsets and can replay history.…

• Further Reading

  - Original paper: Kreps et al., "Kafka: a Distributed Messaging System for Log Processing" (2011) — the LinkedIn design doc. - Confluent blog: Kafka internals, best practices, and design patterns (from the creators of Kafka). - "I Heart…