Design a Durable Distributed Message Broker

An order service must hand off work without waiting on every downstream system. Fulfillment, analytics, and notifications each need to consume the same events at their own pace, survive outages, and replay history when they need to recover or recompute results.

What, why, and use cases

The system in one sentence

A distributed message broker accepts events from one application and lets other applications read them later, independently and at their own pace.

Suppose an order API must notify fulfillment, update analytics, and send email. Calling all three services before replying makes the order request depend on their latency and availability. A temporary email outage can fail an otherwise valid order. A message broker accepts the event once and lets each downstream service work at its own pace. It can buffer a temporary gap between production and consumption rates. That buffer is finite: if producers keep writing faster than consumers can catch up, lag and retained bytes continue to grow until the system adds capacity, throttles producers, or rejects new writes.

Before the design: a small vocabulary

• Event (or record): one immutable fact, such as OrderCreated.
• Topic: the named stream that holds related events.
• Partition: one ordered shard of a topic; order applies only within that shard.
• Broker: a server that stores partitions and serves reads and writes.
• Producer: the application that writes an event to a topic.
• Consumer group: cooperating readers that divide partitions among themselves; different groups can read the same events independently.
• Offset: a record’s position within one partition and a group’s saved place for resuming or replaying.
• Leader and replicas: one leader accepts writes while replicas copy its log so acknowledged records can survive a failure.
• Retention: the time or size limit that determines how long stored events remain available for replay.

Functional requirements

Required behavior

1. The system lets an authorized operator create a topic with a partition count, retention policy, and replica count; clients can discover its partition leaders.
2. The system lets a producer publish a size-bounded record with an optional routing key and receive the assigned partition and offset once the configured acknowledgment condition is satisfied.
3. The system lets a consumer join a named group, fetch batches from assigned partitions, process them, and commit the next offset to read. Another group can read the same retained records independently.
4. The system lets a group resume after a consumer crash or deliberately replay retained records by resetting its offset. Records expire under the topic’s retention policy even if a group has not caught up.
5. The system lets an operator inspect lag, replication health, failed publishes, and access changes; a client gets explicit errors for bad requests, throttling, unavailable partitions, and offsets outside retention.

Optional capabilities and scope boundaries

• Task-queue delivery: One worker leases a message at a time, hiding it from other workers for a visibility timeout while it is processed. An acknowledgment before the timeout deletes the message; otherwise, the lease expires and the message becomes available for redelivery.
• Failure isolation: A retry topic and dead-letter topic (DLQ) can separate repeatedly failing work from normal processing.
• Other delivery behavior: Message priority, delayed or scheduled delivery, and metadata-based filtering are outside this base design and need explicit policies.

Non-functional requirements

The targets below make the broker’s expected behavior measurable.

Property	Design target
Broker acknowledgement latency	p99 below 100 ms at peak load, measured from the broker receiving a publish request to returning success.
End-to-end publish latency	p99 below 250 ms from the order commit to the producer receiving broker success, including brief batch wait, retries, and route refresh.
Availability	99.9% monthly publish and fetch objective. Tolerate one broker or one availability-zone failure; measure actual failover time before committing to a stronger target.
Consistency	One active leader orders each partition, and one consumer in a group owns each partition at a time. The design guarantees order within a partition, not across the entire topic.
Durability	Acknowledge OrderCreated only after the leader and its current caught-up replicas have stored it, with at least two replicas in sync across three zones.
Scale	Handle 3,000 records/s on average and 10,000 records/s at peak, with 1 KiB average records, 1 MiB maximum records, 64 partitions, and two groups reading the full stream.
Retention	Keep accepted records for at least 14 days. At the disk limit, throttle or reject new writes rather than remove younger records.
Security	Require TLS, authenticated clients, topic and group permissions, encryption at rest, byte-rate quotas, and audit logs.
Cost	Plan disk for three replicas, indexes, recovery headroom, and consumer egress; validate compression and network costs with production measurements.

Core concept

A retained log with independent positions

The central insight is to store each event once as an immutable position in an ordered partition, then track each consumer group’s position separately. Publishing does not wait for fulfillment or analytics. Each group can stop, resume, or replay while topic retention still governs when the stored bytes disappear. This is the mechanism that makes decoupling, pace matching, and repeatable consumption compatible.

A topic names an event stream. A partition is one ordered slice of that stream. A broker is a server that stores and serves several partitions. Each record gets an offset: its increasing position inside one partition, not a global message ID. A consumer group saves its own next position for each partition, so Fulfillment and Analytics can read the same stored record independently. A routing key keeps related records on one partition while the partition map stays stable.

Two rules keep the system predictable. Only one leader writes to a partition at a time, so its records stay in order. Within a consumer group, only one consumer handles a partition at a time, so two consumers do not process the same record concurrently. A committed offset only records where that group should start next; it does not confirm that an email was sent, a payment succeeded, or a database change finished.

Messaging systems commonly expose three interaction patterns. A retained log directly supports the first two; request/reply can be layered on top, but it makes the caller wait for a downstream result again.

Pattern	Client-visible behavior	Typical use
Point-to-point within a group	One group member owns a partition at a time; processing attempts may repeat.	Email jobs or background tasks.
Publish/subscribe	Each independent group reads the event.	Notifications, analytics, or recommendation refreshes.
Request/reply	The caller waits for a correlated reply.	A workflow that needs a result, at the cost of renewed coupling.

Follow record 81

Order event flow Expand diagram

×

Normal path. The routing key sends all events for order-42 to partition 7. Its leader gives this event offset 81, waits for the required copies to store it, and only then reports success. Fulfillment performs its business action and saves 82, meaning “start with the record after 81 next time.” Analytics keeps a separate saved position, so it can read record 81 without waiting for Fulfillment.

Worker crash after the business action. If Fulfillment applies the effect but crashes before saving 82, its saved position is still 81. The replacement worker correctly starts at 81 and may repeat the effect. This is why a consumer must make its business action safe to repeat—for example, by using the order ID as an idempotency key.

Order saved but publish never accepted. If the database commits order-42 but the broker never accepts OrderCreated, neither consumer group can see it. When that gap is unacceptable, the order service writes an outbox row—a durable database record of the event waiting to be published—alongside the order in the same database transaction. A separate relay publishes that row later. The outbox makes the handoff recoverable; it can still publish twice after a crash, so the event ID must remain stable.

Event schema and versioning

The order service defines the OrderCreated event shape because it owns the meaning of an order. Every event carries a serialization format and schema version so consumers know how to read it.

Change	Why it is safe or risky	Consumer response
Add an optional field, such as couponCode	Older consumers can ignore a field they do not use.	Continue processing; newer consumers may use the field.
Add a field with a clear default	A consumer that does not receive the field can use the documented default.	Continue processing with that default.
Rename a field, remove one, or change its meaning	An older consumer may read the wrong value or make an unsafe business decision.	Coordinate a migration: publish a compatible version first, update consumers, then retire the old form.

When a consumer receives a version it does not understand, it must choose deliberately. A critical consumer such as Fulfillment should stop that partition and alert before moving its saved position; skipping an order is worse than pausing. A non-critical consumer may put the record in a dead-letter queue (DLQ)—a separate place for records needing inspection—or quarantine it. It may only continue when its compatibility policy proves the newer version is safe to ignore.

High-level design

Why the components stay separate

Different parts of the broker have different jobs and failure patterns. Keeping them separate prevents a busy data path from delaying a safety decision, while still making clear who owns each decision.

Component	Problem it solves / responsibility	Why this choice was made (trade-off)
Producer client	Chooses a partition from the key and briefly collects small records for the same destination into a batch.	Sending one tiny network request per record wastes round trips. Doing this in the client avoids a mandatory middle service, but clients must refresh their route after a failover.
Partition broker	Stores event bytes, assigns offsets on the leader, copies them to replicas, and serves reads.	The data path handles large, append-heavy traffic. Keeping it close to the stored partitions improves throughput, but each broker must manage disk and replication carefully.
Metadata controller	Tracks topic configuration, replica placement, and which broker currently leads each partition.	These are small but correctness-critical decisions. Separating them means heavy event traffic cannot delay a leader election, but clients need cached metadata.
Group coordinator	Decides which consumer owns each partition in a group and rejects progress updates from an old owner.	A group can add or lose workers without changing stored event records. Coordination adds rebalance pauses, but prevents two workers from owning one ordered partition at once.
Progress store	Saves each group’s next record to read, independently of the retained event bytes.	A group can resume after a crash even while old segments are later removed. Only the latest position matters, so a replicated store that keeps the latest position is efficient.
Consumer	Performs the real business action, then saves progress.	The broker can prove that it stored a record, not that an email was sent or a payment was charged. Keeping the effect with the consumer makes that boundary explicit, but effects must tolerate retries.

The table makes ownership and failure boundaries explicit before the request trace.

Part	Owns and receives	Returns or calls	If unavailable
Producer client	Payload, key, publish ID, metadata cache, and batch buffer	Sends directly to a partition leader and retries unknown outcomes	Buffered records can expire; caller sees unknown or failed result.
Consumer client	Assignment and local processing state	Fetches batches from leaders, applies effects, and commits progress	Another group member can take the partition and repeat work.
Metadata controllers and log	Topic policy, broker health, placement, and leader epoch	Persist metadata and supply versioned routes	Existing paths may continue; elections and changes stall.
Partition brokers and logs	Leader/follower segment files, offset index, and replication position	Leader accepts writes and serves reads; followers pull and append	An eligible follower may lead; under-replication can stop writes.
Group coordinator	Membership, assignment generation, committed offsets; runs as a broker role	Assigns partitions and writes commits to progress store	Joins, rebalances, and commits stall until a coordinator recovers.
Progress store	Durable (group, topic, partition) -> next_offset	Replies to coordinator reads and writes	New owners cannot safely resume without a position.

The progress store is a logical component; in this reference design, brokers implement it as a replicated compacted log rather than as a separate database. Compaction retains the latest record per (group, topic, partition) key and removes older versions in the background, which fits progress because only the newest position is needed.

Architecture and trust boundaries

Notice the producer’s direct route to the partition leader and the separation of message data, metadata, and group progress. Solid arrows are request or data paths; dashed arrows are discovery, coordination, or replica-copy paths.

Broker architecture and trust boundaries Expand diagram

×

• The producer caches the partition map, batches by partition, and sends to its leader. Followers pull records from that leader and store their own copies.
• A group consumer gets an assignment, fetches from a leader, processes, then commits its next offset. Fetching alone does not commit progress.
• Controllers persist configuration and leader assignments; the group coordinator persists progress. Neither control store holds payloads.

Brokers authenticate clients, store each partition’s ordered log, and handle reads and writes. Controllers maintain the cluster map: which brokers are available, where replicas live, and which broker leads each partition. A group coordinator—a broker role, not necessarily a separate server—assigns partitions to consumers, detects membership changes, and stores each group’s next offset. Splitting these jobs keeps high-volume event traffic separate from cluster and group coordination, while making each publish and consume step easy to trace.

API surface

Start with the user-visible path: create a topic, publish to its leader, fetch as a group, then commit the next offset only after processing. The HTTP-shaped paths below illustrate broker RPCs, not an actual HTTP data path or Kafka endpoints. consume is a client-library wrapper around joinGroup and fetch; it does not add another broker hop. Clients use TLS; brokers authenticate clients and authorize every operation. Topic administration, publishing, reading, and group commits require distinct permissions.

Goal	Example interface	Input	Output and key failures
Create a topic	POST /v1/topics	{ name, partitionCount, replicationFactor, retention }	{ topicVersion } after metadata is persisted and replicas can be placed; reject unauthorized requests or insufficient placement.
Delete a topic	DELETE /v1/topics/{name}	expectedVersion to prevent deleting a changed topic	{ deletionPending: true }; the reference policy stops writes and starts a recoverable deletion window.
Publish a record	POST /v1/topics/{topic}/partitions/{partition}/records to the discovered partition leader	{ routingKey?, recordId, payload }, with a stable recordId for retries	{ partition, offset, leaderEpoch } only after the configured replica acknowledgment. Reject unauthorized, oversized, or throttled writes; NOT_LEADER means refresh the route. A timeout leaves the outcome unknown.
Consume a batch	Client-library consume(topic, groupId, maxBatchSize, waitMs)	Topic, group, maximum batch size, and bounded wait	{ records, nextOffsets } from assigned partitions; an empty result or rebalance signal is possible. Returning records does not commit progress.
Commit group progress	PUT /v1/groups/{groupId}/offsets/{topic}/{partition} to the group coordinator	{ generation, nextOffset }, where nextOffset is the next record to read	{ committed: true } only after durable progress storage; reject a stale group generation.
Join a group	Coordinator RPC joinGroup(topics, groupId, memberId)	Subscribed topics, group, and member identity	{ generation, assignments }; membership changes can trigger a rebalance.
Fetch from a partition	Leader RPC fetch(topic, partition, fromOffset, maxRecords, waitMs)	Partition, starting offset, count limit, and bounded wait	{ records, nextOffset }; may return an empty batch, offset-out-of-range error, or NOT_LEADER when the route is stale.

createTopic, deleteTopic, publish, and commitOffset are mutations with different retry rules. expectedVersion and generation reject stale administrative or group changes. A timed-out publish may already have appended, so retry with the same recordId; broker deduplication needs a documented scope and expiry, and producer sequence numbers can suppress duplicate appends within a session. With multiple batches in flight, retrying an earlier non-idempotent request after a later batch succeeds can reorder those batches. Neither mechanism deduplicates a consumer’s external side effect.

Publish, pull, process, commit

Assume the order service has committed an order and publishes OrderCreated keyed by order-42. If the order commits but no publish is ever accepted, consumers never learn about it; use an outbox when that handoff must be reliable. A timeout alone does not prove failure because the broker may have accepted the record before the response was lost.

Three failure timelines to test against the trace:

1. The leader appends 81, but its response is lost (see the “Follow record 81” diagram). The producer cannot infer failure from a timeout; it retries with the same recordId. If the broker’s deduplication window has expired, a duplicate append remains possible and consumers must tolerate it.
2. Fulfillment applies 81, then crashes before committing 82. The new group member starts at the committed position and may apply 81 again, so the downstream effect needs idempotency or reconciliation.
3. Fulfillment stays offline beyond retention. A fetch at 81 returns offset-out-of-range; the group stops and alerts, then backfills or reconciles missing orders before an explicit reset. Resetting to latest would skip them.

The routing key selects partition 7, and each group has its own progress. A group preserves processing order only if it does not run effects from one partition concurrently out of order. Retention, not group progress, removes the record.

Deep dive

Partitioned log, ordering, and performance

The tension

The broker must store many records quickly and still let readers replay them later. Updating one database row every time a job is delivered creates extra update and index work. One giant ordered file avoids those updates but would force all traffic through one narrow path. Tiny network writes waste round trips; waiting too long to collect a group of records delays publication.

The choice

Store events in several partition logs. Within each partition, the broker only appends new records to the end. When the current log file reaches its size limit, the broker starts a new file, called a segment. To read from an offset, a small index points near that record’s position in the file, so the broker scans only a short distance instead of the entire log. Producers briefly batch records for the same partition to reduce network and disk overhead; consumers replay retained history by requesting an earlier offset. A relational task table is a better fit when each job needs to be individually claimed and updated, but those extra writes add contention. An in-memory queue can be fast, but it needs a separate durable store if jobs must survive a restart.

The mechanics

Store bytes in segments, and keep control state separate.

The message path mostly does two things: add a record to the end of a file and read forward from a known point. Disk storage is efficient at that pattern because it avoids jumping between many unrelated locations; this is the reason for an append-only log. Splitting the log into bounded segments makes retention practical: delete an old file after its retention period instead of removing individual rows. A sparse offset index helps find the right place inside a segment. The operating system’s page cache keeps recently used file data in memory for faster reads, but it is not proof that data survived a power loss. Batches reduce network calls; compression reduces bytes but spends CPU.

Records, offsets, and access paths.

Partition-log replicas hold immutable records; the progress store owns group positions. This small schema is illustrative. A checksum detects corruption but does not repair it; replicas or backups are needed for recovery.

Record { topicId, partition, offset, key?, recordId, timestamp,
         headers, schemaVersion, payloadBytes, payloadSize, checksum }
GroupProgress { groupId, topic, partition, nextOffset, generation }

Hot access paths append by partition, fetch from an offset, look up a leader, and update group progress. The broker stores schemaVersion without interpreting payload bytes. A cyclic redundancy check (CRC) detects damaged bytes but cannot repair them; implementations may checksum batches rather than individual records. A sparse index maps sampled offsets to byte positions; the broker scans from the nearest entry. Kafka memory-maps its index file, though this reference design need not. Kafka index implementation. Rotation bounds files; retention deletes eligible segments. An offset older than retained data raises an error for the group to resolve.

Notice that two groups share the same physical record but keep separate progress; the numbers are next offsets, not record counts.

Independent consumer group progress Expand diagram

×

• The log is retained by topic policy; neither group’s commit deletes its records.
• Each group’s progress can diverge without duplicating stored message bytes.
• A group that falls behind the retention boundary must reconcile missing work before choosing a new offset.

Sequential storage, indexes, and retention.

A partition’s active segment receives sequential batches; sealed segments serve reads. Sealed segments become eligible for deletion only after their records pass the 14-day minimum, but cleanup runs asynchronously and deletes whole segments, so expiry is segment-granular rather than record-exact. The log is the retained message data rather than a prelude to updating rows.

For orientation, the following directory sketch is illustrative; brokers can choose different filenames and layouts. Each replica keeps its own copy.

topic-A/
  partition-0/segment-0, segment-1
  partition-1/segment-0, segment-1

Only each partition’s latest segment accepts appends. Sequential disk access and the operating system page cache help throughput, but random reads and forced flushes still have costs; no storage medium makes all disk access fast.

Notice the physical hierarchy from topic to offset-bearing records. Solid arrows from topic through segment show containment; the dashed arrows show lookup or expiry.

Partition log and retention layout Expand diagram

×

• The leader appends records only to the active segment and periodically seals it. Each record’s offset is unique within its partition, not across the topic.
• Fetch seeks using the nearest indexed offset and scans forward. Page cache may serve reads without disk access; a checksum detects corruption but cannot restore bytes.
• Retention removes eligible sealed segments independently of group commits, so storage capacity and maximum tolerable lag are linked.

Deeper look: page cache and zero-copy fetch.

The next bottleneck can be serving the same bytes to many consumers. A normal file-to-socket path may copy file data into an application buffer and back into the kernel. A zero-copy fetch path such as Linux sendfile can avoid that user-space round trip for eligible file-backed records. Kafka does not use sendfile when SSL is enabled, so this optimization may not apply to the TLS-protected client path drawn here. Benchmark the deployed transport; this optimization does not change producer append or replica acknowledgment.

Notice which path the optimization shortens.

Ordinary and zero-copy fetch paths Expand diagram

×

• The ordinary path copies bytes through application memory; the optimized path can avoid those copies for file-backed fetches.
• The page cache is RAM managed by the operating system. It may hold unflushed writes, so it is not proof of physical-disk durability.
• This diagram describes consumer transfer only; replication and publish acknowledgment follow separate contracts.

Partition order is narrower than business order.

Records stay in append order within one partition, not across the whole topic. The broker keeps related records together by routing the same key to the same partition, but it does not guarantee business-time order when producers race or retry.

Adding partitions can change where a key is routed, leaving its older records in one partition and newer ones in another. For long-lived per-key ordering, keep a stable key-to-shard map or deliberately migrate to a new topic. Consumers must also discover new partitions and use safe starting offsets; Kafka allows increasing a topic’s partition count, not reducing it. Kafka topic operations

Replication and producer acknowledgment

The tension

The durability target is threatened when a leader fails after accepting bytes but before followers copy them. Waiting for every configured follower can block writes during a slow or failed replica; acknowledging only the leader exposes a loss window.

The choice

Use one leader per partition and let its follower replicas copy from it. A replica is in sync when it has caught up enough to safely participate. The leader reports success only after every currently in-sync replica has appended the record, with at least two in sync. This costs a replica round trip and deliberately rejects writes when too few safe copies remain.

The mechanics

Notice the difference between a write’s local append, its successful acknowledgment, and consumer visibility.

Publish acknowledgment lifecycle Expand diagram

×

• Unknown describes what the producer knows. The record may exist, so retry must tolerate duplicates.
• Visible describes the broker under this reference policy. It does not mean any consumer has processed the record.
• Acknowledgment is not proof of physical flush on every disk or of survival after correlated failure.

Deeper look: producer acknowledgment choices.

These settings govern when a producer considers its publish successful.

Setting	Producer observes	Cost or failure window
acks=0	No broker confirmation	Lowest wait, but even receipt is unknown; retries cannot depend on an acknowledgment.
acks=1	Leader appended locally	Fewer replica waits; an immediate leader loss before follower catch-up can lose an acknowledged record.
acks=all	All current in-sync replicas acknowledge, subject to the configured minimum	Stronger ordinary-failure survival, with replica latency and write unavailability when the minimum is unmet.

The reference design selects the last row. “All” refers to the current in-sync set, which may shrink after a lagging follower is removed; it is neither every configured replica nor a fixed quorum count. Idempotent producer IDs and per-partition sequence numbers suppress retry duplicates within their defined scope.

Replication and failover without magical guarantees.

The pressure is broker loss after a successful publish. One leader orders appends; followers fetch and append copies on other brokers. The chosen in-sync replica set (ISR) contains the leader and followers caught up enough for acknowledgment and eligible election. A lagging follower leaves the set and may rejoin after catching up. The leader tracks catch-up; controllers record membership changes, not one change per write. The leader waits for every current ISR member, with at least two required. Page-cache-backed appends may still be unflushed on every replica at acknowledgment.

The high watermark is the boundary below which records have met the replication visibility condition. Consumers normally fetch from the leader and only below that boundary; nearest-replica or follower reads are an optional locality optimization requiring a visibility policy. The leader’s Log End Offset (LEO)—the next offset after its last appended record—can be ahead of the high watermark while followers catch up. A newer leader epoch, assigned by the controller, fences a recovered old leader from writing under obsolete ownership. If a consumer is offline, the log end can advance while its committed next offset stays fixed, increasing lag; catching up reduces lag only if consumption outpaces new production and retention has not erased the gap. The high watermark protects against exposing unreplicated records, not every correlated failure or loss of unflushed pages.

Notice that follower catch-up precedes promotion or removal during planned broker movement.

Leader failover and replica promotion Expand diagram

×

• A follower that has not caught up to the committed boundary is not automatically eligible. Failover time depends on detection, election, and client refresh.
• A planned broker move copies a replacement replica first, waits for catch-up, then removes the old one; copying consumes network and disk bandwidth.
• If too few in-sync replicas remain, publishing stops under the chosen minimum. Allowing an out-of-sync replica to lead would trade acknowledged data for availability.

Consumer coordination and observable effects

The tension

Pulling at different speeds needs independent progress, but a worker crash can reassign a partition after a side effect and before its offset commit. A retry can duplicate work, while committing first can lose it.

The choice

Give each partition to one active worker in a group, save the next offset after work succeeds, and reject updates from a worker that has been replaced. This produces at-least-once attempts: a crash can make a record run again, so the downstream action must be safe to repeat or checked later. A per-message lease queue is a different contract for individual jobs that need a claim and deadline.

The mechanics

Why group state is separate from delivery.

A coordinator answers two questions for a consumer group: “who owns this partition now?” and “where should that owner restart?” In this design, one broker plays that role. It records durable next offsets before confirming a commit. When a worker joins, leaves, or stops sending its “I am alive” heartbeat, the coordinator creates a new assignment version. That version prevents an old worker from committing progress after replacement. These updates are small and frequent, unlike the large event bytes. The coordinator still cannot see whether a downstream email, payment, or database change really happened. Extra workers beyond the partition count wait idle.

Rebalancing and the duplicate window.

A consumer crash strands its assigned partitions until membership changes. Letting two members process the same partition concurrently breaks the group’s single-owner rule; stopping all consumption for a full reassignment hurts availability. A generation-fenced rebalance revokes assignments, chooses new owners, and resumes from durable offsets. It reduces overlapping ownership but cannot erase work done just before a crash.

Notice that the new owner begins from the committed position, even if the old owner performed a later side effect.

Consumer rebalance and duplicate window Expand diagram

×

• A graceful member can finish and commit before revocation; a failed member cannot, so record 81 may be repeated.
• The rejected late commit protects progress from a stale owner, not external effects that owner already made.
• More consumers improve parallelism only up to the partition count and only if downstream capacity exists.

Deeper look: joining and leaving a classic group.

A consumer-elected group leader belongs to a classic group protocol. Start with A owning the partitions and heartbeating. When B joins, the coordinator begins a rebalance, A revokes and rejoins, a chosen consumer leader computes an assignment, and the coordinator distributes it; A and B then resume from their assigned committed offsets. A graceful A sends LeaveGroup, so B can be reassigned promptly. A crash has no leave request: the coordinator waits for heartbeat expiry before reassigning, and replay can duplicate work performed after A’s last commit.

Our reference coordinator computes assignments itself, matching the newer server-side option. Kafka documents both classic and newer consumer protocols; the latter can update assignments incrementally instead of forcing every member through a global rejoin. The invariant for either is one active owner per partition in a group, not one consumer per topic or one partition per consumer. Kafka consumer rebalance protocol

Progress does not equal one external effect.

If the consumer commits before doing work, a crash can skip an event. If it does work first and commits after, a crash between those steps can repeat the work. The latter gives at-least-once attempts, provided records remain retained and the system eventually recovers; it does not guarantee successful delivery to an unavailable external system. For a database effect, record the message ID in the same transaction as the business update. For an external API, use that API’s idempotency contract or reconcile uncertain outcomes. A broker transaction can atomically publish broker output records and commit consumed offsets in supported systems, but it cannot make an arbitrary email or payment side effect atomic.

The three labels describe different failure choices; no one producer setting supplies an end-to-end processing guarantee.

Goal	Consumer behavior	Remaining risk or cost
At-most-once processing attempt	Commit before processing	A crash after commit loses the work.
At-least-once attempt	Process, then commit; retry uncertain publishes	A crash before commit can repeat the effect; retention and outages still bound recovery.
One observable effect	Couple progress to the effect with a transaction or idempotent/reconciled destination	More state and coordination; broker-only transactions do not cover arbitrary external APIs.

Kafka’s idempotent producer addresses duplicate appends; broker transactions can link multiple Kafka topic writes and offsets. For a payment, accounting write, or email, the destination still needs an idempotency or reconciliation contract.

Deeper look: the producer’s database-to-broker gap.

The order transaction can commit while its later publish fails, leaving no event for consumers. A transactional outbox addresses this application boundary: write the order and an outbox row in one database transaction, then have a relay publish the row and mark it sent. A relay crash after publish can publish again, so stable IDs and broker or consumer deduplication remain necessary. The outbox adds storage, relay lag, and cleanup work.

Retry, poison records, and queue-specific features.

A poison record can block a sequential group if retried forever; silent deletion hides it. After bounded attempts, publish it with failure context to a DLQ, await acknowledgment, then commit past it. A crash between those steps may duplicate the DLQ entry; a stable failure ID aids deduplication. A delay topic or timer can schedule retries, but a timer is no proof of processing. Expiry, overflow, and invalid destinations are separate cases: retention may expire old data; overload should throttle or reject; an unknown topic should return an error. Route any of these to a DLQ only under an explicit, durable policy.

Notice that each branch needs an explicit operator or application policy.

Retry and dead-letter flow Expand diagram

×

• Holding progress preserves the original for retry but may stall its partition; a separate retry stream can reduce that stall at the cost of order.
• DLQ arrival means the broker accepted a failure record, not that an operator resolved the problem.
• Visibility timeouts, weighted priority scheduling, aging, and delayed delivery belong to a lease queue or scheduler contract. They are optional, with starvation and duplicate-delivery risks to specify separately.

Client routing, metadata, and flow control

The tension

A metadata lookup for every record makes the control plane a hot-path bottleneck, while stale routes after failover can send clients to the wrong leader. Bursts can exhaust broker or producer memory unless admission is bounded.

The choice

Let broker-aware clients cache versioned leader maps, batch by partition, and refresh on a not-leader response. Controllers persist placement; brokers authenticate and quota clients. A gateway can centralize validation for untrusted clients but adds a hop.

The mechanics

Metadata and progress storage choices.

The system stores two kinds of small information: cluster facts (topic settings and current leaders) and group progress (the next offset for each group and partition). Both need careful, durable updates, but neither should compete with event payload traffic. The controller uses a replicated agreement log for cluster facts. For group progress, brokers keep a replicated compacted log: it eventually keeps only the latest value for each (group, topic, partition) key because the newest next offset is the only one needed. A transactional database can be suitable for a smaller system but adds another dependency to operate. Kafka 4.0 and later use KRaft controllers rather than ZooKeeper for cluster metadata; ZooKeeper is historical, not required here.

Why routing needs a version.

A cached leader address can become stale during failover. NOT_LEADER is the broker’s plain response: “I am no longer the current leader for this partition; refresh your route and try the current leader.” The producer refreshes its versioned partition map, then retries with the same recordId. A leader epoch fences an old leader from accepting new writes. Broker-aware clients remove a routing hop and can batch by destination, but they need metadata, authentication, and retry logic. A stateless gateway behind load balancers is a credible alternative for untrusted clients: it centralizes validation, quotas, and auditing but adds a hop and must forward batches efficiently. A gateway does not inherently prevent batching. The chosen path authenticates and enforces quotas at the brokers.

Why metadata stays off the hot path.

Every publish needs a leader address, but a controller lookup on each request would bottleneck the control plane. Producer and consumer libraries cache a versioned partition map. A NOT_LEADER response forces refresh; version checks prevent an old map from replacing a newer one. Bypassing the cache simplifies freshness but increases controller load. Controllers replicate a small metadata set; a much larger one may need sharding with clear topic ownership.

Topic metadata includes owner, creation time, limits, lifecycle state, placement, and leader epoch. A controller leader places replicas, detects broker failures, and elects eligible partition leaders; it holds no payloads. Brokers receive stable publish IDs; any deduplication needs a defined scope. Usage audit is no proof of delivery.

Choose transport, scheduling, and visibility by their failure pressure.

Synchronous per-record calls waste round trips, so the producer buffers records by partition until a size or wait limit is reached—for example, an illustrative 16 KiB or 10 ms, not a universal setting. The broker appends batches; consumers fetch batches too. Compression saves network and disk bytes but spends CPU. Larger batches improve throughput and can increase wait latency; more partitions add parallelism but do not erase batching delay. A long-poll fetch gives consumers rate control without tight empty loops but holds connections open. Push shifts slow-consumer flow control to the broker.

Search is absent from the critical path: full-text indexing opaque payloads adds cost and schema coupling. Export records if search becomes a requirement. Metrics must expose replication and group lag, disk use, throttling, leader changes, and errors. TLS, authentication, access controls, and quotas protect the trust boundary; Kafka documents these as available mechanisms, not automatic deployment properties.

The reference configuration below is pseudocode, not a production broker configuration. Its pressure is simple: an acknowledged write should survive an ordinary single-replica loss, at the price of another replica round trip and lower availability when too few replicas remain.

topic: order-events
partitions: 64
replicas: 3
minimum_in_sync_replicas: 2
producer_ack: all_current_in_sync_replicas
minimum_retention: 14d
disk_limit_policy: reject_new_writes
max_record_bytes: 1048576

The all_current_in_sync_replicas line means every replica currently considered in sync must append before success, and the set must contain at least two. It does not claim a disk fsync on each replica. In Kafka, acks=all and min.insync.replicas have related but distinct roles; a minimum of two does not mean the leader waits for only two when three replicas are in sync.

Topic creation and deletion.

Topic creation checks permission and limits, places replicas across failure domains, persists metadata, then marks the topic ready. Clients refresh their route. Deletion stops writes, waits through an assumed recovery window, then removes data and metadata; the action is audited. This is a reference policy, not a product guarantee.

Background work follows the normal path.

The controller watches broker health and moves leadership only to an eligible replica; follower catch-up, retention cleanup, rebalancing, and metrics run outside the user request. Membership changes revoke old assignments and issue a new generation. All client requests carry an authenticated identity: a producer needs write permission, a consumer needs read and group permissions, and topic deletion requires a separate administrative permission and preferably a recoverable delay. Batch compression is useful only when CPU and latency measurements justify it. Per-tenant quotas and bounded producer buffers turn overload into a visible throttle instead of unbounded memory growth.

Estimation

These are planning estimates, not broker guarantees. They use decimal storage units and deliberately exclude compression, protocol overhead, indexes, safety margin, and rebalancing traffic. Add those measured costs before committing to hardware.

Inputs and retained storage

Assumption or calculation	Value	What it tells us
Average write rate	3,000 records/s	Normal planning load.
Average record size	1 KiB (1,024 bytes)	Payload assumption; real payloads must be measured.
Raw ingress	3,000 × 1,024 = 3.072 MB/s	Bytes arriving before replicas.
Data per day	3.072 × 86,400 ≈ 265 GB	One day of logical retained data.
Logical data for 14 days	265 GB × 14 ≈ 3.716 TB	One unreplicated retained copy.
Data with three replicas	3.716 TB × 3 ≈ 11.15 TB	Disk before indexes, safety headroom, and replica movement.
Broker disk assumption	1 TB usable per broker, operated to 70%	Leaves room for recovery and imbalance.
Bare disk floor	11.15 / (1 × 0.70) ≈ 16 brokers	Arithmetic minimum only. Start closer to 20 only after checking each zone’s disk, network, and recovery headroom.

Peak network, consumer parallelism, and backlog

Scenario	Calculation	Planning result	Important limit
Peak producer ingress	10,000 records/s × 1 KiB	10.24 MB/s	Does not include protocol overhead or retries.
Replica traffic	Peak ingress × two follower copies	20.48 MB/s	Copies move between brokers; uneven leader placement can concentrate it.
Consumer egress	Peak ingress × two full-stream groups	20.48 MB/s	Each independent group reads its own copy of the stream over the network.
Combined replica + consumer egress	20.48 + 20.48	40.96 MB/s	Excludes protocol, rebalances, retries, and skew.
One group’s maximum parallel workers	64 partitions	64 active owners at most	More workers than partitions sit idle; a hot key can still overload one partition.
Backlog growth	300 incoming - 200 processed records/s	100 records/s, or 360,000 records/hour	At 1 KiB, about 368.6 MB/hour accumulates until consumption exceeds production.
Tenfold sustained traffic	11.15 TB × 10 replicated retained bytes	about 111.5 TB; roughly 160 one-terabyte brokers at 70%	The next limit may instead be a hot key, network egress, metadata load, or downstream effects.

Adding consumers helps only when there are unused partitions and the downstream service has capacity. “One broker handles 1 TB and 10,000 records/s” is not a universal fact: payload size, disks, network, compression, replication, and fan-out all change the answer.

What each scaling lever buys

Scaling one part does not automatically raise the capacity of the others. The table shows the useful move and its ceiling.

Lever	Move	Limit or transition cost
Producers	Add client instances and batch by destination	A hot routing key still lands on one partition; broker byte quotas may throttle clients.
Consumers	Add members to a group, or add an independent group for another use case	A group has at most one active owner per partition; more members than partitions sit idle. Rebalancing can replay uncommitted work.
Brokers	Add brokers and move replicas to spread disk, network, and leaders	Create and catch up replacement replicas before removing old ones; copying consumes capacity. Keep replicas on separate failure domains.
Partitions	Plan enough parallelism early; add partitions when the group or leader count is the limit	Old records stay in their original partitions. Clients refresh metadata, groups rebalance, and simple key hashing may remap new records. More partitions increase per-broker file handles and replication overhead, and can lengthen failover because each partition elects independently.

Removing a broker uses the same safe sequence in reverse: establish caught-up copies elsewhere, move leadership as needed, then remove its replicas. Kafka supports increasing a topic’s partition count but not decreasing it in place; to consolidate, create a replacement topic and migrate producers and consumers while the old topic drains or expires. Marking a partition read-only until retention expires is a possible new-topic migration policy, not a Kafka partition-decrease operation. Kafka topic operations

Issues, challenges, and limitations

Failure cases and remaining limits

Each mitigation has a remaining limit; recovery is an operation, not a guarantee that the original request completed.

Failure or pressure	Response	Remaining limit
Leader or zone fails	Controller promotes an eligible in-sync replica; clients refresh routes	Writes pause; correlated replica loss or unflushed power loss can lose data.
Follower lags	Exclude it from in-sync set, repair or replace it	Below the minimum set, writes fail until capacity returns.
Downstream service fails	Bounded backoff, circuit breaker, DLQ policy	Lag grows; retention can expire unread records.
Offset is older than retention	Fail the fulfillment group visibly, alert, and choose an explicit backfill or order reconciliation plan before resetting	Resetting to latest silently skips orders; missing history may need another source.
Disk fills or tenant overloads	Quotas, alerts, and added capacity; throttle or reject new writes at the disk limit	Accepted records keep the 14-day minimum, but new writes may fail until capacity returns.
Controller or progress store stalls	Keep serving safe existing paths where possible; restore quorum	Leader changes, joins, or commits may stop; do not claim full availability.
Hot key or 10× growth	Isolate heavy streams, review key design and placement	Splitting one ordered key sacrifices its single-partition order.

A hot partition can arise even when the cluster has free capacity. If the application can relax its ordering boundary, use a better-distributed or composite routing key; if it cannot, isolate that stream and apply producer backpressure rather than claiming another consumer can process the same ordered partition concurrently. Enforce per-client byte-rate quotas at brokers, with explicit throttle responses and bounded client buffers. Quotas contain a noisy tenant but cannot create capacity for a persistently overloaded cluster.

Signals that expose the failure

Alert on oldest unprocessed record age, consumer lag, under-replicated partitions, in-sync set size, leader changes, disk use, throttling, retry rate, DLQ rate, broker acknowledgment latency, and end-to-end publish latency. Stored records remain after consumption, so group lag is the backlog signal. Track authentication failures and administrative changes for audit.

Quiz

Broker design challenge

Can you keep the stream correct?

Five failure scenarios. Choose the guarantee you can actually defend.

1 / 5

Question 1 of 5

A producer times out after sending an order event. The leader may have appended it, but the producer never received the acknowledgment. What is the safest retry?

ASend a new event with a new identityBRetry with the same stable publish identity and sequenceCQuery every consumer before retryingDAssume the event failed and update the database directly