Capacity Planning for 1 Million Messages per Second

Sustaining 1 million messages per second in Apache Kafka pushes a cluster out of general-purpose-message-bus territory and into high-performance data infrastructure. At this scale, bottlenecks that are theoretical in small clusters become hard limits: network interface bandwidth, disk I/O, replication overhead, and the page cache hit ratio. The job of capacity planning is to translate a throughput target into a concrete bill of materials — broker count, CPU cores, RAM, NVMe drives, and NIC speed — with enough headroom to survive a broker failure without dropping the target.

This guide works through a calculation-driven model for sizing a cluster to 1 million messages per second. Every assumption is made explicit so you can re-run the arithmetic against your own workload. It is a scale-specific extension of the broader discipline in Capacity Planning for Kafka Clusters: Throughput and Storage; the same estimation principles apply, but here we focus on the practical limits of NICs, disk subsystems, and replication. The wider context — metadata management, rack awareness, hardware selection — lives in Cluster Architecture & Provisioning.

Defining the Workload and Core Assumptions

“1 million messages per second” is meaningless without a message size. One million 100-byte messages and one million 10 KB messages are entirely different machines. For this exercise we assume a typical event-streaming/telemetry profile:

• Target throughput (T): 1,000,000 messages/s.
• Average message size (S): 1 KB (1,024 bytes).
• Replication factor (RF): 3.
• Retention: 24 hours (86,400 seconds).
• Producer acks: acks=all — the durable, safe default for this class of workload.
• Compression: producer-side compression.type=lz4, a good speed/ratio balance for text and JSON, often achieving roughly 2x–4x reduction.

From these we derive the baseline:

• Uncompressed ingest: 1,000,000 × 1,024 bytes ≈ 1,024 MB/s (~1 GB/s).
• With RF=3, every byte a leader writes is also written by two followers, so the aggregate disk write demand across the cluster is 3 × ingest.

Why compression changes the numbers that matter

The 1 GB/s figure is application-level throughput — the logical record rate your producers and consumers reason about. It is not what crosses the wire or lands on disk. With compression.type=lz4 producers and the broker default compression.type=producer, the data is compressed once, at the producer, and stays compressed end to end.

Assume a conservative 2.5x lz4 ratio, giving an effective on-disk record size of ≈ 410 bytes:

The reason the replication row stays compressed is mechanical: the broker stores the producer’s already-compressed batches verbatim, and followers fetch those same compressed batches. Kafka does not decompress and recompress on the leader unless the topic’s codec differs from the producer’s. The practical rule follows directly:

• Plan storage and inter-broker bandwidth on the compressed numbers (~410 MB/s ingest, ~820 MB/s replication).
• Size producer NICs and consumer-decode CPU on the logical record rate (~1 GB/s), since decompression happens at the consumer.

Get this wrong in the optimistic direction and you under-provision disk; get it wrong in the pessimistic direction and you buy 2.5x the storage you need.

Network Throughput: The First Hard Ceiling

Network bandwidth is the most common ceiling at this scale. Each broker carries three flows: inbound producer traffic to the partitions it leads, replication (outbound to followers for partitions it leads, inbound for partitions it follows), and consumer fetch egress.

Working in compressed bytes — what actually crosses the wire:

• Producer ingest: ≈ 410 MB/s into the leaders.
• Replication: each leader ships every batch to RF−1 = 2 followers ≈ 820 MB/s in aggregate.
• Consumer egress: workload-dependent. Assume ≈ 410 MB/s for one balanced consumer group reading the stream once; each additional consumer group adds another ~410 MB/s of fan-out.

That is ≈ 1.6 GB/s aggregate (~13 Gbps) for a single consumer group, spread across the cluster. The figure is dominated by fan-out, not ingest: three consumer groups already push egress past replication and ingest combined. If you discard the compression benefit on the wire — uncompressed producers that force broker-side compression, or many uncompressed fanned-out consumers — the logical aggregate climbs toward 4 GB/s (~32 Gbps). Plan the NIC for your worst-case fan-out, not the steady-state single-consumer case.

Choosing a NIC

Never plan links above ~70% utilization; you need slack for failover bursts and re-replication. After that cap and protocol overhead:

The compressed ~13 Gbps aggregate spread perfectly would suggest as few as ~3 brokers on 10 GbE, but you should provision more for failure headroom (see the final section) and to bound per-broker partition leadership. The marginal cost of 25 GbE over 10 GbE is usually small relative to operating more brokers, and it removes the NIC as the thing you worry about first. For this workload, 25 GbE is the default choice.

Monitor live per-interface throughput on a broker with sar from the sysstat package:

# Per-interface network throughput, 10 one-second samples (Linux)
sar -n DEV 1 10 | grep -E "eth0|ens"

Disk I/O and Storage Capacity

Throughput. With end-to-end compression the cluster sustains ≈ 1,230 MB/s of replicated writes (410 MB/s ingest × RF=3). A single modern NVMe SSD sustains roughly 2–3 GB/s of sequential writes, so the raw rate is met by a few drives. But Kafka’s pattern is not purely sequential append: segment rolling, index writes, log cleaning/compaction, and the periodic page-cache flush add random I/O and latency spikes. Size for those, not just the headline sequential number.

Capacity. Storage is driven by retention. Using the compressed on-disk record size of ~410 bytes:

• Daily unreplicated volume: 1,000,000 × 410 bytes × 86,400 s ≈ 35.4 TB/day.
• Replicated (RF=3): 35.4 × 3 ≈ 106 TB/day.

Add overhead for segment/index files, the __consumer_offsets topic, and filesystem slack — a 1.2x factor brings this to ≈ 127 TB. On 7.68 TB NVMe drives that is ≈ 17 drives across the cluster, e.g. ~3 drives per broker in a 6-broker cluster. This 35.4 TB is the compressed footprint; because Kafka stores the compressed batches, it is the number that consumes capacity.

Mount data drives with noatime and use XFS (or ext4); XFS is the more common choice for Kafka at scale:

# Example /etc/fstab entry for a Kafka data drive
/dev/nvme0n1 /data/kafka xfs defaults,noatime,nodiratime 0 2

Watch utilization and latency under load:

# Extended per-device I/O stats, 10 one-second samples
iostat -x 1 10

The columns that matter are %util (saturation) and await/w_await (latency). A drive consistently above ~70% %util or with write latency in the tens of milliseconds is a bottleneck. Treat %util cautiously on NVMe — for devices with deep parallel queues it can read near 100% while the device still has headroom, so corroborate with await and actual MB/s.

CPU and Memory Sizing

Kafka is comparatively CPU-light, but at 1M msg/s CPU is a first-class concern. The main consumers are TLS termination, request/network processing, replication, and — if the broker has to recompress on a codec mismatch — compression. With compression.type=producer and a matching producer codec, the broker avoids recompression, leaving network and replication as the dominant costs.

As a rough planning figure, a modern server core handles on the order of 50–100 MB/s of Kafka traffic depending on TLS, compression, and request size. Covering ingest plus replication for this workload lands around 30–60 cores cluster-wide; in a 6-broker cluster that is ~5–10 cores per broker, well within a single 16-core Xeon or EPYC. Validate this empirically — TLS and small-batch workloads can easily double CPU per byte.

Memory is governed by the page cache, which is how Kafka serves consumer reads from RAM instead of disk. You cannot cache 24 hours of data — that is ~35 TB compressed. Size instead for the active working set: the recent window consumers actually read. At 1M msg/s, 15 minutes of compressed data is ≈ 370 GB cluster-wide (~60 GB per broker in a 6-broker cluster) — still larger than typical RAM, so accept that cold reads hit NVMe and provision fast disks accordingly. The realistic goal is to keep the tail of each partition — the last few segments that lagging consumers and followers read — resident, not the whole retention window.

Set a fixed, modest JVM heap. 8–12 GB is ample for a broker; Kafka deliberately keeps data off-heap in the page cache, so a larger heap mostly hurts GC:

# Pin the broker heap (avoid growth/GC churn)
export KAFKA_HEAP_OPTS="-Xmx10G -Xms10G"

Leave the rest of system RAM (e.g. 64 GB+ total) to the OS page cache. It will not hold the full working set, but it absorbs reads for hot partitions and the replication tail.

Partition Count and Producer Configuration

Partitions are the unit of parallelism. Within a consumer group, each partition is read by at most one consumer, and a single partition’s write throughput is bounded by one leader broker’s resources. A common rule of thumb puts a single partition in the low tens of MB/s; to spread ~410 MB/s of compressed ingest you want dozens of partitions at minimum, and more for consumer parallelism and rebalance headroom.

For this scenario, 100 partitions gives comfortable headroom: ~17 leaders per broker in a 6-broker cluster. Kafka handles thousands of partitions per broker under KRaft, but more partitions increase metadata, end-to-end latency, and leadership-failover time, so do not over-shard. Match the partition count to your peak required consumer parallelism plus a margin — and remember it sets a ceiling you cannot easily lower, since reducing partition count requires recreating the topic.

Producer settings that matter at this throughput:

# High-throughput, durable producer
acks=all
enable.idempotence=true
compression.type=lz4
batch.size=65536
linger.ms=5
max.in.flight.requests.per.connection=5

Notes:

• enable.idempotence=true is the default in modern Kafka and implies acks=all and retries>0. It preserves per-partition ordering across retries as long as max.in.flight.requests.per.connection is ≤ 5 — hence the value 5, which keeps pipelining without sacrificing ordering.
• batch.size=65536 (64 KB) plus linger.ms=5 lets the producer accumulate larger batches, cutting per-request overhead and improving the compression ratio. Larger batches generally help throughput up to the point where added latency becomes unacceptable.
• min.insync.replicas is a broker/topic setting, not a producer property. Set it to 2 on the broker or topic (see below) so that acks=all requires the leader plus one follower: this tolerates one in-sync replica being down while still rejecting writes when only the leader remains.

Broker Configuration and OS Tuning

The defaults are deliberately conservative. For this throughput, raise the thread pools, socket buffers, and replication fetchers:

# server.properties — high-throughput broker
num.network.threads=8
num.io.threads=16
num.replica.fetchers=4
socket.send.buffer.bytes=1048576
socket.receive.buffer.bytes=1048576
socket.request.max.bytes=104857600
min.insync.replicas=2
default.replication.factor=3
num.partitions=100
compression.type=producer
log.segment.bytes=1073741824
log.retention.hours=24
unclean.leader.election.enable=false

What changed from defaults and why:

• num.network.threads (default 3) handles socket I/O; num.io.threads (default 8) handles request processing including disk. Scale these toward the core count, but measure — over-provisioning threads wastes context-switch budget.
• num.replica.fetchers (default 1) is the often-missed knob for replication throughput: it sets how many threads each broker uses to pull from leaders. With RF=3 and 100 partitions, 4 fetchers materially speeds catch-up and re-replication. This is what actually keeps replication off the critical path; the network budget alone does not.
• socket.send.buffer.bytes / socket.receive.buffer.bytes default to 100 KB; 1 MB reduces syscalls on fat, high-latency links. (Set -1 to let the OS autotune.)
• socket.request.max.bytes default is 100 MB (104857600). Leave it at the default unless you genuinely send larger requests; raising it without bound lets a single oversized request pressure broker memory.
• compression.type=producer keeps the producer’s codec end-to-end, avoiding broker recompression CPU and a re-replication cost.
• unclean.leader.election.enable is false by default and should stay false: with min.insync.replicas=2, durability over availability is the whole point of this design, and enabling unclean election would let an out-of-sync replica become leader and silently drop acknowledged writes.
• log.retention.hours is a server-level default; per-topic retention is set with the retention.ms topic config, which takes precedence and is the only one tunable after startup.

At the OS level, raise file-descriptor limits (Kafka opens an FD per segment/index) and tune the TCP stack:

# /etc/security/limits.conf
kafka soft nofile 131072
kafka hard nofile 131072

# /etc/sysctl.d/99-kafka.conf
net.core.rmem_max = 134217728
net.core.wmem_max = 134217728
net.ipv4.tcp_rmem = 4096 87380 134217728
net.ipv4.tcp_wmem = 4096 65536 134217728
net.ipv4.tcp_max_syn_backlog = 8192
vm.swappiness = 1

These let the kernel buffer large network windows and discourage swapping the broker heap. The Linux IP sysctl documentation explains each TCP parameter.

The Resulting Bill of Materials

Pulling the arithmetic together, a 6-broker cluster sized for this workload looks like:

Every cell is a re-runnable function of the assumptions at the top. Change the message size or retention and the table moves with it.

Validating the Design with Load Testing

No plan is real until you have driven the cluster to target and measured it. Use Kafka’s bundled kafka-producer-perf-test.sh to generate load:

bin/kafka-producer-perf-test.sh \
  --topic test-topic \
  --num-records 10000000 \
  --record-size 1024 \
  --throughput 1000000 \
  --producer-props bootstrap.servers=broker1:9092,broker2:9092 \
                   acks=all compression.type=lz4 linger.ms=5 batch.size=65536

--throughput -1 removes throttling to find the ceiling; a positive value pins a target rate so you can observe latency at that load. Pair it with kafka-consumer-perf-test.sh to measure end-to-end fetch throughput, and run enough consumer groups to reproduce your real fan-out — a single consumer group will understate egress.

Watch broker JMX during the run. The load-bearing MBeans are:

• kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec and name=BytesOutPerSec — produce/replication/consume byte rates.
• kafka.network:type=RequestMetrics,name=TotalTimeMs,request=Produce — end-to-end produce latency. Its breakdown MBeans (RequestQueueTimeMs, LocalTimeMs, RemoteTimeMs, ResponseQueueTimeMs, ResponseSendTimeMs) tell you where time goes; with acks=all, RemoteTimeMs is the wait for follower acknowledgement.
• kafka.server:type=ReplicaManager,name=UnderReplicatedPartitions — should be 0 in steady state; non-zero means replication is falling behind.

Combine those with OS-level iostat/sar. The point of load testing is not a pass/fail at 1M msg/s — it is to find which resource saturates first, so you know what to add when the real workload grows past the model.

Failure Scenarios, Detection, and Headroom

A cluster sized exactly for the target fails the instant a broker dies. Size for N−1: in a 6-broker cluster, each broker must absorb 1/5 of the load when one is down, not 1/6 — a 20% per-broker increase you must provision for up front.

You also pay for re-replication. When a broker is lost, surviving brokers re-replicate its partitions to restore RF; this consumes network and disk on top of steady-state traffic, and num.replica.fetchers governs how fast it completes. Budget 30–50% headroom above steady state. Combined with the N−1 share, this is the real reason a 6-broker cluster provisions each broker well above the nominal 1/6.

Finally, account for consumer-group rebalances: while a group rebalances, processing pauses and lag accumulates, after which the group must catch up faster than real time without saturating the cluster. This — along with re-replication and N−1 failover — is the underlying reason to keep every resource at or below ~70% in steady state.

Each failure mode has a concrete signal; wire these into alerting rather than discovering them in an incident:

A follower that lags past replica.lag.time.max.ms is removed from the ISR; if min.insync.replicas=2 and that leaves only the leader in sync, acks=all producers begin failing — which is the system correctly choosing durability over availability, not a bug to tune away. The Apache Kafka monitoring documentation lists the full MBean catalog and recommended alerts.

The arithmetic here is simple; the discipline is in validating every assumption and instrumenting every component. At this scale there are no minor details — only the ones you have not measured yet.