Broker Sizing and Hardware Selection for Kafka Clusters

Broker sizing is one of the most consequential decisions in building an Apache Kafka platform. Get it wrong and you face either runaway infrastructure cost from over-provisioning or — worse — production incidents from resource exhaustion under peak load. This guide gives an engineering methodology for translating a measured workload into concrete hardware specifications and broker counts.

The guidance assumes a multi-tenant, production-grade deployment where predictable latency, durability, and availability are non-negotiable. These choices directly shape the foundational Cluster Architecture & Provisioning strategy you adopt. Broker sizing is never one-size-fits-all: a cluster optimized for high-throughput telemetry with relaxed latency SLOs looks radically different from one built for low-latency, transactional event sourcing. The work proceeds in four stages — profile the workload, translate the profile into per-resource demands, calculate broker count, then select hardware — and it interacts with the metadata plane, since KRaft vs ZooKeeper architectures change the CPU and memory headroom you must leave on controller nodes.

Workload Profiling: Throughput, Retention, and Latency

Before sizing a single broker, quantify the work across three dimensions: throughput, retention, and latency. Rigorous profiling prevents the two most common failure modes — underestimating network bandwidth, which causes backpressure and producer timeouts, and underestimating storage I/O, which causes unbounded consumer lag during read-heavy retention windows.

Quantifying Throughput: Ingress and Egress

Throughput is measured in messages per second and, more critically, in megabytes per second. Kafka’s batching and compression mean message count is a poor proxy for resource use. A workload of 100,000 small 100-byte messages per second (10 MB/s) strains CPU and network very differently from 1,000 large 1 MB messages per second (1,000 MB/s), even though the latter has a lower message rate.

Profile ingress (producer-to-broker) and egress (broker-to-consumer) independently. Many teams size only for ingress, forgetting that one produced message can be consumed by multiple consumer groups, multiplying egress. If five consumer groups read a topic with 100 MB/s of ingress, aggregate egress reaches 500 MB/s, saturating links sized only for the producer side.

Use the broker JMX metrics kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec and kafka.server:type=BrokerTopicMetrics,name=BytesOutPerSec, aggregated across all brokers. For pre-production sizing, run load tests with representative payload sizes, batching configurations, and compression codecs. Sample a metric from a running broker with JmxTool (deprecated as of Kafka 3.5 but still ships and works; for production, prefer scraping via the JMX Exporter):

kafka-run-class.sh org.apache.kafka.tools.JmxTool \
  --object-name kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec \
  --jmx-url service:jmx:rmi:///jndi/rmi://broker1:9999/jmxrmi \
  --reporting-interval 1000

Record peak throughput over 1-minute and 5-minute windows and size for the peak, not the average — averaging hides exactly the bursts that cause incidents. If the workload is diurnal, capture a full 24-hour profile and size for the maximum sustained peak. One subtle trap: the BytesInPerSec rate excludes replication traffic, but the NIC does not. Always re-introduce the replication multiplier (covered below) before you draw a network conclusion from these numbers.

Retention Policy and Storage Footprint

Retention drives storage capacity directly. At the broker level Kafka supports time-based retention (log.retention.hours, log.retention.minutes, log.retention.ms) and size-based retention (log.retention.bytes); the effective retention is whichever limit is hit first. These broker defaults can be overridden per topic with retention.ms and retention.bytes. Estimate the footprint as:

Storage = Ingress_Bytes_per_Second × Retention_Seconds × Replication_Factor

For a topic ingesting 50 MB/s with 7-day retention (604,800 s) and replication factor 3:

50 MB/s × 604,800 s × 3 = 90,720,000 MB ≈ 90.7 TB (≈ 86.5 TiB)

Distribute this across the cluster. With 10 brokers and uniform placement, each broker provisions roughly 9 TB of usable storage for that topic alone. Uniform distribution is not guaranteed: partition skew — where some partitions receive disproportionate traffic, often because a hot key or a low-cardinality partition key concentrates writes — fills some brokers faster than others. Add a 20-30% safety margin to absorb skew, compaction overhead, and temporary spikes, and remember that retention is enforced at segment granularity. A partition cannot delete its active segment, and segments roll at log.segment.bytes (default 1 GB) or log.roll.ms, so a low-throughput partition can hold data well past its nominal retention window until the active segment finally rolls. Size for that lag on small-but-long-retention topics.

For compacted topics the calculation is harder. Log compaction retains at least the latest value per key, so storage depends on key cardinality and average message size, not on ingress rate. Monitor kafka.log:type=LogCleanerManager,name=max-dirty-percent to confirm the cleaner keeps up. If the dirty ratio runs high, raise log.cleaner.threads or the cleaner I/O throttle (log.cleaner.io.max.bytes.per.second). The failure signature of a starved cleaner is storage that grows without bound on a topic you expected to stay flat — alert on per-topic log size, not just dirty percent, because a cleaner that is merely slow (not stalled) still leaks disk.

Latency SLOs and Their Impact on Sizing

Latency requirements determine whether you can tolerate spiky I/O or must provision for sustained low-latency performance. A 99th-percentile produce-latency SLO of 10 ms cannot afford page-cache misses that trigger disk reads, so the hot data set must fit in the page cache — which means sizing memory to the working set of hot partitions.

Define SLOs explicitly: end-to-end latency from producer send to consumer receive, broker-side produce latency, and fetch latency. Measure client-side latency with the producer and consumer metrics under kafka.producer:type=producer-metrics,client-id=* and kafka.consumer:type=consumer-fetch-manager-metrics,client-id=*. If SLOs are strict, over-provision memory and prefer SSDs with high random-read IOPS over high-capacity HDDs.

Translating Workload Profiles into Resource Demands

With a quantified profile in hand, map throughput, retention, and latency onto CPU, memory, storage, and network. The translation is not linear; Kafka’s performance changes at inflection points set by batching efficiency, compression ratios, and I/O capabilities.

CPU Sizing: Threads, Compression, and SSL

Kafka brokers use a network thread pool and an I/O thread pool, configurable via num.network.threads and num.io.threads (defaults 3 and 8). On busy brokers a common starting point is to raise both, keeping I/O threads at roughly the count of data directories times a small multiple. The largest CPU demands, though, come from compression and TLS.

When producers compress (recommended for most workloads), the broker decompresses to validate records and assign offsets, then may recompress before writing. Setting compression.type=producer on the broker retains the producer’s codec and avoids recompression on the write path. For predictable CPU, standardize on one codec — lz4, snappy, or zstd — across producers and keep compression.type=producer on the broker. The gotcha: mixing codecs across producers on the same topic, or leaving the broker default (producer) while some clients send uncompressed, produces a broker that recompresses unpredictably and a CPU graph you cannot capacity-plan against.

TLS is CPU-intensive. A broker terminating ~1 Gbps of encrypted traffic can consume several modern x86 cores for handshakes and symmetric encryption. Provision headroom and use hardware with AES-NI, which accelerates symmetric ciphers substantially. Benchmark AES throughput on candidate hardware:

openssl speed -elapsed -evp aes-256-gcm

(Kafka’s TLS cipher suites are predominantly AEAD GCM modes; benchmark aes-256-gcm or aes-128-gcm rather than CBC to reflect real traffic.) Monitor request-handler saturation via kafka.server:type=KafkaRequestHandlerPool,name=RequestHandlerAvgIdlePercent — a gauge from 0 (fully busy) to 1 (fully idle). If it falls below ~0.2, the I/O thread pool is starved; add CPU, raise num.io.threads, or add brokers.

Memory and the Page Cache Advantage

Kafka leans heavily on the OS page cache for read performance. When a consumer fetches records still resident in the page cache, the broker serves them from memory without touching disk. JVM heap is separate: the heap handles broker operations while the page cache uses the remaining system RAM.

A well-tuned broker allocates roughly 6 GB to the JVM heap (via KAFKA_HEAP_OPTS) and leaves the rest of physical RAM to the page cache. On a 64 GB host this leaves ~56 GB for caching. If the working set — the volume actively being read — exceeds the page cache, reads start hitting disk and latency degrades sharply. Observe page-cache behavior with cachestat from the bcc-tools suite (it reads kernel tracepoints and needs root):

sudo cachestat 1 10

A persistently low hit ratio means the working set exceeds available memory. The tell-tale operational signature is fetch latency that climbs in lockstep with consumer lag: lagging consumers read old segments that are no longer cached, evicting hot data and dragging well-behaved consumers down with them. The fix is to add RAM, add brokers to spread the working set, or reduce retention to shrink the active data set — and, where the cold reads come from a handful of catch-up consumers, tiered storage (below) keeps those reads off the local page cache entirely.

Storage Subsystem: IOPS, Throughput, and Capacity

Storage is where most sizing mistakes occur. Kafka’s write pattern is sequential append, friendly to HDDs and SSDs alike, but reads become random when consumers fetch from many partitions or catch up on historical data. The subsystem must deliver sequential write throughput for ingestion and random read IOPS for fetches.

Derive write throughput from the ingress profile. A cluster ingesting 200 MB/s across 10 brokers needs roughly 20 MB/s of sequential writes per broker, within reach of a single HDD (100-150 MB/s sequential). The read side is more demanding: with no locality, reads become random. A 7,200 RPM HDD delivers ~75-100 random IOPS, only a few MB/s of random reads at small message sizes.

For workloads with significant random reads, SSDs are effectively mandatory. A commodity NVMe SSD delivers 100,000+ random read IOPS, removing the I/O bottleneck for most workloads; the trade-off is cost per GB. A common pattern places low-latency, read-heavy topics on SSD-backed brokers and high-retention, write-heavy telemetry on HDD-backed brokers. Note that Kafka does not support pinning a topic to a specific log directory through topic configuration — log.dirs is a broker-level setting only, and there is no topic-level directory override. To separate storage tiers you place such topics on dedicated brokers (or dedicated clusters), or use Kafka’s Tiered Storage to offload cold segments to object storage.

Benchmark storage with fio using a Kafka-like pattern. Sequential writes with 64 KB blocks approximate segment writes:

fio --name=kafka-write --ioengine=libaio --direct=1 --bs=64k \
    --size=10G --rw=write --numjobs=4 --group_reporting \
    --filename=/mnt/kafka-data/fio-test

For the read side, simulate random reads:

fio --name=kafka-read --ioengine=libaio --direct=1 --bs=4k \
    --size=10G --rw=randread --numjobs=8 --group_reporting \
    --filename=/mnt/kafka-data/fio-test

Compare results against your calculated requirements and keep at least 30% headroom for spikes.

Network Bandwidth and Throughput Ceilings

Network bandwidth is frequently the first resource to bottleneck. A broker’s NIC carries producer ingress, consumer egress, and replication to followers. With replication factor 3, a leader sends each produced byte to two followers, so:

Total_Network = Ingress + Egress_to_Consumers + (Replication_Factor - 1) × Ingress

For a leader handling 100 MB/s of producer ingress and 200 MB/s of consumer egress at RF=3:

100 + 200 + (2 × 100) = 500 MB/s

That argues for a 10 GbE interface (~1,250 MB/s) to retain headroom; do not run production brokers above trivial throughput on 1 GbE. In cloud environments, instance network bandwidth is often burst-based and throttled after sustained use — consult your provider’s baseline and burst limits, and size for the baseline.

Broker Count Calculation and Distribution

With per-broker requirements understood, determine how many brokers you need by balancing fault tolerance, scalability, and operational complexity. The How to Calculate Optimal Broker Count for Your Workload guide gives a detailed calculator; the core methodology follows. Compute the count three ways — partition balance, throughput, and storage — and take the maximum.

Partition Distribution and Leadership Balance

Partition count sets the floor for balanced leadership. Each partition has one leader and several followers, and Kafka spreads leaders across brokers — but only as evenly as the partition count allows. With 100 partitions and 10 brokers, each broker leads about 10 partitions; with 11 brokers the split is uneven. Aim for a partition count that is a multiple of the broker count for even leadership.

Use kafka-reassign-partitions.sh in --generate mode to produce a candidate plan that spreads topics across brokers:

kafka-reassign-partitions.sh --bootstrap-server broker1:9092 \
  --topics-to-move-json-file topics.json \
  --broker-list "0,1,2,3,4,5,6,7,8,9" \
  --generate

This prints a proposed reassignment.json you review (and optionally edit) before applying with --execute. Even with good placement, monitor kafka.server:type=ReplicaManager,name=UnderReplicatedPartitions to catch replication shortfalls and leadership imbalance after restarts or network partitions.

Throughput-Driven Broker Count

Divide total throughput by per-broker capacity. If the cluster must handle 1,000 MB/s of ingress and each broker sustains 150 MB/s (limited by network or disk), you need at least 7 brokers — but you must also survive failures. Losing one of 7 brokers raises load on each survivor by roughly 17%, which may exceed capacity.

For N+1 redundancy, size so the cluster handles full load with one broker down. For the example above, provision 8 brokers (7 for load plus 1 spare). In the cloud, account for Availability Zone failures: if you deploy across three AZs as in Multi-AZ Kafka Deployment Strategies, confirm that losing an entire AZ does not drop capacity below the required minimum. With three AZs that means roughly a third of brokers can vanish at once — a far harsher constraint than single-broker N+1, and the reason AZ-spread clusters often run materially more brokers than throughput math alone suggests.

Storage-Driven Broker Count

Storage often dictates a higher broker count than throughput alone. The footprint formula already includes replication, so if your retention math yields 200 TB of total (replicated) data and each broker holds 10 TB of usable disk, you need at least 20 brokers. Keep replication and total-storage accounting consistent — do not multiply by the replication factor twice.

With JBOD (multiple data directories per broker), per-broker usable capacity is the sum of the directories. Be deliberate with JBOD: if one disk fails, Kafka marks only the partitions on that log directory offline (other directories keep serving), but that still triggers leadership changes for the affected partitions. In KRaft mode, robust per-directory failure handling arrived with KIP-858 (Kafka 3.7+). Mitigate disk-failure blast radius with RAID-10 for redundancy (at the cost of usable capacity), RAID-0 only where you accept whole-broker loss on a single disk failure, or by ensuring the replication factor tolerates losing one disk’s worth of partitions.

Hardware Selection Matrix: CPU, Memory, Storage, and Network

With requirements quantified, select components. The table below summarizes three common profiles; the sections that follow give the reasoning. Treat these as starting points to validate with the fio, openssl speed, and load-test measurements above — not as universal truths.

CPU Selection: Core Count vs. Clock Speed

Kafka’s threading model benefits more from core count than raw clock speed, especially with TLS and compression. Server CPUs with 16-32 cores provide ample parallelism for network and I/O threads. For high-throughput telemetry, prioritize cores: a 32-core host comfortably runs large network and I/O thread pools with headroom for the OS and background tasks.

For low-latency workloads, single-thread performance matters more because per-request handling is sensitive to clock speed. Choose CPUs with higher base frequencies and avoid models that throttle aggressively under sustained load; recent Intel Xeon Scalable or AMD EPYC parts with 3.0 GHz+ base frequencies are suitable. Avoid burstable cloud instance types for latency-sensitive brokers — CPU credits can exhaust and cause sudden latency spikes that look like a Kafka problem but are an instance-class problem.

Memory Configuration: Heap and Page Cache

As established, allocate roughly 6 GB to the JVM heap and the remainder to the page cache. On a 64 GB host, set KAFKA_HEAP_OPTS="-Xms6g -Xmx6g". On a 128 GB host you can raise the heap toward 8-10 GB when a broker hosts very many partitions, but going much higher risks longer GC pauses that surface as produce-request timeouts; large heaps rarely help because the page cache, not the heap, drives read performance.

Kafka ships with the G1 collector and sane defaults. Its bundled KAFKA_JVM_PERFORMANCE_OPTS is equivalent to:

export KAFKA_JVM_PERFORMANCE_OPTS="-server -XX:+UseG1GC \
  -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 \
  -XX:+ExplicitGCInvokesConcurrent -Djava.awt.headless=true"

These defaults are well-tuned; change them only with evidence. Watch GC with jstat -gcutil <pid> 1000 and validate that pauses stay within the MaxGCPauseMillis target under peak load. Sustained pauses well above the target usually mean too large a heap or too many partitions per broker — add brokers to reduce per-broker partition count.

Storage Media: NVMe SSDs vs. HDDs

For low-latency workloads, NVMe SSDs are the only viable choice; they supply the random read IOPS to serve fetches from disk on a page-cache miss. A single datacenter NVMe drive sustains hundreds of thousands of random read IOPS and multiple GB/s of sequential writes, far exceeding any single broker’s needs. Use multiple NVMe drives as separate JBOD directories to add capacity and parallelize I/O.

For high-throughput telemetry with relaxed latency SLOs, enterprise 7,200 RPM HDDs remain cost-effective. Use 4+ drives per broker as JBOD to aggregate write throughput into the 400-600 MB/s range. Avoid consumer-grade HDDs, which lack the vibration tolerance and error-recovery controls needed in dense server chassis. One operational cost that surprises HDD-JBOD operators: after an unclean shutdown the broker rebuilds log indexes and recovers segments on startup, parallelized by num.recovery.threads.per.data.dir (default 1 through Kafka 4.0; 2 from 4.1). On a broker with many large directories this can stretch startup to minutes, lengthening the window of reduced redundancy after a restart — raise this value on dense HDD brokers.

A small NVMe drive as a write-back cache in front of HDDs is possible at the OS/storage layer but adds complexity and is not a Kafka feature. Prefer Kafka’s tiered storage to separate hot and cold data — see Tiered Storage. Tiered storage (KIP-405) is generally available and marked production-ready as of Kafka 3.9.

Network Interface: 10 Gbps Minimum

Provision 10 GbE on all production brokers. In the cloud, select instance types that guarantee 10 Gbps of baseline network throughput rather than “up to 10 Gbps” burst networking, where the baseline can be as low as 1-2 Gbps and will throttle replication during peak loads. For brokers exceeding ~500 MB/s of total throughput, consider 25 GbE or higher; the premium is usually justified by eliminating network bottlenecks, which are hard to diagnose because they surface as latency rather than explicit errors.

Rack Awareness and Fault Tolerance Configuration

Hardware selection is half the battle; how you spread it across failure domains determines resilience. Kafka’s rack awareness distributes partition replicas across racks (or AZs) so a single rack failure does not cause data loss or unavailability.

Configuring Rack Awareness

Set broker.rack on each broker to identify its rack or AZ:

broker.rack=us-east-1a

Kafka then places each partition’s replicas in distinct racks, provided the replication factor does not exceed the number of racks. With RF=3 and three racks, every partition lands one replica per rack.

Rack-aware replica placement is automatic from broker.rack. To let consumers read from a same-AZ follower instead of always the leader — cutting cross-AZ transfer cost and latency — enable follower fetching (KIP-392, Kafka 2.4+) by setting replica.selector.class=org.apache.kafka.common.replica.RackAwareReplicaSelector on the brokers and setting client.rack on each consumer to its AZ. Without client.rack the broker has no rack to match and consumers fall back to the leader. Inspect replica placement with:

kafka-topics.sh --bootstrap-server broker1:9092 \
  --describe --topic my-topic

The output lists the replica broker IDs per partition; cross-reference those IDs against each broker’s broker.rack to confirm replicas are spread across racks. If two replicas share a rack, reassign to correct it.

Multi-AZ Deployment Considerations

Across AZs, inter-AZ network latency adds to produce and fetch latency. In AWS this is typically on the order of 1-2 ms — acceptable for most workloads but potentially SLO-breaking for the strictest ones. If you require very low produce latency, you may confine a partition’s leader and followers to one AZ and use rack awareness purely for fault tolerance, accepting a failover latency penalty. The Multi-AZ Kafka Deployment Strategies guide treats these trade-offs in depth, including minimizing cross-AZ transfer costs.

Operational Best Practices and Day-2 Management

Sizing is not a one-time activity. As workloads evolve, re-evaluate capacity, set a regular review cadence, and automate metric collection.

Monitoring and Alerting on Capacity Headroom

Instrument brokers with Prometheus and Grafana via the JMX Exporter. Exporter metric names mangle the underlying MBeans; the key signals are:

• kafka_server_BrokerTopicMetrics_BytesInPerSec / ..._BytesOutPerSec — throughput trends
• kafka_server_ReplicaManager_UnderReplicatedPartitions — replication health
• kafka_network_RequestChannel_ResponseQueueSize and ..._RequestQueueSize — request/response queue backpressure
• kafka_server_BrokerTopicMetrics_TotalProduceRequestsPerSec / ..._TotalFetchRequestsPerSec — request-rate trends
• kafka_server_KafkaRequestHandlerPool_RequestHandlerAvgIdlePercent — I/O thread headroom

Alert when a resource crosses ~70% of provisioned capacity. For a 10 GbE NIC (~1,250 MB/s), alert when BytesInPerSec + BytesOutPerSec exceeds ~875 MB/s, giving lead time to add capacity before hitting the ceiling. Pair each capacity alert with its failure signature so on-call can act fast: under-replicated partitions climbing means a broker or its disk is falling behind; request-queue size growing while idle-percent drops means the I/O pool is saturated; fetch latency rising with consumer lag means a page-cache working-set overflow, not a network problem.

Scaling Brokers: Vertical vs. Horizontal

At capacity limits you can scale vertically (bigger hosts) or horizontally (more brokers). Vertical scaling is operationally simpler but bounded by the largest feasible single server. Horizontal scaling adds operational work but provides near-linear capacity and better fault tolerance.

To scale out, provision new hosts matching existing specs and software versions, set their broker.rack, and join them to the cluster. Then use kafka-reassign-partitions.sh to migrate partitions onto them. Tools like Cruise Control continuously monitor balance and can generate and execute reassignment plans automatically.

Hardware Refresh and Data Migration

Server hardware typically lasts 3-5 years. Refresh by adding new brokers, migrating partitions off the old ones, then decommissioning. Reassignment moves data online with no downtime, but it is I/O-intensive and must be throttled. Cap replication bandwidth with --throttle (bytes/sec):

kafka-reassign-partitions.sh --bootstrap-server broker1:9092 \
  --reassignment-json-file reassignment.json \
  --throttle 50000000 \
  --execute

This limits replication to 50 MB/s, keeping the migration from saturating network or disk. Monitor progress with --verify, which also removes the throttle once reassignment completes — forgetting to run it leaves the throttle in place and silently caps normal replication, a classic post-migration incident where a later broker failure recovers far slower than expected. Tune the throttle against headroom, not the maximum: a migration that consumes all spare bandwidth will starve live produce/consume traffic, so leave the same ~30% margin you reserved during sizing.

Conclusion

Quantify throughput in bytes per second at the peak, calculate storage with replication counted exactly once, provision memory for page-cache dominance, choose storage media by I/O pattern, and compute broker count three ways (partition balance, throughput, storage) taking the maximum — do these and you build clusters that deliver predictable performance under peak load without overspending. The hardest failures are not capacity miscalculations but silent ones: a stalled log cleaner, a forgotten replication throttle, a working set that just outgrew the page cache. Instrument for those signatures, set a review cadence, and revisit the profile as the platform grows; that discipline is what keeps a Kafka cluster a reliable backbone rather than a source of chronic incidents.