Capacity Planning for Kafka Clusters: Throughput and Storage

Capacity planning is what separates a Kafka deployment that absorbs growth gracefully from one that pages you at 3 a.m. with under-replicated partitions and a full disk. The job is to translate a business requirement—“this new service emits a few million events an hour”—into concrete numbers: how much network each broker must sustain, how much disk the fleet needs, and how many partitions and brokers that implies. Get it wrong and the failure modes are familiar and measurable: saturated NICs, expanding consumer lag, disks crossing their high-watermark, and a cloud bill that grew faster than the workload.

This guide builds a capacity model from first principles for the two resources that dominate a Kafka cluster: network throughput and disk storage. It factors in replication, consumer fan-out, and the protocol overhead that naive estimates ignore, so your Cluster Architecture & Provisioning decisions rest on real arithmetic rather than guesswork.

Deconstructing Kafka Throughput: A Producer-Centric Model

Throughput planning starts with the producer, not the broker. Cluster ingress is a function of the number of producers, the average record size, and the per-second record rate. The common mistake is to size on payload alone and ignore per-record overhead.

That overhead is concrete, not a fudge factor. In the v2 (default) record format, each record adds at most ~21 bytes on top of its key and value: a length varint (up to 5 bytes), one attributes byte, and varint-encoded timestamp-delta and offset-delta fields (deltas, so usually 1–2 bytes each in steady state). Records are wrapped in a record batch whose fixed header is 61 bytes (base offset, batch length, CRC, attributes, producer ID/epoch for idempotence, base sequence). Because the header is amortized across every record in the batch, the effective per-record overhead falls as batch size rises—a 1-record batch pays the full 61 bytes, a 1,000-record batch pays ~0.06 bytes of it. Add header bytes if your records carry application headers (KIP-82). A practical planning allowance is therefore roughly 30–100 bytes of overhead per record for small messages: use the low end for large, well-batched payloads and the high end for small records, headers, or poorly batched idempotent/transactional producers.

The base ingress formula in bytes per second is:

Ingress (bytes/s) = (Avg Record Size + Per-Record Overhead) × Records/s × Producers

That raw ingress is only the application’s view. The load Kafka actually moves over the network is a multiple of it, driven by replication and consumer fan-out. With replication factor 3, every byte produced is written once by the leader and then fetched by two followers—so write-side network traffic across the cluster scales with the replication factor. Independently, each consumer group reading the topic fetches the full byte stream once. A useful cluster-wide aggregate is therefore:

Cluster network (bytes/s) ≈ Ingress × (RF + Consumer Groups)

Treat this as a cluster-wide figure, not per broker: the RF term captures the produce write plus RF−1 follower fetches summed across all brokers, and each consumer group adds one full read pass. To get per-broker load, divide by broker count and account for skew. This is the number you provision broker NICs against. For the hardware that handles these loads, see Broker Sizing and Hardware Selection for Kafka Clusters.

To make it concrete: 10 producers, each emitting 10,000 records/s at an average 512-byte payload, with an 80-byte overhead allowance.

# Raw ingress in MB/s
echo "scale=2; (512 + 80) * 10000 * 10 / 1048576" | bc
# Output: 56.45

With replication factor 3 and 2 independent consumer groups, the cluster-wide multiplier is RF + groups = 3 + 2 = 5:

echo "scale=2; 56.45 * 5" | bc
# Output: 282.25   # MB/s, cluster-wide network traffic

So ~56 MB/s of application data becomes ~280 MB/s of network traffic the cluster must carry. Provision for peak, not average: add a 20–30% headroom margin to absorb partition reassignment, consumer catch-up after an outage, and traffic spikes.

Storage Capacity: Retention, Replication, and Realistic Overheads

Storage is often estimated as ingress × retention, which understates the real number. Start with the replicated raw figure:

Raw Storage = Ingress (MB/s) × Retention (s) × Replication Factor

Then adjust for Kafka internals and operational headroom:

• Index files. Each log segment (default log.segment.bytes = 1 GiB) has an offset index and a time index, each capped at segment.index.bytes (default 10 MiB). In practice indexes add on the order of 1–2% to segment size for default settings—small, but include it.
• Compaction headroom. For cleanup.policy=compact topics, the log cleaner rewrites segments and needs free space to operate; a compacted partition can grow toward roughly 2× its steady-state size during a cleaning cycle.
• Operational headroom. Don’t plan to fill the disk. Keep utilization under ~70% so you have room for traffic spikes, log recovery after an unclean shutdown, and partition reassignment moving data onto the broker. Page cache lives in RAM, not on the data disk, but a near-full disk leaves no slack for any of the above.

For 7-day retention on the 56.45 MB/s stream at RF 3:

# Raw replicated storage in TB (1 TiB = 1024^4 bytes; MB here = 10^6-ish, kept as MiB→TiB chain)
echo "scale=2; 56.45 * 86400 * 7 * 3 / 1024 / 1024" | bc
# Output: 97.68   # TiB

Applying a 1.2× factor for indexes plus operational slack and a 0.70 maximum fill ratio:

echo "scale=2; 97.68 * 1.2 / 0.7" | bc
# Output: 167.45   # TiB usable required

That ~167 TiB is spread across the broker fleet. At 10 TiB usable per broker you need at least 17 brokers for storage alone (ceil(167.45 / 10) = 17), independent of throughput. Throughput and storage must be sized together, because either can dominate the broker count. For high-volume workloads, Capacity Planning for 1 Million Messages per Second covers additional scaling patterns.

Partition Count and Its Impact on Capacity

Partition count is the variable that quietly undermines both throughput and storage plans. Each partition is a unit of parallelism, but it also costs file handles, memory, and controller/replication work, and it lengthens recovery and leader-election time after a failure. Over-partitioning “to be safe” is a real anti-pattern: it inflates metadata and slows failover. The KRaft vs ZooKeeper: Choosing the Right Metadata Management decision matters here—KRaft removes the ZooKeeper metadata bottleneck and supports far more partitions per cluster.

Partition count must satisfy two independent constraints: target throughput per partition, and consumer parallelism. A single partition can sustain on the order of tens of MB/s on good hardware, but the realistic per-partition target is usually set lower by consumer processing speed. The throughput-driven minimum is:

Min Partitions = ceil(Total Ingress / Target Throughput per Partition)

For 56.45 MB/s at a conservative 20 MB/s per partition:

# ceil(56.45 / 20). bc truncates, so add (divisor-1) before integer division.
echo "(56 + 20 - 1) / 20" | bc
# Output: 3   # partitions (ceiling)

Consumer parallelism often raises this floor: a consumer group can have at most one active consumer per partition, so to run 5 consumer instances in a group you need at least 5 partitions, or some instances sit idle. Note that partition count is effectively a one-way door—kafka-topics.sh --alter --partitions only increases it, and adding partitions changes key-to-partition routing, so plan the ceiling up front rather than reacting to lag.

For operational limits, widely used guidance keeps partitions to roughly 4,000 per broker as a soft ceiling. ZooKeeper-backed clusters were additionally bounded around 200,000 partitions cluster-wide; KRaft removes that cluster-wide bound and scales to millions. These are guidelines—validate against your hardware and recovery-time objectives. To see how partitions are actually distributed (and how much disk each consumes) across brokers, use kafka-log-dirs.sh, not kafka-topics.sh:

# Per-broker partition placement and on-disk size (JSON output)
kafka-log-dirs.sh --bootstrap-server localhost:9092 --describe

To check replication health (under-replicated partitions are an early warning of an overloaded or failing broker):

kafka-topics.sh --bootstrap-server localhost:9092 --describe --under-replicated-partitions

Failure Modes: What Saturation Looks Like Before It Pages You

A capacity model is only useful if you can spot the moment reality outruns it. Each resource ceiling has a distinct, observable signature—learn the leading indicator so you scale on a trend, not an incident.

The two replica-manager gauges are the highest-signal pair. A persistent nonzero UnderReplicatedPartitions means a broker can’t pull data fast enough to keep followers in sync—usually network or disk saturation, sometimes a slow disk on one node. UnderMinIsrPartitionCount is the same problem after it has become a correctness issue: once in-sync replicas drop below min.insync.replicas, every acks=all produce to that partition is rejected. Replicas added during a reassignment are deliberately excluded from these counts, so a brief blip during a planned rebalance is expected; a steady climb is not. Treat both as page-on-nonzero alerts, and trace RequestHandlerAvgIdlePercent alongside them—an idle ratio approaching zero confirms the broker is CPU/IO-bound rather than network-bound, which changes the remediation from “add NICs” to “add brokers.”

Modeling Consumer Fan-Out and Multi-Topic Aggregation

Real clusters host many topics, each with its own replication factor, retention, and set of consumer groups. The aggregate plan sums leader and follower traffic across every topic. Consider three topics:

Per topic, the cluster-wide network multiplier is RF + consumer groups:

echo "scale=2; (30 * (3+2)) + (45 * (3+3)) + (15 * (2+1))" | bc
# Output: 465.00   # MB/s, cluster-wide

Distribute 465 MB/s across the fleet. A 10 GbE NIC is 1,250 MB/s; at a 70% utilization target each broker can carry ~875 MB/s, so this workload is comfortably network-bound to a single broker’s worth of bandwidth—storage and partition counts will drive the actual broker count higher. Encoding these constraints as code keeps the math honest as topics multiply; Automating Kafka Cluster Provisioning with Terraform and Confluent for Kubernetes shows how.

Monitoring and Validating Your Capacity Model

A capacity plan is a hypothesis; production metrics are the test. Divergence between modeled and observed values exposes wrong assumptions about record-size distribution, fan-out, or overhead. Track at minimum:

• Broker bytes in/out per second (kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec / BytesOutPerSec): validates your throughput and multiplier assumptions. ReplicationBytesInPerSec/ReplicationBytesOutPerSec isolate follower-fetch traffic, so you can confirm the RF term empirically.
• Disk usage and growth rate: confirms storage projections and catches retention misconfiguration before the disk fills.
• Partition count and leadership skew per broker: uneven leadership creates hotspots even when totals look fine.
• Consumer lag (records-lag-max on the consumer, or per-group via kafka-consumer-groups.sh --describe): tells you whether consumers keep up with producers.

See Key Metrics to Monitor for Kafka Cluster Health for the full framework. For a quick on-host pull of the broker byte-rate MBeans over JMX, use the bundled JmxTool (the class moved to org.apache.kafka.tools.JmxTool in Kafka 3.5; the old kafka.tools.JmxTool still works with a deprecation warning):

# Stream broker bytes-in/out per second from JMX
kafka-run-class.sh org.apache.kafka.tools.JmxTool \
  --object-name 'kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec' \
  --object-name 'kafka.server:type=BrokerTopicMetrics,name=BytesOutPerSec' \
  --jmx-url service:jmx:rmi:///jndi/rmi://localhost:9999/jmxrmi \
  --reporting-interval 1000

For storage validation, compare on-disk size against your model—either per partition via kafka-log-dirs.sh --describe, or quickly per host:

du -sh /var/lib/kafka/data/* | sort -rh | head -20

When numbers drift, revisit your overhead and batching assumptions; both per-record overhead and effective on-disk size depend heavily on batch size and compression. Enabling broker- or producer-side compression (e.g. compression.type=zstd) cuts network and disk at the cost of CPU—fold that trade-off back into broker sizing. See the broker config reference for compression.type and the Kafka hardware and OS guidance for disk, filesystem, and NIC recommendations.

Operationalizing Capacity Planning: From Spreadsheet to Code

Spreadsheets stop scaling once you have dozens of topics and several environments. A small script that encodes the equations gives you consistent, repeatable estimates you can drop into a provisioning pipeline:

#!/usr/bin/env python3
"""Kafka capacity planner for throughput and storage."""

import argparse
import math

MiB = 1024 * 1024

def ingress_mbps(msg_size, overhead, msg_rate, producers):
    """Application ingress in MB/s (MiB/s)."""
    return (msg_size + overhead) * msg_rate * producers / MiB

def cluster_network_mbps(ingress, rf, consumer_groups):
    """Cluster-wide network load in MB/s: write (RF) + read (groups)."""
    return ingress * (rf + consumer_groups)

def storage_tib(ingress, retention_hours, rf, overhead=1.2, max_fill=0.7):
    """Usable storage required in TiB.

    ingress is MiB/s; * seconds -> MiB; / 1024^2 -> TiB.
    """
    raw_tib = ingress * retention_hours * 3600 * rf / (1024 * 1024)
    return raw_tib * overhead / max_fill

def main():
    p = argparse.ArgumentParser(description="Kafka Capacity Planner")
    p.add_argument("--msg-size", type=int, required=True, help="Avg record size (bytes)")
    p.add_argument("--overhead", type=int, default=80, help="Per-record overhead (bytes)")
    p.add_argument("--msg-rate", type=int, required=True, help="Records/s per producer")
    p.add_argument("--producers", type=int, required=True, help="Number of producers")
    p.add_argument("--rf", type=int, required=True, help="Replication factor")
    p.add_argument("--consumers", type=int, required=True, help="Independent consumer groups")
    p.add_argument("--retention", type=int, required=True, help="Retention (hours)")
    p.add_argument("--nic-mbps", type=float, default=1250.0, help="Per-broker NIC MB/s")
    p.add_argument("--nic-util", type=float, default=0.7, help="Target NIC utilization")
    p.add_argument("--disk-tib", type=float, default=10.0, help="Usable disk per broker (TiB)")
    a = p.parse_args()

    ing = ingress_mbps(a.msg_size, a.overhead, a.msg_rate, a.producers)
    net = cluster_network_mbps(ing, a.rf, a.consumers)
    sto = storage_tib(ing, a.retention, a.rf)

    print(f"Ingress:            {ing:.2f} MB/s")
    print(f"Cluster network:    {net:.2f} MB/s")
    print(f"Usable storage:     {sto:.2f} TiB")
    print(f"Brokers (network):  {math.ceil(net / (a.nic_mbps * a.nic_util))}")
    print(f"Brokers (storage):  {math.ceil(sto / a.disk_tib)}")

if __name__ == "__main__":
    main()

python3 kafka_capacity_planner.py \
  --msg-size 512 --msg-rate 10000 --producers 10 \
  --rf 3 --consumers 2 --retention 168

The script reports Brokers (network) and Brokers (storage) separately on purpose: provision for whichever is larger, then re-check partition count against both. Wired into your observability stack, the same model can ingest real BytesInPerSec and disk-growth metrics, flag capacity drift, and feed scaling decisions. The Confluent running-in-production guide covers complementary deployment and sizing considerations.

Capacity planning is a recurring discipline, not a one-off. Re-run these models against current production metrics on a regular cadence and scale the fleet before the headroom runs out—UnderReplicatedPartitions going nonzero means you are already late. Model, measure, reconcile: that loop is what keeps a Kafka platform reliable, predictable, and cost-efficient.