How to Choose the Right Number of Partitions for a Topic

Partition count sets a topic’s throughput ceiling, caps consumer parallelism, shapes latency, and drives the operational cost of the cluster. It is also the one number you can’t cleanly walk back: you can raise it on a live topic, but doing so breaks key-to-partition mappings and never rebalances existing data. This guide gives a doc-grounded, operationally focused method for getting it right before the topic exists.

For the underlying model of how partitions map to storage and network I/O, see Topics, Partitions & Data Modeling. The partition is Kafka’s fundamental unit of parallelism, and the choices here ripple across every layer of a streaming architecture.

Understanding the Throughput and Parallelism Contract

A topic partition is a strictly ordered, append-only sequence of records that enforces a hard concurrency boundary. Within a single partition, writes are appended in order, and within one consumer group the partition is read by exactly one consumer at a time. The maximum throughput a single partition can sustain is therefore bounded by the slowest link in the chain: network bandwidth to the leader broker, disk write capacity on that broker, and the rate at which one consumer thread can process records and commit offsets.

Increasing the partition count multiplies the number of parallel write paths producers can use and the number of consumer instances that can actively process records. That parallelism is not free. Each partition adds metadata the controller must track, consumes file descriptors and page cache on every broker holding a replica, and lengthens controlled shutdowns and leader elections. The goal is to provision enough partitions to meet throughput and latency targets without crossing the point where per-partition overhead degrades cluster stability.

Start by sizing from target throughput. If you expect a sustained produce rate of 100 MB/s and empirical testing shows a single partition on your hardware comfortably handles 20 MB/s of writes, you need at least five partitions to absorb the write load. But the consumer side usually binds tighter. If a single consumer instance processes only 10 MB/s and you must keep up with 100 MB/s, you need at least ten consumers — and therefore at least ten partitions — to parallelize the work. The larger of the two numbers is your floor.

To establish that single-partition baseline, run a controlled experiment with kafka-producer-perf-test.sh and kafka-consumer-perf-test.sh on a representative broker, using your actual payload sizes and compression settings. The following measures write throughput with 1 KB records. Note that every producer property, including bootstrap.servers, is passed as a space-separated key=value token to --producer-props:

kafka-producer-perf-test.sh \
  --topic throughput-benchmark \
  --num-records 5000000 \
  --record-size 1024 \
  --throughput -1 \
  --producer-props bootstrap.servers=broker1:9092,broker2:9092 \
                   acks=1 \
                   compression.type=lz4

--throughput -1 disables client-side rate limiting so you measure the true ceiling. Run this against a one-partition topic at replication factor 3, then repeat with increasing partition counts on the same broker set to see where throughput plateaus. Two things matter as much as the peak number: measure with acks=all if that is your production setting, since waiting on the full ISR lowers per-partition throughput, and measure at your real record size, because batching efficiency and compression ratios shift sharply between 200-byte and 10 KB records. Document the per-partition ceiling under production-equivalent settings; that number anchors every subsequent sizing decision.

The Consumer Parallelism Ceiling and Group Dynamics

The relationship between partitions and consumers is frequently misunderstood. Within a consumer group, Kafka assigns each partition to exactly one consumer instance. If you have more consumers than partitions, the excess consumers sit idle. If you have more partitions than consumers, some consumers carry multiple partitions, and their per-instance processing capacity becomes the bottleneck. The partition count is a hard cap on the parallelism of a single consumer group.

This forces you to reason about the maximum number of consumer instances you will ever deploy. That number is bounded by your deployment topology: a Kubernetes Deployment is limited by node resources and the maxReplicas of its Horizontal Pod Autoscaler; a bare-metal service is limited by the hosts or VMs dedicated to it. Set the partition count at least as large as that maximum, and ideally larger so you have headroom during rebalances and rolling restarts.

Consider a clickstream service. During normal traffic it runs 12 consumers, each handling ~8 MB/s. During a flash sale, traffic triples and the HPA scales to 36 pods. If the topic has only 24 partitions, 12 pods sit idle while the active pods overload and lag spikes. Provisioning 48 partitions covers the 36-pod peak with a 1.33x buffer and leaves room for rolling restarts without starving the group.

Partition count also interacts with max.poll.records and max.poll.interval.ms. A consumer must return from poll() and call poll() again within max.poll.interval.ms (default 300000 ms), or the group coordinator considers it dead and triggers a rebalance. More partitions per consumer means each poll() can return records spanning multiple partitions, increasing batch size and the risk of exceeding the interval. Tune these in lockstep with assignment. A conservative starting point:

# Consumer configuration for high-partition-count topics
max.poll.records=500
max.poll.interval.ms=300000
session.timeout.ms=30000
heartbeat.interval.ms=10000
partition.assignment.strategy=org.apache.kafka.clients.consumer.CooperativeStickyAssignor

CooperativeStickyAssignor (available since Kafka 2.4, via KIP-429) performs incremental cooperative rebalances: it revokes only the partitions that actually need to move instead of stopping the whole group, which sharply reduces the “stop-the-world” pause when consumers join, leave, or the partition count changes. That pause is precisely what makes high-partition-count groups painful to operate, so on any topic where you expect frequent scaling or rolling restarts, treat this assignor as the default rather than an optimization.

Operational Overhead and Cluster Stability Boundaries

Every partition consumes resources on the controller and on each broker hosting a replica. In KRaft mode — the only mode supported since Apache Kafka 4.0, after ZooKeeper was removed — the active controller maintains the cluster metadata log, including leader, ISR, and leader-epoch state for every partition. KRaft raised the practical partition ceiling dramatically: ZooKeeper-based clusters topped out near ~200,000 partitions, whereas KRaft clusters have been demonstrated into the millions, with markedly faster controlled shutdown and recovery because metadata no longer has to be shuttled out of an external store. Even so, partition count is not free at the controller, and very large counts still lengthen failover and metadata propagation.

Brokers pay a per-partition tax in file descriptors. For each log segment, a partition replica opens the segment’s .log, .index, and .timeindex files — roughly three descriptors per active segment, and more while older segments are still open for serving reads, during log cleaning, and during segment rolling. A broker hosting 5,000 partition replicas can easily run into the tens of thousands of open descriptors before you count network sockets. Linux defaults to a per-process limit of 1,024, which is far too low; the official guidance is to allow at least 100,000. Raise it with a systemd override:

[Service]
LimitNOFILE=100000
LimitNPROC=32768

Apply with systemctl daemon-reload and restart the broker. Watch actual usage via the count of entries in /proc/<pid>/fd to confirm headroom. The failure mode if you skip this is abrupt and easy to misdiagnose: the broker logs Too many open files, stops accepting new connections and rolling new segments, and replicas can drop out of the ISR — symptoms that look like a network or disk problem until you check the descriptor count.

Memory pressure is subtler. Each segment’s index and timeindex are memory-mapped, and active segments are served from the OS page cache. Kafka relies on the page cache rather than a large JVM heap for read performance, so when the working set of active segments exceeds physical RAM, reads fall back to disk and latency spikes. A broker with thousands of partitions and a 1 GB default segment size can reference far more data than fits in 64 GB of RAM; whether that hurts depends on how much of it is hot. Treat per-broker partition limits as workload-dependent: a common rule of thumb is to stay under a few thousand partitions per broker on spinning disks and allow more on NVMe with ample RAM, but validate by measuring CPU steal, iowait, and network saturation at peak rather than trusting a fixed number.

Partition count also drives controlled-shutdown time. On a graceful shutdown, a broker migrates leadership of its partitions to other brokers; the broker config keys controlled.shutdown.max.retries (default 3) and controlled.shutdown.retry.backoff.ms (default 5000) govern retry behavior when that migration partially fails. A broker leading thousands of partitions can take a meaningful amount of time to drain, which complicates rolling restarts and OS patching. If a runbook allows a 30-minute maintenance window, confirm that the total drain-plus-startup time across all brokers fits inside it. That requirement often imposes a tighter per-broker partition cap than the raw throughput limit would.

These constraints push in different directions, so it helps to see which one actually binds before settling on a number:

The first three set a floor; the last three set a per-broker ceiling. A defensible partition count is one that clears the floor without crossing any ceiling — if it can’t, the answer is more brokers, not more partitions per broker.

Keying Strategies and Data Distribution

Partition count is inseparable from your keying strategy. For keyed records, the default partitioner routes by hash(key) % numPartitions. Change the partition count and the mapping changes, which breaks per-key ordering and any stateful processing that assumes a key stays on one partition. This is why Partitioning Strategies for High-Throughput Topics must be evaluated alongside sizing.

When you key by a business entity such as user_id, all events for that user land on one partition and stay ordered. Increase the count from 12 to 24 and a given user_id may now hash to a different partition; a consumer that relied on per-user ordering will observe records out of order across the change boundary. This is a common, real source of corruption in stateful stream processors whose local state stores are keyed by entity.

The cleanest mitigation is to over-provision from the start. If you expect a topic to need 24 partitions eventually, create it with 48 and accept the modest overhead. You can then scale consumer parallelism without ever changing the count. The trade-off is paying that overhead from day one even at low throughput — usually worth it, because it avoids the risky repartitioning of a live topic.

A more advanced option decouples logical sharding from physical partitions: compute a stable shard id in the producer (for example, a consistent hash of user_id into a fixed, large shard space) and use a custom partitioner to map shards onto the current physical partitions. Because the logical shard of a key never changes, you can grow physical partitions while preserving the shard-to-key relationship your application depends on. This requires custom producer logic and disciplined operations, but it is the most flexible approach for long-lived topics. The Kafka community explored an in-protocol version of this idea in KIP-253, which is useful background even though it was not delivered as a turnkey feature.

Estimating Partition Count from Producer and Consumer Throughput

With the foundations in place, here is a concrete sizing procedure. It assumes you have already benchmarked single-partition throughput for your hardware and payload profile.

Step 1 — Target produce throughput. Use load-test or production projections, in MB/s or records/s. For bursty traffic, size to the 95th-percentile rate over a representative window, not the average, because brief bursts that exceed the per-partition ceiling create leader hotspots and lag.

Step 2 — Target consume throughput. This is often the binding constraint. If consumers do CPU-heavy work or synchronous downstream calls, per-instance throughput can be far below what the broker can serve. Profile one consumer under realistic load, then divide total target throughput by that per-instance rate to get the minimum consumer count, and hence the minimum partition count.

Step 3 — Partition floor. Take the larger of the producer-required and consumer-required counts. If producers need 5 and consumers need 10, the floor is 10.

Step 4 — Growth multiplier. Throughput rarely stays flat. Apply 1.5x–3x based on growth projections and how painful repartitioning would be. Use 3x for a central, hard-to-migrate topic; 1.5x for a short-retention, loosely coupled one.

Step 5 — Validate against operational limits. Confirm the result does not push any broker past the per-broker partition cap you established. If it does, either accept a lower ceiling or add brokers to spread replicas more thinly.

Step 6 — Validate against max consumer-group size. Ensure the count is at least the maximum number of consumers you will ever run — for Kubernetes, your HPA maxReplicas.

The following script automates the arithmetic:

#!/usr/bin/env bash
# partition-sizer.sh - Calculate minimum partition count for a topic
set -euo pipefail

PRODUCER_TARGET_MBPS=${1:-100}
CONSUMER_TARGET_MBPS=${2:-100}
PER_PARTITION_PRODUCE_MBPS=${3:-20}
PER_PARTITION_CONSUME_MBPS=${4:-10}
GROWTH_MULTIPLIER=${5:-2}

# Round up to the next whole partition (ceiling division).
producer_partitions=$(echo "($PRODUCER_TARGET_MBPS + $PER_PARTITION_PRODUCE_MBPS - 1) / $PER_PARTITION_PRODUCE_MBPS" | bc)
consumer_partitions=$(echo "($CONSUMER_TARGET_MBPS + $PER_PARTITION_CONSUME_MBPS - 1) / $PER_PARTITION_CONSUME_MBPS" | bc)

if [ "$producer_partitions" -gt "$consumer_partitions" ]; then
  floor=$producer_partitions
else
  floor=$consumer_partitions
fi
recommended=$(echo "$floor * $GROWTH_MULTIPLIER" | bc)

echo "Producer-required partitions: $producer_partitions"
echo "Consumer-required partitions: $consumer_partitions"
echo "Partition floor: $floor"
echo "Recommended partitions (${GROWTH_MULTIPLIER}x growth): $recommended"

Run it with your measured values to get a defensible starting point for a team discussion — not a substitute for the benchmark in step 1.

Repartitioning: When and How to Change Partition Count

Eventually you may need to add partitions to an existing topic. You can, with kafka-topics.sh --alter --partitions <new-count>, but understand exactly what it does and does not do. It increases the partition count online, but it does not rebalance existing data: records already written stay on their original partitions, and only new records are spread across the larger set. The result is skew — old partitions stay full while new ones start empty — and, for keyed topics, a break in key-to-partition mapping that disrupts ordering as described above.

kafka-topics.sh --bootstrap-server broker1:9092 \
  --alter --topic my-topic --partitions 48

For topics where ordering does not matter and retention is finite, the skew is often acceptable: old data ages out and load eventually evens. For topics with long retention or strict ordering, you need a full repartition. The standard, safe approach is to create a new topic with the target partition count, have producers dual-write (or use MirrorMaker 2 to copy historical data), let the new topic catch up, then cut consumers over to it. This is multi-step and demands careful coordination, but it preserves correctness.

Be clear about what the reassignment tool is for. kafka-reassign-partitions.sh moves existing partition replicas between brokers (and can change replication factor); it cannot add partitions and is not an alternative to --alter. Reach for it after a partition increase to rebalance replica placement across brokers, or to drain a broker — not to grow a topic’s partition count.

The takeaway: growing a live topic’s partition count is expensive and risky, and the time to size correctly is before the topic is created. That is exactly why the growth multiplier matters — it buys time to plan a proper migration instead of scrambling under fire.

Monitoring and Iterating on Partition Count

Partition sizing is not one-and-done. As traffic patterns shift, your initial assumptions drift, so instrument the two signals that tell you a topic is nearing its partition-imposed limits: per-partition produce throughput and consumer lag.

For produce throughput, the broker JMX MBean kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec gives the aggregate rate, and adding the topic=<topic> key — kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec,topic=<topic> — gives per-topic bytes in. Divide that by the partition count to estimate the average per-partition rate (a useful proxy; real load is rarely perfectly even). When any partition consistently exceeds ~70% of your benchmarked per-partition ceiling, plan an increase.

For consumer lag, the client-side MBean kafka.consumer:type=consumer-fetch-manager-metrics,client-id=<client> exposes the records-lag-max attribute: the maximum record lag across the partitions assigned to that consumer. Read it together with the per-partition view, because the shape of the lag tells you the fix. Lag that climbs monotonically across all consumers while they stay healthy means the per-partition consume rate is genuinely insufficient — add partitions and consumers. Lag concentrated on one partition while the rest keep up means key skew, not a partition shortfall — adding partitions won’t help, and you need to revisit the keying strategy instead.

Alert on these via a JMX exporter. With the Prometheus JMX exporter’s default naming (dots and MBean attributes lowercased and joined with underscores), BytesInPerSec surfaces as kafka_server_brokertopicmetrics_bytesinpersec with a topic label. The rule below fires when estimated per-partition throughput stays above 15 MB/s against a 20 MB/s ceiling. kafka_topic_partitions is not a built-in Kafka metric — you must supply the per-topic partition count yourself (for example via a custom exporter or a recording rule), so verify this metric exists in your stack before relying on the expression:

groups:
  - name: kafka_partition_throughput
    rules:
      - alert: HighPerPartitionThroughput
        expr: |
          sum by (topic) (
            rate(kafka_server_brokertopicmetrics_bytesinpersec[5m])
          ) / on(topic) group_left kafka_topic_partitions > 15 * 1024 * 1024
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "Topic {{ $labels.topic }} approaching per-partition throughput limit"
          description: "Estimated per-partition throughput is {{ $value | humanize }}B/s. Consider increasing partitions."

Beyond automated alerts, run a periodic capacity review for every topic above a meaningful throughput threshold (say, 10 MB/s). Compare actual growth against the projections you sized with; if growth is outpacing your multiplier, start a repartitioning project before the topic hits its ceiling rather than after the 2 a.m. page.

Partition sizing is a balance: enough partitions to meet throughput and consumer-parallelism targets, not so many that controller failover, file descriptors, page cache, and shutdown time become liabilities. Benchmark single-partition throughput under production-equivalent settings, take the larger of the producer and consumer floors, apply a growth multiplier, validate against per-broker and consumer-group limits, and monitor BytesInPerSec and records-lag-max so you know the moment your assumptions stop holding.