Partitioning Strategies for High-Throughput Topics

Within Topics, Partitions & Data Modeling, the partition is Kafka’s unit of parallelism. For platform and SRE engineers running production clusters, the partitioning strategy determines a topic’s throughput ceiling, latency profile, and operational stability. Get it wrong and you get data skew, consumer lag concentrated on a few instances, and rebalances that stall stream-processing pipelines.

This article covers the mechanics of the producer partitioner as it actually behaves in current Kafka (3.3+), the trade-offs between key-based and uniform distribution, a quantitative method for sizing partition counts, and the operational procedures for changing partition layout safely. It pairs with Keying Strategies for Ordered Message Processing.

How Partitions Dictate Throughput

A Kafka topic is a partitioned, append-only log. Each partition is a single sequential log; you cannot parallelize writes within one partition, only across partitions. A partition’s write throughput is bounded by the leader broker’s sequential write speed, its network bandwidth, and the replication overhead of the topic’s acks and replication settings.

The practical write ceiling for a topic is roughly P * T, where P is the partition count and T is the per-partition throughput your hardware sustains. T is not a constant — you must benchmark it. A partition carrying 1 KB records with acks=all and replication factor 3 has a very different profile from one carrying 1 MB records with acks=1. Producer batching, compression codec, and the leader’s disk I/O all interact.

As partitions per broker climb, the cost of leader election, ISR tracking, and open file handles grows. Each partition replica keeps an active segment with an associated .index and .timeindex file. At thousands of partitions per broker, the random I/O from many concurrent active segments erodes the sequential-write advantage Kafka depends on. The How to Choose the Right Number of Partitions for a Topic guide expands on balancing parallelism against broker saturation.

Benchmarking Single-Partition Throughput

Before sizing anything, measure the throughput ceiling of a single partition on your infrastructure, using your production record size, compression, and acks. Use kafka-producer-perf-test against a one-partition topic with your production replication factor:

# Single partition, 1 KB records, acks=all, lz4 compression
kafka-producer-perf-test \
  --topic benchmark-topic \
  --num-records 5000000 \
  --record-size 1024 \
  --throughput -1 \
  --producer-props bootstrap.servers=broker1:9092,broker2:9092 \
    acks=all compression.type=lz4 batch.size=65536 linger.ms=5 \
    max.in.flight.requests.per.connection=5

--producer-props takes inline key=value producer settings; if you keep production settings in a file, --producer.config client.properties loads them and you can still override individual keys with --producer-props. Watch the broker disk during the run with iostat -x 1 for write MB/s and average queue depth. Record the stable-state result (records/s and MB/s) as T_partition, then repeat with your largest expected record to find the bandwidth-limited ceiling. That number is the denominator in the partition-count calculation below.

The Producer Partitioner: Behavior and Pitfalls

The producer’s partitioner maps each record to a partition. Behavior changed in Kafka 3.3 with KIP-794, so the version you run matters.

For a record with a non-null key, the built-in partitioner hashes the key with murmur2 and maps it to a partition: partition = (murmur2(keyBytes) & 0x7fffffff) % numPartitions. The mask clears the sign bit so the result is non-negative. This mapping is deterministic, so every record for a given key lands on the same partition (as long as the partition count is unchanged), preserving per-key order.

For a record with a null key, behavior depends on version:

• Before 3.3, the default (DefaultPartitioner, and the explicit UniformStickyPartitioner) “stuck” to one partition until the current batch filled or linger.ms elapsed, then moved on. This batched well, but it is not actually uniform: because a slower broker drains batches more slowly, the sticky window lasts longer for it, so it receives more records — the “runaway” skew KIP-794 was written to fix.
• Since 3.3, the built-in partitioner is the strictly-uniform sticky partitioner. It still batches by partition, but allocates roughly batch.size bytes to a partition before switching and weights selection by queue size so slower brokers are not overloaded. DefaultPartitioner and UniformStickyPartitioner are deprecated; to get the new behavior, leave partitioner.class unset. To make even keyed records distribute uniformly, set partitioner.ignore.keys=true.

The pitfall for keyed high-throughput topics is skew. Key-based partitioning is only as uniform as your key distribution. A clickstream topic keyed by user_id with a power-law user base sends a disproportionate share of traffic to the partitions owning the heavy users. Those partitions show elevated produce latency, and the consumer instances assigned to them lag while others idle — exactly the parallelism you provisioned for, lost.

Diagnosing Partition Skew

Kafka does not expose a per-partition byte-rate metric over JMX; BrokerTopicMetrics,name=BytesInPerSec is scoped per topic (or all-topics), not per partition. To measure skew you work at the partition level with two real sources.

On-disk size per partition replica, via kafka-log-dirs:

# Per-partition on-disk sizes for a topic
kafka-log-dirs \
  --bootstrap-server broker1:9092 \
  --describe \
  --topic-list high-throughput-events \
  --broker-list 0,1,2

This returns JSON with a partitions array of { "partition": "topic-N", "size": <bytes>, "offsetLag": ..., "isFuture": ... } per log directory. Diff size across partitions to spot storage skew (pipe it through jq to tabulate).

For write-rate skew, compare partition log-end offsets over a fixed interval using kafka-get-offsets (or kafka-run-class kafka.tools.GetOffsetShell) and divide the offset delta by elapsed time:

kafka-get-offsets \
  --bootstrap-server broker1:9092 \
  --topic high-throughput-events \
  --time -1

Compute the coefficient of variation (standard deviation / mean) of the per-partition rates. As a rule of thumb, a healthy topic stays under ~0.2; a value above ~0.5 means the partitioning strategy needs intervention.

Uniform Distribution When Ordering Is Not Required

When per-key ordering is not needed — or ordering is reconstructed downstream by sequence number or event time — uniform distribution maximizes balance. On 3.3+, the simplest path is the built-in partitioner with partitioner.ignore.keys=true, which spreads records (even keyed ones) across all partitions while keeping efficient batching. If you need strict, deterministic round-robin regardless of batching, the explicit RoundRobinPartitioner distributes records equally across partitions “regardless of record key hash” (per its javadoc).

// Java producer: uniform distribution on Kafka 3.3+
Properties props = new Properties();
props.put("bootstrap.servers", "broker1:9092,broker2:9092");
props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");

// Option A (recommended): keep the built-in partitioner, ignore keys.
props.put("partitioner.ignore.keys", "true");
// Option B: strict round-robin regardless of key.
// props.put("partitioner.class", "org.apache.kafka.clients.producer.RoundRobinPartitioner");

props.put("batch.size", "131072");      // 128 KB
props.put("linger.ms", "5");
props.put("compression.type", "zstd");

KafkaProducer<String, String> producer = new KafkaProducer<>(props);
ProducerRecord<String, String> record =
    new ProducerRecord<>("high-throughput-topic", key, eventPayload);
producer.send(record, callback);

The trade-off is that you give up per-key ordering. If any downstream consumer relies on per-key order for stateful processing, uniform distribution is off the table. For stateless ingestion, telemetry, or independent events it is the simplest route to balanced load. It also makes partition expansion painless: with no key-to-partition affinity, adding partitions immediately increases parallelism without remapping existing keys.

Semantic Partitioning with Composite Keys

When you need ordering within an entity but a natural key is skewed, semantic partitioning with a composite key preserves order inside an entity boundary while spreading load. Instead of keying solely on a skew-prone natural key, combine the entity identifier with a sharding suffix.

Take an order pipeline that must preserve order per order_id but co-locate everything for a customer_id so customer state stays consistent. Keying on customer_id alone routes a hot customer entirely to one partition. A composite key splits a hot customer across a bounded set of partitions while keeping all of a single order on one partition:

// Composite key: route by customer, shard hot customers by order
public String buildCompositeKey(String customerId, String orderId, int shardCount) {
    int shard = Math.floorMod(orderId.hashCode(), shardCount);
    return customerId + "_" + shard;
}

// Give known-hot customers more shards
public String buildCompositeKeyForHotCustomer(String customerId, String orderId) {
    return buildCompositeKey(customerId, orderId, 16);
}

Math.floorMod is used rather than Math.abs(hashCode()) % n because Math.abs(Integer.MIN_VALUE) is still negative — a real edge case that otherwise throws or produces a negative index. Note the explicit cost: events for the same customer are now spread across up to shardCount partitions, so there is no longer a total order per customer, only per (customer, shard). That is the price of breaking up a hot key, and it must be acceptable to your consumers.

This pattern needs operational awareness of key distribution. Track hot entities and adjust shard counts deliberately. A common approach keeps an entity→shard-multiplier map in a compacted topic or an external store; raising a customer’s multiplier spreads new traffic wider. Because that changes which shard a given order maps to, drain or fence in-flight work for that entity before the change so you do not interleave two orderings.

Reconstructing Order on the Consumer

If you sharded an entity across partitions, downstream you must either assign all of that entity’s partitions to one logical processor or merge the shard streams in an event-time window. Consumers extract the base entity from the composite key:

// Composite key format: customerId_shardNumber
public String extractEntityKey(String compositeKey) {
    int lastUnderscore = compositeKey.lastIndexOf('_');
    return lastUnderscore > 0 ? compositeKey.substring(0, lastUnderscore) : compositeKey;
}

Reach for this only after you have measured that natural-key partitioning produces unacceptable skew. For most topics a well-chosen, uniformly distributed natural key is sufficient and far cheaper to operate.

Partitioning, Idempotence, and Transactions

High-throughput topics often back exactly-once pipelines, so it is worth being precise about how partitioning interacts with the idempotent and transactional producer — because there is a common myth here.

The idempotent producer maintains a sequence number per (producer ID, topic-partition). The broker accepts a batch only if its sequence is exactly one greater than the last it persisted for that PID and partition (see KIP-98). Sequences are scoped to each partition independently.

The myth: that increasing a topic’s partition count breaks transactional state, invalidates sequence numbers, or triggers ProducerFencedException. It does not. Existing partitions keep their sequence continuity; newly added partitions simply start their own sequence at zero. ProducerFencedException comes from a newer producer instance claiming the same transactional.id (epoch fencing), not from changing partition counts. Adding partitions during an open transaction does not corrupt that transaction either.

The real partitioning gotcha for EOS is the same one that affects any keyed topic: when you add partitions, murmur2(key) % numPartitions changes for most keys, so a key that lived on partition 7 may now route to partition 19. That breaks per-key ordering across the change and can place two records for one key on different partitions consumed out of order. For keyed, order-sensitive topics, treat partition count as effectively fixed and plan changes as a migration (drain, switch, or re-key into a new topic) rather than a live --alter. For uniform-distribution topics there is no key affinity, so expansion is safe at any time.

Create such a topic with the partition count you intend to keep:

kafka-topics --bootstrap-server broker1:9092 \
  --create \
  --topic financial-ledger \
  --partitions 50 \
  --replication-factor 3 \
  --config min.insync.replicas=2 \
  --config cleanup.policy=compact \
  --config retention.ms=-1

Retention vs Compaction: When to Use Each is relevant here because compacted topics holding transactional or keyed state interact with transaction.timeout.ms and the log-cleaner in ways that need care.

Partition Count Sizing: A Quantitative Framework

Sizing a high-throughput topic balances throughput, consumer parallelism, broker limits, and growth. It is not a one-time calculation; revisit it as traffic shifts.

Step 1 — Throughput target. Take your peak observed throughput (records/s and MB/s) and add headroom. A 2x margin over peak is a floor; 3–4x is prudent for critical paths.

T_target = Peak observed throughput * Headroom multiplier

Step 2 — Single-partition capacity. Use the benchmark above to get T_partition under production record size and replication. Do not rely on rules of thumb; storage subsystems vary too much.

Step 3 — Throughput-driven floor.

P_throughput = ceil(T_target / T_partition)

If T_target is 500 MB/s and T_partition is 25 MB/s, you need at least 20 partitions.

Step 4 — Consumer parallelism. Partition count caps consumer-group parallelism: a group can have at most one consumer per partition. For C consumer instances you need at least C partitions; commonly P >= 2C so you have rebalance headroom and room to scale out.

P_consumer = C * ConsumerHeadroomMultiplier   // typically 2–3x

Step 5 — Broker constraints. Each partition replica costs memory, file handles, and replication CPU on its broker. Modern KRaft clusters tolerate far more partitions than ZooKeeper-era clusters, but per-broker limits still exist; pick a ceiling from your own metrics rather than a universal number. A few thousand partitions per broker is a reasonable starting point on moderate hardware.

P_max_per_broker = 4000   // tune to your hardware and observed metrics

Step 6 — Final count.

P_final = max(P_throughput, P_consumer)

Then check that placing P_final partitions (times the replication factor) across your brokers stays under the per-broker ceiling. If not, add brokers, reconsider the replication factor with a clear risk assessment, or split the topic.

#!/usr/bin/env bash
# Recommend a partition count from throughput and consumer needs
T_TARGET_MBPS=500
T_PARTITION_MBPS=25
CONSUMER_COUNT=30
HEADROOM=2
BROKER_COUNT=6
RF=3
MAX_PER_BROKER=4000

P_THROUGHPUT=$(( (T_TARGET_MBPS + T_PARTITION_MBPS - 1) / T_PARTITION_MBPS ))  # ceil
P_CONSUMER=$(( CONSUMER_COUNT * HEADROOM ))
P_FINAL=$(( P_THROUGHPUT > P_CONSUMER ? P_THROUGHPUT : P_CONSUMER ))

# Replica slots available cluster-wide, divided by RF for leader+follower copies
P_CLUSTER_MAX=$(( MAX_PER_BROKER * BROKER_COUNT / RF ))

echo "Throughput-driven partitions: $P_THROUGHPUT"
echo "Consumer-driven partitions:   $P_CONSUMER"
echo "Recommended partitions:       $P_FINAL"
echo "Cluster-wide partition capacity (incl. replicas): $P_CLUSTER_MAX"

if [ "$P_FINAL" -gt "$P_CLUSTER_MAX" ]; then
  echo "WARNING: recommended partitions exceed cluster capacity. Add brokers or reduce RF."
fi

Operational Patterns for Partition Lifecycle

Expanding Partitions

kafka-topics --alter --partitions increases partition count. It is additive only: Kafka cannot reduce a topic’s partition count via --alter (a removal path exists only behind a recent, experimental metadata-version gate). Expansion does not move existing data — records already written stay where they are, and only new records use the new partition layout. For keyed topics this remaps most keys going forward, breaking per-key ordering across the boundary, so confirm no order-sensitive consumer depends on stable key placement before running it.

# Increase partition count (cannot be undone via --alter)
kafka-topics --bootstrap-server broker1:9092 \
  --alter \
  --topic high-throughput-events \
  --partitions 60

After expansion, confirm new partitions are taking traffic by comparing per-partition log-end offsets (kafka-get-offsets) before and after, since BytesInPerSec won’t show you per-partition rates.

Reassigning Partitions Across Brokers

New brokers receive no existing partitions until you reassign. kafka-reassign-partitions generates and applies a plan to spread partitions across the expanded broker set. For high-throughput topics, throttle the reassignment so replication traffic does not starve live producers and consumers.

# 1. Generate a candidate plan
kafka-reassign-partitions --bootstrap-server broker1:9092 \
  --topics-to-move-json-file topics-to-move.json \
  --broker-list "0,1,2,3,4,5" \
  --generate

# 2. Execute with an inter-broker throttle (bytes/sec)
kafka-reassign-partitions --bootstrap-server broker1:9092 \
  --reassignment-json-file reassignment-plan.json \
  --execute \
  --throttle 52428800        # 50 MB/s

Track progress with --verify. Running --verify after completion also removes the throttle config — leave a reassignment “complete but unverified” and the throttle keeps limiting the cluster. You can raise or lower the throttle mid-flight by re-running --execute with a new --throttle value (it updates the same broker config keys) if you see consumer lag climbing.

Monitoring Partition Health

Alert on these real MBeans:

• kafka.server:type=ReplicaManager,name=UnderReplicatedPartitions — any sustained non-zero value warrants investigation.
• kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec,topic=<topic> — per-topic (not per-partition) ingress; track per broker to catch leadership imbalance.
• kafka.network:type=RequestMetrics,name=TotalTimeMs,request=Produce — per-broker produce latency; a spike on one broker often signals a hot partition it leads.
• kafka.consumer:type=consumer-fetch-manager-metrics,client-id="<client-id>", attribute records-lag-max (with ,topic=,partition= variants for per-partition lag) — exported on the consumer JVM; use it to find lag hot spots that point back to skew.

Put per-partition on-disk size (kafka-log-dirs) and per-partition offset deltas on a dashboard alongside these. A tight distribution is healthy; a long tail is a partitioning problem.

Conclusion

Partitioning for high-throughput topics is a spectrum between uniform distribution and key-based ordering, chosen from the ordering and skew requirements of the workload. The mechanics matter and are version-specific: the strictly-uniform sticky partitioner (KIP-794) since 3.3, murmur2-mod-N mapping for keyed records, and the fact that idempotent sequences are per-partition — so adding partitions is safe for EOS correctness but breaks per-key ordering.

Treat partitioning as an ongoing operational concern: benchmark single-partition throughput empirically, size partitions from real targets and broker limits, watch on-disk and offset-based skew (not a non-existent per-partition byte metric), and treat partition-count changes on keyed topics as migrations rather than live edits. That keeps a high-throughput topic balanced as it scales.