Optimizing Producer Batching and Compression for Throughput

When operating Apache Kafka at scale, producer efficiency directly dictates the upper bound of system throughput. While broker and consumer tuning receive significant attention, the producer’s batching and compression strategy is often the single most impactful lever for maximizing bytes-per-second while managing resource costs. This guide dissects the mechanics of batch.size, linger.ms, and compression codecs, providing actionable, version-aware configurations for platform and SRE teams running production workloads.

Understanding the interplay between these settings requires a solid grasp of the producer’s internal architecture. If you haven’t established a baseline for your producer’s behavior, revisiting the fundamentals in Producer Performance Tuning for Low Latency will ground the advanced batching strategies discussed here. That foundation covers the throughput-versus-latency trade-offs we now optimize explicitly for throughput-dominant use cases. For a broader view of how these producer behaviors interact with the entire messaging lifecycle, the Producers, Consumers & Delivery Semantics pillar provides context on delivery guarantees and their performance implications.

Know Your Defaults: They Changed

Before tuning anything, anchor on the current defaults, because several changed in recent major releases and most tuning advice on the web is stale. As of Apache Kafka 4.x:

The single most common mistake in 2026-era tuning is assuming linger.ms=0 and acks=1. Verify the defaults for your client version against the producer configuration reference before reasoning about batch behavior.

The Anatomy of a Producer Batch

The Kafka producer maintains a per-partition deque of record batches inside the RecordAccumulator. The calling thread appends records to the batch at the tail; a single background I/O thread (the Sender) drains ready batches and dispatches them to the appropriate broker leaders.

A batch becomes ready to send when either it fills to batch.size bytes or its linger.ms timer expires—whichever comes first. A batch is never held longer than linger.ms, but it can be sent earlier if it fills. For high-throughput partitions, batch.size is the effective governor; linger.ms is the backstop that bounds latency on lower-volume partitions.

# Producer configuration for throughput-oriented batching
batch.size=131072
linger.ms=10

A value of 131072 (128 KB) or 262144 (256 KB) is a common throughput-oriented starting point, well above the 16 KB default. batch.size is an upper bound per partition, not a guarantee—if records arrive faster than they drain, batches fill; if not, linger.ms ships them partially full.

A single record larger than batch.size is still sent in its own batch (the producer does not split a record), but it must fit within max.request.size (default 1048576, 1 MB) and the broker/topic limit max.message.bytes (broker default message.max.bytes, 1048588). Oversized records raise RecordTooLargeException synchronously from send(); the producer does not silently shrink the batch or retry around it, so size your limits deliberately rather than relying on a fallback.

Tuning linger.ms

Because linger.ms now defaults to 5, the producer already trades a few milliseconds of latency for batching out of the box. Tuning it is therefore about deciding how much additional delay you can spend to fill batches—not about turning batching “on.”

Setting linger.ms to 0 reverts to send-immediately behavior, which is wasteful for throughput-bound workloads: each produce request carries fixed network round-trip and broker-side processing overhead that amortizes far better across larger batches. For throughput-dominant pipelines, a value of 5–20 ms is typically enough to consolidate records without introducing user-perceptible latency. The optimal value depends on arrival rate: a high-frequency producer fills batches with little linger, while a spiky producer benefits from a longer window to absorb bursts.

// Java producer configuration demonstrating linger.ms and batch.size
Properties props = new Properties();
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "broker1:9092,broker2:9092");
props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG,
    "org.apache.kafka.common.serialization.StringSerializer");
props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG,
    "org.apache.kafka.common.serialization.StringSerializer");
props.put(ProducerConfig.BATCH_SIZE_CONFIG, 131072);   // 128 KB
props.put(ProducerConfig.LINGER_MS_CONFIG, 10);        // 10 ms linger
props.put(ProducerConfig.COMPRESSION_TYPE_CONFIG, "lz4");
// acks defaults to "all"; set explicitly if you intend a weaker guarantee
props.put(ProducerConfig.ACKS_CONFIG, "all");

Monitor batch-size-avg and batch-size-max (MBean kafka.producer:type=producer-metrics,client-id=*) to confirm your settings yield the batch sizes you expect. If batch-size-avg stays far below batch.size on a busy partition, raise linger.ms incrementally until you hit diminishing returns.

Compression Codecs: CPU vs. Network Trade-offs

Compression amplifies batching by shrinking the bytes transmitted and stored, and—importantly—Kafka compresses the entire batch as a unit, so larger batches compress better. Kafka supports gzip, snappy, lz4, and zstd. Each sits at a different point on the ratio-versus-CPU spectrum.

• lz4 — a strong default for throughput-oriented workloads: fast, with a useful ratio on text-like payloads.
• zstd — higher compression ratio at higher CPU cost; choose it when network or storage is the binding constraint. It also exposes a tunable level via compression.zstd.level (added in Kafka 3.8, KIP-390).
• gzip — highest CPU cost; reserve it for cases where storage cost dominates everything else.
• snappy — fast but generally a lower ratio than lz4.

Actual ratios are payload-dependent—JSON and logs compress well, already-compressed or encrypted blobs do not—so measure rather than assume a fixed percentage.

# Broker-side compression configuration (server.properties)
compression.type=producer

compression.type=producer (the broker default) tells the broker to retain whatever codec the producer used, avoiding a decompress/recompress cycle and preserving end-to-end compression. Note that the broker may still decompress to validate records (for example, when assigning offsets or when the inter-broker message format requires it); setting a different broker codec forces an unconditional recompress that can become a CPU bottleneck under high throughput. Keep producer and broker codecs aligned unless you have a specific reason not to.

Batching and Compression Under Memory Pressure

Both larger batches and compression draw on producer heap. The accumulator allocates from a buffer pool of size buffer.memory (default 33554432 bytes, 32 MB). When the pool is exhausted, send() blocks for up to max.block.ms (default 60000 ms) before throwing TimeoutException, effectively throttling the application.

# Monitor producer buffer-pool usage via JMX (Kafka 4.x)
# kafka-jmx.sh wraps kafka-run-class.sh org.apache.kafka.tools.JmxTool.
# The legacy kafka.tools.JmxTool path was removed in 4.0 (KIP-906).
bin/kafka-jmx.sh \
  --jmx-url service:jmx:rmi:///jndi/rmi://<producer-host>:<jmx-port>/jmxrmi \
  --object-name 'kafka.producer:type=producer-metrics,client-id=*' \
  --attributes buffer-total-bytes,buffer-available-bytes,bufferpool-wait-time-ns-total,waiting-threads

bufferpool-wait-time-ns-total (cumulative nanoseconds threads spent waiting for buffer space) and waiting-threads (threads currently blocked on the pool) are the load-bearing signals here; a steadily climbing wait time means the pool is too small for your batch size and partition fan-out.

Right-size the pool for your fan-out. A workable rule of thumb is batch.size * active_partitions plus headroom for in-flight requests and compression scratch space. For a producer writing 256 KB batches across 50 partitions, the batch buffers alone need ~12.5 MB; budget 32–64 MB to absorb in-flight requests and bursts. Because the producer compresses a batch only when it is sealed and ready to send, peak memory transiently exceeds the steady-state estimate—another reason to leave headroom rather than size to the theoretical minimum.

If you see sustained buffer waits, either raise buffer.memory or reduce batch.size/partition fan-out until the pressure subsides.

Validating Throughput Gains with Benchmarking

Configuration changes must be validated empirically; isolate one parameter at a time to avoid cargo-cult tuning. The kafka-producer-perf-test.sh tool shipped with Kafka generates a standardized workload.

# Benchmark producer throughput with a given batching + compression profile
bin/kafka-producer-perf-test.sh \
  --topic throughput-test \
  --num-records 5000000 \
  --record-size 1024 \
  --throughput -1 \
  --producer-props bootstrap.servers=broker1:9092,broker2:9092 \
    acks=all linger.ms=10 batch.size=131072 compression.type=lz4

--throughput -1 removes the rate cap so the tool drives as fast as it can; the run prints records/sec, MB/sec, and latency percentiles (avg, 50th, 95th, 99th, 99.9th). Run a baseline, then vary one knob at a time and watch throughput, p99 latency, and broker CPU. The producer configuration reference is the authoritative source for every knob and its default.

A typical sequence for a throughput-bound producer: (1) raise batch.size to 128–256 KB; (2) tune linger.ms (it already defaults to 5, so push it toward 10–20 ms if batches stay undersized); (3) enable lz4 (or zstd if network-bound); (4) scale buffer.memory to the larger batches and fan-out. Validate end-to-end latency against your SLOs at each step, and remember that acks=all couples your throughput ceiling to replication health.

Operational Guardrails and Monitoring

The metrics below—exposed via JMX under kafka.producer:type=producer-metrics,client-id=* and scrapable through a JMX exporter—give a working view of batching and compression health.

# Sample Prometheus alerting rule (metric name depends on your JMX exporter mapping)
groups:
  - name: kafka_producer_alerts
    rules:
      - alert: ProducerBatchUndersized
        expr: avg by (client_id) (kafka_producer_producer_metrics_batch_size_avg) < 65536
        for: 15m
        labels:
          severity: warning
        annotations:
          summary: "Producer batch size is below 64 KB"
          description: "Average batch size for {{ $labels.client_id }} is {{ $value }} bytes, indicating potential throughput loss."

The exact Prometheus metric name depends on your exporter’s MBean-to-metric mapping (the jmx_exporter default lowercases and underscores the MBean path); confirm the series name in your own /metrics output before deploying the rule.

When batches stay undersized despite adequate linger.ms, check the actual production rate per partition. Low-volume partitions naturally produce small batches—expected, not alarming. If high-volume partitions are also undersized, look at whether max.in.flight.requests.per.connection or broker-side request-queue pressure is throttling the Sender.

Watch compression-rate-avg: a value near 1.0 means compression is wasting CPU, common with already-compressed media or encrypted payloads. For such topics, set compression.type=none to drop the CPU cost without losing meaningful network savings.

Producer batching and broker behavior are coupled. Brokers handling large compressed batches can see higher GC pause times under memory pressure; size broker heap with producer batch characteristics in mind. Finally, KIP-480 (Sticky Partitioner)—enabled by default since Kafka 2.4—improves batching for records without explicit keys by sticking to one partition until its batch fills, then rotating, which produces fuller batches and fewer, larger requests.

Treating batching and compression as one tuning surface—rather than isolated knobs—lets platform teams extract substantial throughput gains while keeping the durability guarantees (acks=all, idempotence) that production pipelines depend on.