Producer Performance Tuning for Low Latency

Achieving consistently low producer latency in Apache Kafka requires deliberate tuning. When your application demands end-to-end latencies in single-digit milliseconds—for fraud detection, real-time bidding, or operational monitoring—every stage in the producer path must be measured and tuned. This guide gives actionable strategies for platform engineers, SREs, and backend developers who need to minimize latency while keeping acceptable throughput and durability.

The producer sits at the head of the Kafka pipeline, and its configuration shapes the latency profile of everything downstream. Before tuning individual parameters, understand how producer choices interact with the broader Producers, Consumers & Delivery Semantics framework. The delivery semantics you choose—at-most-once, at-least-once, or exactly-once—constrain your latency options. Stronger guarantees require synchronous acknowledgment from multiple brokers, which adds latency that no client-side tuning can remove.

Defaults changed in Kafka 4.0. KIP-1030 and earlier work changed several producer defaults. As of Kafka 4.0+, acks defaults to all, enable.idempotence defaults to true, retries defaults to Integer.MAX_VALUE (2147483647), and linger.ms defaults to 5. Several configurations in this guide override those defaults; the constraints between them (for example, idempotence requiring acks=all) are called out where they apply. Always confirm against the producer config reference for your version.

Understanding the Producer Send Path and Latency Budget

To tune for low latency, decompose the send path into stages and measure the contribution of each. A send() call does not immediately transmit data; the record enters an internal pipeline with batching and queuing points. End-to-end latency from send() to acknowledgment comprises:

1. Serialization — converting application objects to byte arrays with the configured key and value serializers.
2. Accumulator queuing — time spent in the RecordAccumulator waiting to be batched.
3. Batching delay — governed by linger.ms and batch.size.
4. Compression — CPU time to compress the batch (if compression.type is not none).
5. Network transmission — socket write to the broker, influenced by TCP settings and RTT.
6. Broker processing — log append, replication to followers (depending on acks), and acknowledgment dispatch.

Serialization is typically fast, but complex formats such as Avro with schema-registry lookups can add measurable latency. You can time it directly:

long startNanos = System.nanoTime();
byte[] serializedKey = keySerializer.serialize(topic, key);
byte[] serializedValue = valueSerializer.serialize(topic, value);
long serializationNanos = System.nanoTime() - startNanos;

After serialization, the record enters the accumulator and waits to be batched with other records for the same partition, governed by linger.ms and batch.size. Even with linger.ms=0, the producer can still micro-batch: records produced concurrently are grouped by the accumulator, and there is a fixed cost to allocate and fill MemoryRecords buffers. The network phase is the socket write to the broker, influenced by TCP congestion windows, Nagle’s algorithm, and RTT. Broker-side processing covers log append, replication, and acknowledgment dispatch.

To attribute latency to specific stages, wrap send() and record timestamps in the callback. Because the callback fires when the broker acknowledges (or the send fails), this measures wall-clock latency from enqueue to completion:

public class LatencyTrackingProducer<K, V> implements Producer<K, V> {
    private final KafkaProducer<K, V> delegate;
    private final Metrics metrics;

    @Override
    public Future<RecordMetadata> send(ProducerRecord<K, V> record, Callback callback) {
        long enqueueTime = System.nanoTime();
        return delegate.send(record, (metadata, exception) -> {
            long totalLatencyNanos = System.nanoTime() - enqueueTime;
            metrics.histogram("producer.total.latency.ns").update(totalLatencyNanos);
            if (callback != null) {
                callback.onCompletion(metadata, exception);
            }
        });
    }
}

The resulting distribution usually shows long-tail behavior driven by JVM GC pauses, leader elections, or network retransmissions. Focus on p99 and p999 rather than averages, since tail latency drives user-facing SLAs. HdrHistogram records these distributions with high dynamic range and no loss of precision.

A practical way to read this budget: in a single-rack cluster with acks=1, the local log append is sub-millisecond and the network RTT (often 0.2–1 ms intra-datacenter) dominates, so a healthy producer p99 lands in the low single-digit milliseconds. If you see a p99 of tens of milliseconds, the cause is almost always not a tuning knob in this guide but one of the long-tail drivers above—measure before you tune.

Configuring linger.ms and batch.size for Minimal Latency

linger.ms and batch.size are the primary controls over the latency/throughput trade-off in the batching layer. linger.ms is the maximum time a batch waits in the accumulator for more records before being sent; batch.size is the maximum size in bytes of a batch for a single partition. A batch is dispatched when it reaches batch.size or linger.ms expires, whichever comes first.

As of Kafka 4.0, linger.ms defaults to 5 (KIP-1030), not 0. The rationale is that slightly larger batches usually produce similar or lower end-to-end latency because fewer, larger requests reduce per-request IO overhead on both client and broker. For the strictest latency targets you may still set linger.ms=0, but benchmark it: on busy producers, the default of 5 often wins on p99 because it cuts request count. Setting linger.ms=0 removes the intentional delay but does not disable batching—concurrent send() calls are still coalesced.

A low-latency configuration that keeps some batching efficiency:

# Producer configuration for low-latency workloads
linger.ms=0
batch.size=16384
buffer.memory=33554432
acks=1
enable.idempotence=false

batch.size=16384 (16 KiB) is the default and keeps batches small enough to fill quickly under load. buffer.memory (default 32 MiB) is the total memory available to the accumulator; if it is exhausted, send() blocks for up to max.block.ms (default 60000), which manifests as a severe latency spike. Watch the bufferpool-wait-time-ns-total metric (and bufferpool-wait-ratio) to detect buffer pressure.

A common mistake is setting batch.size too low. Tiny batches (for example 256 bytes) force many small allocations, increasing GC pressure and CPU. Very large batches (for example 1 MiB) raise end-to-end latency because the batch takes longer to fill or to hit linger.ms. For latency-sensitive work, 8 KiB–32 KiB is usually a good range; tune it against your average record size.

max.in.flight.requests.per.connection (default 5) limits unacknowledged requests per broker connection. Setting it to 1 preserves strict ordering on retries but serializes requests and raises latency under load. Values up to 5 allow pipelining, overlapping network round trips. Note the idempotence constraint: with enable.idempotence=true (the default), Kafka requires max.in.flight.requests.per.connection ≤ 5 and acks=all. See Idempotent Producers and Exactly-Once Semantics for the full set of constraints.

The partitioner is part of your latency budget

A subtle source of tail latency hides in partition selection. Since Kafka 3.3 (KIP-794), the built-in partitioner for keyless records is sticky: rather than round-robining every record, it keeps producing to one partition until batch.size bytes have accumulated there, then switches. This is deliberate—it builds full batches even at linger.ms=0, which is exactly what you want for latency. Keyed records are unaffected; they always hash to a fixed partition (unless you set partitioner.ignore.keys=true, default false).

The same KIP adds adaptive partitioning (partitioner.adaptive.partitioning.enable, default true): the producer biases keyless records toward partitions on faster brokers, using accumulator queue depth as the load signal. This directly trims tail latency when one broker is briefly slow—but it only helps keyless traffic. If you key your records (common for ordering), a single slow broker hosting that key’s leader will still inflate your p99, and adaptive partitioning cannot route around it.

For a hard guard against a stalled broker, set partitioner.availability.timeout.ms (default 0, meaning disabled). With it enabled, a partition whose batches have not been accepted within the timeout is treated as unavailable and skipped for keyless records:

partitioner.availability.timeout.ms=100

This requires partitioner.adaptive.partitioning.enable=true to take effect and, again, applies only to keyless records.

Optimizing Acknowledgments and Durability for Speed

acks is the single most impactful setting for producer latency: it controls how many brokers must acknowledge a write before it is considered successful. The three settings—acks=0, acks=1, and acks=all (alias acks=-1)—span minimal latency to maximal durability. As of Kafka 4.0 the default is acks=all.

With acks=0, the producer does not wait for any broker acknowledgment; the record is “sent” once written to the socket buffer. This gives the lowest producer-perceived latency but no durability—if the broker drops the record, it is lost silently. Use it only for non-critical telemetry. Note that acks=0 is incompatible with idempotence and retries, so set enable.idempotence=false.

acks=1 has the leader acknowledge after writing to its local log, without waiting for followers. This balances latency and durability for many workloads. The leader writes to the OS page cache (not necessarily flushed to disk—Kafka relies on replication, not fsync, for durability), so the local-append component is typically sub-millisecond; observed latency is dominated by network RTT. If the leader fails before a follower replicates the record, it can be lost.

acks=all requires acknowledgment from all in-sync replicas (controlled together with the broker-side min.insync.replicas) before success. This survives leader failure but adds latency bounded by the slowest in-sync replica, roughly RTT + max(replica_replication_latency). With cross-region replicas this can add tens to hundreds of milliseconds.

For low-latency work that still wants durability, a common pattern is acks=1 with replicas spread across failure domains via rack-aware placement, accepting the small data-loss window on leader failure. Because acks=1 cannot be combined with the default idempotent producer, disable idempotence explicitly:

acks=1
enable.idempotence=false
max.in.flight.requests.per.connection=5
retries=0

retries=0 stops the producer from retrying failed sends, removing retry-induced latency spikes. (It also prevents reordering, which can otherwise occur when max.in.flight.requests.per.connection > 1 without idempotence.) If you do need retries, bound their total time with delivery.timeout.ms, the upper bound on the time from send() to success or failure, including batching and retries. For a low-latency target, set it just above your p99 goal:

delivery.timeout.ms=100
request.timeout.ms=80
retries=3
retry.backoff.ms=10

This permits a few quick retries but fails the send if the total exceeds 100 ms, preventing retry storms from cascading into application timeouts. Note that delivery.timeout.ms must be ≥ linger.ms + request.timeout.ms; with the default request.timeout.ms=30000, a 100 ms delivery timeout would be rejected, so lower request.timeout.ms (here to 80 ms) accordingly. The trade-off: a tight request.timeout.ms will fail sends during a transient broker GC pause rather than wait it out, so size it from your observed broker request-latency-max, not below it.

Network and Socket-Level Tuning for the Producer

The network between producer and broker is a frequent source of tail latency. TCP congestion control, Nagle’s algorithm, and socket buffer sizes all matter.

Nagle’s algorithm coalesces small packets and, combined with delayed ACKs, can add up to ~40 ms of latency to small writes. The Kafka clients already disable it: the producer sets TCP_NODELAY on its sockets internally. There is no Kafka producer configuration property to toggle this—it is not exposed, so there is nothing to set or verify in your producer properties. (Earlier drafts of this guide referenced a socket.tcp.nodelay property; no such property exists.)

Socket buffer sizes bound how much data the kernel queues before the producer blocks on a write. The producer exposes send.buffer.bytes (SO_SNDBUF, default 131072 = 128 KiB) and receive.buffer.bytes (SO_RCVBUF, default 32768 = 32 KiB); a value of -1 uses the OS default. Note the exact property names—there is no socket. prefix. For bursty low-latency workloads, raising the receive buffer can help:

send.buffer.bytes=131072
receive.buffer.bytes=131072

The OS congestion-control algorithm also affects latency. Linux defaults to CUBIC, which favors throughput on long-fat networks but can add queueing delay under loss. Within a single datacenter (low loss), BBR often reduces tail latency. This is a kernel setting, not a Kafka config. Check the current algorithm:

sysctl net.ipv4.tcp_congestion_control

Enable BBR on a recent kernel:

modprobe tcp_bbr
sysctl -w net.ipv4.tcp_congestion_control=bbr

Also size the NIC transmit queue (txqueuelen, often 1000–5000 on 10 GbE) to absorb bursts, and watch netstat -s for TCP retransmissions and ip -s link for drops—both are leading indicators of network-induced latency.

connections.max.idle.ms (default 540000 = 9 minutes) closes idle connections. For sporadic producers, reopening a connection incurs a TCP handshake plus a metadata refresh. Raise it, or set -1 to keep connections open indefinitely—at the cost of holding broker connection slots, which matters if brokers restart or you run many clients:

connections.max.idle.ms=-1

Compression and Serialization Overhead

Compression trades producer/consumer CPU for less network and disk. The producer supports gzip, snappy, lz4, and zstd. The trade-offs for a latency-sensitive producer:

For latency-sensitive workloads, lz4 is usually the best choice: moderate ratios with very fast compress/decompress. For small records or unconstrained bandwidth, compression.type=none removes the compression step entirely. Quantify the impact with kafka-producer-perf-test:

# Benchmark producer latency with a given compression type
kafka-producer-perf-test \
  --topic test-topic \
  --num-records 1000000 \
  --record-size 1024 \
  --throughput -1 \
  --producer-props bootstrap.servers=localhost:9092 \
    acks=1 linger.ms=0 compression.type=lz4

(The script is kafka-producer-perf-test.sh on Linux / .bat on Windows.) Re-run with compression.type set to none, snappy, lz4, and zstd, and compare the average and p99 latency the tool reports. Note that compression is most effective at the batch level, so its benefit grows with linger.ms and batch.size—which means a linger.ms=0 latency config gets less compression benefit than a throughput config, another reason to benchmark rather than assume.

Serialization can dominate the budget when serializers do heavy work or external lookups (such as Avro against a schema registry). To minimize it:

1. Pre-serialize in your application thread pool before calling the producer, so the work happens off the producer’s single sender thread.
2. Cache schemas aggressively. The Confluent schema-registry client already caches schema IDs after the first lookup per schema; make sure the cache is sized for your schema count so steady-state sends avoid registry round trips.
3. Use primitive serializers (StringSerializer, ByteArraySerializer, LongSerializer) where possible—their overhead is negligible.

A pre-serialization pattern that decouples serialization from the send path:

ExecutorService serializationPool = Executors.newFixedThreadPool(4);
Producer<byte[], byte[]> producer = new KafkaProducer<>(props);

CompletableFuture<RecordMetadata> sendAsync(MyRecord record) {
    return CompletableFuture
        .supplyAsync(() -> {
            byte[] key = keySerializer.serialize(topic, record.getKey());
            byte[] value = valueSerializer.serialize(topic, record.getValue());
            return new ProducerRecord<>(topic, key, value);
        }, serializationPool)
        .thenCompose(serializedRecord -> {
            CompletableFuture<RecordMetadata> result = new CompletableFuture<>();
            producer.send(serializedRecord, (metadata, exception) -> {
                if (exception != null) {
                    result.completeExceptionally(exception);
                } else {
                    result.complete(metadata);
                }
            });
            return result;
        });
}

This parallelizes serialization but adds memory and complexity, so use it only when serialization is demonstrably the bottleneck. (It completes the future from the producer’s send callback rather than blocking a pool thread on future.get(), which would otherwise waste threads.)

JVM and Operating System Tuning for the Producer

The producer runs in a JVM, so GC, thread scheduling, and memory management shape its tail latency. Tuning for latency differs from tuning for throughput: the goal is to minimize pause times and avoid stop-the-world events.

G1 (G1GC) is the default collector since JDK 9 and a sound choice for low-latency producers, performing most work concurrently for predictable pauses. A reasonable starting point for a 2 GiB heap:

-Xms2g -Xmx2g
-XX:+UseG1GC
-XX:MaxGCPauseMillis=20
-XX:InitiatingHeapOccupancyPercent=35
-XX:+DisableExplicitGC
-Xlog:gc*:file=/var/log/kafka/producer-gc.log:time,uptime,level,tags

-Xlog:gc* is the unified-logging form required on JDK 9+. The legacy flags -XX:+PrintGCDetails, -XX:+PrintGCDateStamps, and -Xloggc:<file> were deprecated in JDK 9 and removed in JDK 11; do not use them. MaxGCPauseMillis=20 targets ~20 ms pauses, and InitiatingHeapOccupancyPercent=35 starts concurrent marking earlier than the default 45%, lowering the chance of an evacuation failure (full GC). Watch the GC log for “to-space exhausted”, which signals the heap is too small or the pause target is too aggressive.

For stricter targets, ZGC (JDK 11+; production-ready and generational from JDK 21) and Shenandoah aim for sub-millisecond, heap-size-independent pauses. Enable ZGC with:

-XX:+UseZGC
-XX:+ZGenerational   # JDK 21+; default and non-removable from JDK 23
-Xms2g -Xmx2g

ZGC does almost everything concurrently, including compaction, but uses more CPU and can lower throughput versus G1. Benchmark to confirm the latency win justifies it. A heap-fixing detail that matters for tail latency: pin -Xms equal to -Xmx (as above) so the JVM commits all heap at startup; otherwise heap growth during a traffic spike triggers page faults and zeroing that show up as latency outliers.

On thread sizing: the producer uses a single sender (I/O) thread per producer instance—there is no producer-side config to add I/O threads. (The broker’s num.io.threads is a broker setting for request-handler threads and has nothing to do with the producer.) Watch io-wait-time-ns-avg to see whether the sender thread is saturated. If it is, run multiple producer instances in your application to spread load across connections, partitioned by key range or by topic.

Finally, pin the JVM to dedicated cores with taskset or cgroups to avoid scheduler migration and the cache misses and jitter it causes:

taskset -c 2-5 java -jar my-producer-app.jar

This binds the JVM to cores 2–5, isolating it from other processes.

Monitoring and Alerting on Producer Latency

Tuning without monitoring is guesswork; you need visibility to confirm gains and catch regressions. The producer exposes metrics via JMX under the MBean kafka.producer:type=producer-metrics,client-id=<id>. Key latency-relevant metrics:

• record-send-rate / record-send-total — records sent per second / cumulative.
• record-queue-time-avg / record-queue-time-max — time records spend in the accumulator.
• request-latency-avg / request-latency-max — time from request dispatch to acknowledgment.
• record-retry-rate — retry rate, indicating transient failures.
• bufferpool-wait-time-ns-total / bufferpool-wait-ratio — time spent waiting for buffer memory; non-zero means backpressure.
• io-wait-time-ns-avg — average time the sender thread waited for a socket to be ready.

(The -ns- suffixes were standardized in KIP-773; older Kafka versions exposed bufferpool-wait-time-total without -ns-.)

Read these together rather than in isolation. A worked example of diagnosis: if request-latency-avg is high but record-queue-time-avg is near zero, the producer is dispatching promptly and the delay is on the broker or network—producer tuning will not help. If instead record-queue-time-avg is high, records are stalling in the accumulator: check linger.ms, a saturated sender thread (io-wait-time-ns-avg), or buffer pressure (bufferpool-wait-ratio). This two-metric split tells you which side of the wire to investigate before you touch any config.

Export these to Prometheus with the JMX Exporter agent. A minimal jmx_exporter.yml that scrapes the producer gauges/averages above:

startDelaySeconds: 0
lowercaseOutputName: true
rules:
  - pattern: 'kafka.producer<type=producer-metrics, client-id=(.+)><>(record-send-rate|record-send-total|record-queue-time-avg|record-queue-time-max|request-latency-avg|request-latency-max|record-retry-rate|bufferpool-wait-time-ns-total|io-wait-time-ns-avg)'
    name: kafka_producer_$2
    labels:
      client_id: "$1"

Launch the producer with the agent:

java -javaagent:./jmx_prometheus_javaagent-1.0.1.jar=8080:jmx_exporter.yml \
     -jar my-producer-app.jar

The Kafka client emits these as Prometheus gauges (e.g. request-latency-avg is already an average computed inside the client), not histograms. There are no _bucket series, so histogram_quantile() does not apply. Alert directly on the client-reported average/max, and reserve true percentiles for a histogram you build yourself (for example from the LatencyTrackingProducer above or an HdrHistogram exported via the client SDK). An example alert on the client’s average request latency:

groups:
  - name: kafka_producer_latency
    rules:
      - alert: ProducerHighRequestLatency
        expr: kafka_producer_request_latency_avg > 50
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Producer avg request latency above 50ms for 5m"

Correlate with broker-side request metrics to separate producer-induced from broker-induced latency: kafka.network:type=RequestMetrics,name=TotalTimeMs,request=Produce and name=RequestQueueTimeMs,request=Produce. High broker request-queue time points at the broker, where producer tuning will not help—investigate broker load (and how consumer behavior feeds it via Consumer Group Rebalancing: Protocols and Tuning) or scale the cluster.

Display these alongside throughput, error rate, and resource utilization. The Optimizing Producer Batching and Compression for Throughput page covers the complementary throughput side when you need both.

Advanced Techniques: Asynchronous Send Patterns and Backpressure

The synchronous send().get() pattern blocks the calling thread until acknowledgment and caps throughput. Asynchronous sends keep the application moving but shift failure handling and backpressure to you.

The simplest async form is “fire and forget”: call send() and ignore the result—acceptable only where loss is tolerable. For robustness, use a non-blocking callback that records errors and metrics:

long sendTimeNanos = System.nanoTime();
producer.send(record, (metadata, exception) -> {
    if (exception != null) {
        errorCounter.increment();
        logger.warn("Failed to send record", exception);
    } else {
        successCounter.increment();
        latencyHistogram.update(System.nanoTime() - sendTimeNanos);
    }
});

The risk is unbounded memory growth if the application produces faster than the producer can send. The producer’s accumulator provides natural backpressure: once buffer.memory is exhausted, send() blocks for up to max.block.ms. A rising bufferpool-wait-time-ns-total (or non-zero bufferpool-wait-ratio) is your direct backpressure signal—throttle the application before it saturates the buffer. Note that max.block.ms also bounds the initial metadata fetch, so a producer that blocks on its very first send() to an unreachable cluster is hitting the same timeout, not buffer exhaustion.

For strict ordering with low latency, route related records to one partition via a partition key (or a single-partition topic). This removes cross-partition coordination and lets you keep ordering even with pipelining; with the default idempotent producer, ordering is preserved across retries even at max.in.flight.requests.per.connection=5.

Producer tuning does not stand alone. End-to-end latency also includes consumer processing, affected by rebalancing, offset management, and application logic. The Producers, Consumers & Delivery Semantics page covers how these interact, and Idempotent Producers and Exactly-Once Semantics details the latency cost of stronger guarantees. Tune the whole pipeline, not just the producer.