Producers, Consumers & Delivery Semantics

In Apache Kafka, the reliability of a streaming platform is defined by the interaction between producers, consumers, and the delivery guarantees that bind them. Platform engineers, SREs, and backend developers running Kafka in production need to move beyond the basic publish-subscribe model and reason about the configuration trade-offs, failure modes, and recovery procedures that actually determine whether data is lost, duplicated, or processed exactly once. This guide builds that mental model: how producers serialize, batch, and commit records; how consumers join groups, fetch data, and manage offsets; and how broker, producer, and consumer coordinate to deliver a given semantic guarantee.

A misconfigured producer can drop messages during a leader election. An improperly tuned consumer group can stall an entire pipeline when one member is slow. The choice between at-least-once and exactly-once semantics ripples through application logic, monitoring, and incident response. Every configuration parameter has a cost, every delivery guarantee has a prerequisite, and every failure mode needs a rehearsed recovery procedure. Throughout, configuration values reference Apache Kafka defaults as of the current release line; where a default changed in a specific version, that is called out explicitly.

The Producer Request Pipeline: From Application Thread to Broker Acknowledgment

When an application calls producer.send(), the record enters a multi-stage pipeline: serialization, partitioning, optional compression, accumulation into batches, and transmission to the partition leader. Each stage adds latency and has configuration levers that affect throughput and durability.

The producer uses two threads: the application thread that calls send(), and a background I/O thread (the Sender) that handles network communication. The application thread serializes the record and places it into a per-partition record accumulator, where it waits until a batch fills or a time threshold elapses. This batching is the dominant factor in producer throughput. Without it, every record pays the full round-trip cost of a produce request; with it, that cost is amortized across many records.

The accumulator is governed by batch.size (default 16384 bytes) and linger.ms (default 0). batch.size is the maximum size of a single batch per partition before it is sent; linger.ms is the maximum time the producer waits to let a batch fill before sending it even if it is below batch.size. With the default linger.ms=0, the producer still batches whatever records have accumulated while a previous request is in flight, but it does not deliberately wait. Raising linger.ms increases batching efficiency at the cost of added latency on low-volume partitions. A practical range is 5–50 ms of linger with batch.size between 16 KB and 1 MB, tuned to record size and throughput.

# Producer configuration for balanced throughput and latency
batch.size=65536
linger.ms=10
compression.type=lz4
max.in.flight.requests.per.connection=5
buffer.memory=33554432

Once a batch is ready, the record is routed to a partition. If the record has a key, the default partitioner hashes the key with murmur2 and maps it modulo the number of partitions, so all records with the same key land on the same partition and preserve order. If the key is null, the producer uses sticky partitioning (KIP-480, default since Kafka 2.4): it fills batches for one partition before switching, which improves batching efficiency over the older round-robin behavior. Custom partitioners are possible but add operational complexity and must be tested under partition count changes.

The I/O thread transmits each batch to the leader broker for its partition. The broker appends the records to the partition log, and—depending on acks—waits for in-sync follower replicas to replicate the write before responding. Retries, governed by retries and retry.backoff.ms, handle transient network failures and leader elections. Without idempotence, a retry can produce a duplicate if the original write succeeded but its acknowledgment was lost. Since Kafka 3.0, idempotence is enabled by default, which closes that gap (covered in the delivery-semantics section).

For deep tuning of this pipeline under high throughput, see Producer Performance Tuning for Low Latency, which covers buffer memory sizing, the I/O thread, and compression selection.

Consumer Group Mechanics: Joining, Syncing, and Partition Assignment

Consumer groups are Kafka’s scaling primitive. Multiple consumer instances cooperatively read from a set of partitions, with each partition assigned to exactly one consumer in the group at a time. This gives both parallelism and fault tolerance: if a consumer fails, its partitions are reassigned to surviving members and processing continues without data loss, provided offsets are managed correctly.

A consumer joins by sending a JoinGroup request to the group coordinator, a broker that manages the group’s metadata. The first member to join becomes the group leader and is responsible for computing the partition assignment. Other members send their subscriptions to the coordinator, which forwards them to the leader. The leader applies the configured partition.assignment.strategy and returns an assignment plan, which the coordinator distributes to all members via SyncGroup.

This process—a rebalance—is a critical operational event. With the older eager protocol, every consumer revokes all of its partitions before the new assignment is distributed; this stop-the-world pause can cause significant processing gaps in large groups or where partition revocation is expensive. The incremental cooperative rebalance protocol, implemented by CooperativeStickyAssignor, lets consumers keep partitions that are not moving and revoke only those being reassigned, continuing to process the rest throughout the rebalance. This shrinks the disruption window and is recommended for production. The default partition.assignment.strategy is [RangeAssignor, CooperativeStickyAssignor]; to use cooperative rebalancing exclusively, set the strategy to CooperativeStickyAssignor alone (and follow the documented upgrade path when migrating an existing group). Rebalance tuning is covered in Consumer Group Rebalancing: Protocols and Tuning, including session timeouts, heartbeat intervals, and static group membership.

// Consumer configuration with the cooperative sticky assignor
Properties props = new Properties();
props.put("bootstrap.servers", "broker1:9092,broker2:9092");
props.put("group.id", "order-processing-service");
props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
props.put("value.deserializer", "org.apache.kafka.common.serialization.ByteArrayDeserializer");
props.put("partition.assignment.strategy", "org.apache.kafka.clients.consumer.CooperativeStickyAssignor");
props.put("session.timeout.ms", "45000");
props.put("heartbeat.interval.ms", "3000");
props.put("max.poll.interval.ms", "300000");

max.poll.interval.ms (default 300000, 5 minutes) deserves special attention. It is the maximum time allowed between calls to poll(). If processing one batch takes longer than this, the consumer is considered failed: it leaves the group, its partitions are reassigned, and a rebalance occurs. A processing spike that exceeds the interval triggers a rebalance, which pauses consumers, which increases lag—a feedback loop that can become a rebalance storm. Set this high enough to cover worst-case processing of a full poll() batch, but not so high that genuine failures go undetected. Note that session.timeout.ms and heartbeat.interval.ms detect process/network death (heartbeats run on a background thread), while max.poll.interval.ms detects a live process that has stopped polling.

Offset Management: The Contract Between Producer and Consumer

Offsets are the bookmarks that let consumers resume from a known position after restarts or failures. Kafka stores committed offsets in the internal compacted topic __consumer_offsets, with the group coordinator acting as offset manager. When and how you commit offsets relative to processing is what determines the delivery semantics your application actually experiences.

The default is automatic commit (enable.auto.commit=true, auto.commit.interval.ms=5000). Auto-commit happens inside poll(): on a poll() call, the consumer commits the offsets of records returned by the previous poll if the interval has elapsed. Because the commit happens only after those records were handed to the application, a crash after processing but before the next auto-commit causes re-delivery on restart. This is an at-least-once posture, and it requires processing to tolerate duplicates.

Manual control gives finer semantics. With enable.auto.commit=false, the application calls commitSync() or commitAsync() where it chooses. commitSync() blocks until the commit is acknowledged and retries on retriable errors; commitAsync() returns immediately and is higher throughput but needs a callback to handle failures, and you must avoid committing a lower offset after a later one has already been committed.

// Manual offset commit with at-least-once semantics
try {
    while (true) {
        ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(1000));
        for (ConsumerRecord<String, String> record : records) {
            processRecord(record);
        }
        consumer.commitAsync((offsets, exception) -> {
            if (exception != null) {
                log.error("Async commit failed for offsets {}", offsets, exception);
                // Implement retry or alerting logic
            }
        });
    }
} finally {
    consumer.close();
}

The operational rule is simple: commit timing relative to processing defines the delivery boundary. Commit before processing and you get at-most-once (records lost if the consumer crashes after the commit). Commit after processing and you get at-least-once (records duplicated if the consumer crashes after processing but before the commit). Exactly-once requires coordinating offset commits with the output writes in a single transaction, covered below.

For a full treatment of commit strategies, the __consumer_offsets topic, and troubleshooting commit failures, see Offset Management Strategies for Reliable Processing.

Delivery Semantics: At-Most-Once, At-Least-Once, and Exactly-Once

Delivery semantics describe how many times a message is delivered and processed. In Kafka these guarantees emerge from the combination of producer acknowledgments, broker replication, and consumer offset management. No single component provides a guarantee in isolation; the end-to-end behavior is a property of the whole pipeline.

At-most-once is the weakest guarantee: messages may be lost but are never duplicated. It arises when the producer sets acks=0 (fire-and-forget) or when the consumer commits offsets before processing. This is rarely appropriate for business-critical data, but it suits high-volume telemetry where occasional loss is acceptable and latency must be minimal.

At-least-once is the most common posture. The producer uses acks=all (the default since Kafka 3.0) with retries enabled, so acknowledged records are durably persisted, and the consumer commits offsets after processing. The trade-off is possible duplicates: historically a producer retry could write a record twice, and a consumer restart between processing and commit can reprocess records. With idempotence on by default (3.0+), producer-side duplicates from retries are eliminated, but consumer-side reprocessing still requires idempotent handling or sink-side deduplication.

The broker side of durability is set by min.insync.replicas combined with producer acks=all. To change this dynamically at the broker level, use kafka-configs --alter (not --describe, which only reads). Inspect first, then alter:

# Read the current dynamic config for a broker
kafka-configs --bootstrap-server localhost:9092 \
  --describe --entity-type brokers --entity-name 0

# Set a broker-level default for min.insync.replicas (cluster-wide default)
kafka-configs --bootstrap-server localhost:9092 \
  --alter --entity-type brokers --entity-default \
  --add-config min.insync.replicas=2

min.insync.replicas is most often set per topic; the broker-level value above acts as the default for topics that do not override it. With acks=all, a write succeeds only once at least min.insync.replicas replicas (including the leader) are in sync; if fewer are available, the producer receives NotEnoughReplicasException and can retry.

Exactly-once semantics (EOS) is the strongest guarantee: each record is processed once, even across producer retries, broker failures, and consumer restarts. It is built from three pieces. The idempotent producer (enable.idempotence=true, on by default since 3.0) attaches a producer ID and per-partition sequence number to each batch so the broker discards duplicates; it requires acks=all, retries>0, and max.in.flight.requests.per.connection<=5. The transactional producer (set transactional.id and call initTransactions(), beginTransaction(), sendOffsetsToTransaction(), commitTransaction()) atomically writes to multiple partitions and commits the consumed offsets in one transaction. Finally, consumers set isolation.level=read_committed (the default is read_uncommitted) to skip records from aborted or in-flight transactions.

EOS has real cost. Transactions require coordinator state. The producer holds open transactions that abort if not committed within transaction.timeout.ms. read_committed consumers can only read up to the last stable offset, so they wait for in-flight transactions to resolve, adding latency. For the full operational picture, see Idempotent Producers and Exactly-Once Semantics and Transactional Messaging with Kafka Producers.

Consumer Lag: Monitoring, Alerting, and Remediation

Consumer lag—the difference between a partition’s log-end offset and the offset last committed by a consumer group—is the single most important health metric for a Kafka pipeline. Sustained lag growth signals a capacity problem, a processing bottleneck, or a failed consumer.

Lag can be read on the broker side from committed offsets versus log-end offsets, and on the client side from the consumer’s records-lag-max JMX metric (per the kafka.consumer:type=consumer-fetch-manager-metrics group). The operational standard is to export both to a time-series system such as Prometheus and alert on them.

# Check consumer group lag from the command line
kafka-consumer-groups --bootstrap-server localhost:9092 \
  --group order-processing-service --describe

# Output includes CURRENT-OFFSET, LOG-END-OFFSET, and LAG columns

Alert thresholds need care. A static threshold (“alert when lag exceeds 10,000”) is simple but brittle across varying throughput. A rate-based approach comparing production rate to consumption rate is more robust but harder to build. A common pattern is to alert on sustained lag growth over a rolling window, with an absolute lag ceiling as a backstop. For building this pipeline, see Monitoring Consumer Lag with JMX and Prometheus.

When lag fires, remediation depends on root cause. If the consumer is CPU-bound, add instances—but only up to the partition count, since each partition is consumed by one member. If it is I/O-bound on a downstream system, connection pooling, batching, or caching help. If a poison-pill record stalls processing, skip or quarantine it (the dead-letter pattern below). For sustained backlogs—partition reassignment, pause/resume, and catch-up strategies—see Handling Large Backlogs and Consumer Lag.

Failure Modes and Resilience Patterns

Production Kafka pipelines fail in predictable ways, categorized by component: producer-side, broker-side, and consumer-side, though failures often cascade across all three.

Producer-side failures surface as TimeoutException, NetworkException, or RecordTooLargeException. Retries handle transient issues; persistent ones need intervention. A common pitfall is buffer.memory exhaustion: if the producer cannot send faster than the application produces, the accumulator fills and send() blocks for up to max.block.ms, then throws. Monitor buffer pool usage and either slow production or add broker capacity.

// Producer with error-handling callback
producer.send(record, (metadata, exception) -> {
    if (exception != null) {
        if (exception instanceof TimeoutException) {
            // Retry with backoff or write to fallback storage
            fallbackWriter.write(record);
        } else if (exception instanceof RecordTooLargeException) {
            // Split record or alert on misconfiguration
            log.error("Record exceeds max.request.size", exception);
        } else {
            log.error("Unexpected producer error", exception);
        }
    } else {
        metrics.recordSuccess(metadata.topic(), metadata.partition());
    }
});

Broker-side failures include leader elections, disk failures, and network partitions. Kafka’s replication protocol does not lose committed data as long as an in-sync replica survives. min.insync.replicas with acks=all defines the durability boundary: if fewer than min.insync.replicas brokers are in sync, the write is rejected and the producer can retry. Too low risks loss during concurrent failures; too high reduces write availability during routine maintenance. For a replication factor of 3, min.insync.replicas=2 is the common choice—it tolerates one failure while keeping writes available.

Consumer-side failures are dominated by rebalance storms and poison pills. A rebalance storm occurs when members repeatedly leave and rejoin, often from exceeding max.poll.interval.ms under load. The immediate mitigation is to raise the poll interval and lower max.poll.records; the durable fix is to move heavy processing off the poll loop into a separate worker pool or stream-processing framework.

Poison-pill records—those that crash or hang processing—need a defensive consumption pattern: wrap processing in try/catch and, on a non-retriable failure, route the record to a dead-letter queue (DLQ) so one bad record cannot block a partition indefinitely. DLQ topic design and replay are covered in Implementing Dead Letter Queues in Kafka Consumers.

// Defensive consumption with dead letter queue routing
while (true) {
    ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(1000));
    for (ConsumerRecord<String, String> record : records) {
        try {
            processRecord(record);
        } catch (NonRetryableException e) {
            dlqProducer.send(new ProducerRecord<>("dlq-topic", record.key(), record.value()));
            // Optionally commit offset to skip the poison record
        } catch (RetryableException e) {
            // Pause partition and retry with backoff
            consumer.pause(Collections.singletonList(new TopicPartition(record.topic(), record.partition())));
            scheduleRetry(record);
        }
    }
    consumer.commitSync();
}

Operational Configuration Reference and Tuning Methodology

Tuning Kafka producers and consumers is iterative: measure, hypothesize, change, validate. The tables below summarize the key parameters, their Apache Kafka defaults, and tuning guidance. Note that since Kafka 3.0 the producer ships with strong defaults—acks=all and enable.idempotence=true—so a new producer is durable and de-duplicated out of the box; weakening these is a deliberate choice.

Producer parameters:

Consumer parameters:

The methodology is a four-step cycle: measure, hypothesize, change, validate. Establish baselines—producer request latency percentiles, fetch latency, end-to-end latency, and consumer lag. Identify the bottleneck (producer throughput, broker I/O, network, or consumer processing), change one parameter on a canary instance, measure, and roll back immediately if it regresses.

# Measure producer performance with kafka-producer-perf-test
kafka-producer-perf-test --topic perf-test \
  --num-records 10000000 \
  --record-size 1024 \
  --throughput -1 \
  --producer-props bootstrap.servers=localhost:9092 \
    acks=all linger.ms=10 batch.size=65536 compression.type=lz4

This writes records as fast as possible (--throughput -1) and reports throughput, latency percentiles, and errors. Use it in a pre-production environment to validate changes before production. The Apache Kafka Producer Configuration reference and Consumer Configuration reference are the authoritative sources for defaults and behavior; the Kafka Improvement Proposals (KIPs) document the rationale behind default changes such as the 3.0 producer hardening.

End-to-End Pipeline Validation and Failure Injection

Confidence in delivery semantics comes from deliberate failure testing, not documentation. A robust suite exercises the full producer-broker-consumer chain under simulated failures and verifies the observed behavior matches the intended guarantee.

The suite should cover at least:

1. Leader election during production: while a producer writes with acks=all, terminate the leader for the target partition. Verify the producer retries, a new leader is elected, and no committed record is lost. Measure the write-unavailability window.

2. Consumer restart with uncommitted offsets: produce a known sequence, consume a subset without committing, terminate the consumer, restart it, and verify the uncommitted records are re-delivered (at-least-once)—or, with transactions, that exactly-once holds.

3. Network partition between consumer and broker: use iptables to drop traffic between a consumer and the cluster. Verify the session times out, a rebalance moves its partitions to other members, then restore connectivity and confirm the consumer rejoins cleanly.

4. Poison-pill injection: produce a record known to fail processing. Verify the consumer routes it to the DLQ, commits the offset, and continues with subsequent records.

# Trigger a preferred leader election for a specific partition
kafka-leader-election --bootstrap-server localhost:9092 \
  --election-type preferred --topic test-topic --partition 0

# Drive load and observe producer behavior during the election
kafka-producer-perf-test --topic test-topic --num-records 100000 \
  --record-size 100 --throughput 1000 \
  --producer-props bootstrap.servers=localhost:9092 acks=all

For transactional pipelines, also test timeout/abort: produce within a transaction, delay the commit beyond transaction.timeout.ms, and verify the transaction aborts, the records are invisible to read_committed consumers, and the producer surfaces the abort so your handling is exercised.

The Apache Kafka documentation on operations covers additional recovery procedures—disk replacement, broker decommissioning, cluster expansion. Put these in runbooks and rehearse them.

Capacity Planning and Scaling Strategies

A Kafka pipeline’s lifecycle includes growth, and it must scale without degrading guarantees or causing unplanned downtime. That requires knowing the throughput limits of each component.

Producer scaling is mostly an application-tier concern: producers are stateless, so adding instances increases aggregate throughput roughly linearly, as long as the broker tier can absorb it. The broker-side limit is per-partition write throughput—on modern hardware a single partition typically sustains tens of MB/s depending on record size, compression, and disk. When aggregate production exceeds what the current partitions provide, increase the topic’s partition count to spread writes across more brokers.

# Increase partition count for an existing topic
kafka-topics --bootstrap-server localhost:9092 \
  --alter --topic high-throughput-topic --partitions 32

# Note: increasing partitions does not redistribute existing data,
# and it changes key-to-partition mapping for future records

Consumer scaling is bounded by partition count: each partition is consumed by exactly one member, so maximum group parallelism equals the partition count. Needing more parallelism means more partitions—but increasing partitions changes the key-to-partition mapping for new records and so can break per-key ordering. For strict key ordering, plan partition counts up front (often over-partitioning) rather than growing them later.

Broker scaling adds brokers and reassigns partitions to spread load. kafka-reassign-partitions moves partitions online, but the replication is I/O-intensive and should be throttled. Monitor the under-replicated-partitions metric during the move.

# Generate a reassignment plan to move partitions to new brokers
kafka-reassign-partitions --bootstrap-server localhost:9092 \
  --topics-to-move-json-file topics.json \
  --broker-list "0,1,2,3,4" --generate

# Execute the reassignment with a 50 MB/s throttle
kafka-reassign-partitions --bootstrap-server localhost:9092 \
  --reassignment-json-file reassignment.json --execute \
  --throttle 52428800

# Verify completion (this also removes the throttle when done)
kafka-reassign-partitions --bootstrap-server localhost:9092 \
  --reassignment-json-file reassignment.json --verify

Note that throttles set by --execute persist until removed; running --verify after the reassignment completes removes them. Throughout scaling, keep watching consumer lag: adding partitions can briefly increase lag as the group rebalances. The cooperative sticky assignor minimizes this, and the pipeline from Monitoring Consumer Lag with JMX and Prometheus provides the visibility to do it safely.

Make scaling predictable, repeatable, and reversible. Validate cluster health before and after reassignments, keep headroom (roughly 30% on broker disk, network, and CPU) for spikes and rebalancing overhead, and document your deployment’s real limits: maximum partitions per broker, throughput per partition, and the group size at which rebalance times become unacceptable.