Handling Large Backlogs and Consumer Lag
In any Apache Kafka deployment operating at scale, consumer lag is a condition every platform and SRE engineer eventually has to manage. Lag is the delta between the latest offset produced to a partition (the log end offset) and the offset last committed by a consumer group for that partition. When this gap widens into a large, sustained backlog, it erodes the real-time guarantees your pipelines depend on and pushes processing latency past your SLAs. Lag by itself does not cause data loss — it causes delay — but a backlog left unattended can collide with a topic’s retention policy, at which point unconsumed data is deleted and the loss becomes real. This guide gives you a systematic way to diagnose, mitigate, and prevent large backlogs, moving past simple threshold alerting toward a working understanding of the mechanics that drive lag accumulation.
We assume a solid grasp of the foundational Producers, Consumers & Delivery Semantics that govern how data flows through your system. The strategies here build on those concepts, translating delivery guarantees into concrete remediation tactics. Whether you are responding to a P1 incident where lag is climbing or proactively tuning consumers for predictable load spikes, the techniques below help you restore and hold processing equilibrium.
Understanding the Anatomy of Consumer Lag
Before combating large backlogs, decompose consumer lag into its parts. Lag is not a single metric; it is a composite signal reflecting production throughput, consumer processing speed, and the mechanics of the consumer group protocol.
For a single partition, lag is the difference between the log end offset (LEO) and the consumer’s committed offset. The LEO is the offset that will be assigned to the next record produced to that partition. The committed offset is the last position the consumer group has acknowledged, stored in the internal __consumer_offsets topic. When you observe high lag, the cause almost always falls into one of three categories: a producer-side throughput spike that outpaces consumer capacity, a consumer-side bottleneck that slows processing below the ingest rate, or a stall caused by a rebalance during which no records are consumed.
The role of rebalances is easy to underestimate. A poorly tuned consumer group can spend a disproportionate share of its lifecycle rebalancing, and during an eager (stop-the-world) rebalance, every consumer in the group revokes all of its partitions and consumes nothing until assignment completes. This is often the hidden cause of a backlog that appears suddenly and then persists. The mechanics are explored in our guide to Consumer Group Rebalancing: Protocols and Tuning. Every second consumers spend negotiating assignment is a second the producer’s LEO advances while committed offsets stay frozen.
To quantify lag accurately, inspect it per partition. The kafka-consumer-groups tool gives the most direct view of group state:
kafka-consumer-groups --bootstrap-server kafka-broker-1:9092 \
--describe \
--group order-processing-serviceThe output has one row per partition with columns TOPIC, PARTITION, CURRENT-OFFSET, LOG-END-OFFSET, LAG, CONSUMER-ID, HOST, and CLIENT-ID. A healthy system keeps LAG oscillating within a predictable band. A partition where LAG increases monotonically signals a consumer that cannot keep pace. When you see that pattern across many partitions in the same group, the bottleneck is systemic rather than partition-specific. Note that if a partition has no active member, the CONSUMER-ID shows - and LAG is still computed from the last committed offset — useful for spotting a group that has stopped entirely.
Instrumenting Lag for Real-Time Visibility
Threshold alerting on absolute lag is brittle: it produces false positives during routine bursts and false negatives during slow, creeping degradation. A robust strategy layers several signals — the raw lag value, the rate of change of lag over a rolling window, and an estimated time-to-clear derived from current consumer throughput.
It helps to know exactly where these numbers live, because lag is exposed differently on the broker and on the client. Kafka brokers expose kafka.server:type=BrokerTopicMetrics,name=MessagesInPerSec (with an optional topic= tag) for production rates, but consumer group lag is not a broker-side JMX metric. To track lag programmatically you have two real options: read committed offsets from __consumer_offsets and compare them against each partition’s LEO (which is what kafka-consumer-groups and most external tools do under the hood), or scrape the client-side consumer metrics. The consumer reports lag under the MBean kafka.consumer:type=consumer-fetch-manager-metrics,client-id="{client-id}", whose records-lag-max attribute is the windowed maximum lag across all of that consumer’s assigned partitions. Per-partition lag is available on the same MBean type with topic and partition tags via the records-lag, records-lag-avg, and records-lag-max attributes. The critical caveat: these are emitted by the consumer process itself, so if a consumer is down, you get no lag reading from it at all — which is exactly when you most need one. That blind spot is why offset-based external tools remain the operational standard.
For teams that prefer a dashboard-driven view, Visualizing Consumer Lag with Confluent Control Center provides a managed interface that plots lag trends and lets you drill into individual partitions — useful during triage when you need to correlate a lag spike with a deployment or an upstream traffic anomaly.
For self-managed stacks, LinkedIn’s open-source Burrow is a common choice for lag evaluation, and it works differently from a threshold alarm — this is the whole point of using it. Burrow consumes __consumer_offsets, maintains a sliding window of recent commits per partition for each group, and from that window decides whether the consumer is making progress. It emits a status of OK, WARN, STOP, STALL, REWIND, or ERR per partition and group based on whether offsets are advancing relative to the incoming offset range, not on a fixed lag number. That is the key insight: a group sitting at 50,000 lag that is steadily draining is reported healthy, while a group at 500 lag that has stopped committing is flagged. Because Burrow is intentionally threshold-free, do not look for warning/error lag thresholds in its config — they do not exist. Its TOML configuration is organized into sections such as [zookeeper], [client-profile.NAME], [cluster.NAME], [consumer.NAME], [evaluator.NAME], and [notifier.NAME]. A minimal cluster-plus-consumer definition looks like this:
[client-profile.prod]
client-id="burrow-lagcheck"
kafka-version="2.8.0"
[cluster.production-cluster]
class-name="kafka"
servers=["kafka-broker-1:9092", "kafka-broker-2:9092"]
client-profile="prod"
topic-refresh=60
offset-refresh=30
[consumer.production-cluster]
class-name="kafka"
cluster="production-cluster"
servers=["kafka-broker-1:9092", "kafka-broker-2:9092"]
client-profile="prod"
start-latest=true
[evaluator.default]
class-name="caching"
expire-cache=600The companion guide Alerting on Consumer Lag with Burrow and Grafana walks through wiring Burrow’s HTTP endpoint into a metrics pipeline and dashboards. For the exact sliding-window evaluation logic, see Burrow’s Consumer Lag Evaluation Rules.
Diagnosing the Root Cause of a Backlog
When a P1 alert fires and the lag graph is trending vertically, a structured diagnostic prevents flailing. The first question: is the backlog producer-driven or consumer-driven?
Start with producer-side rate. If MessagesInPerSec on the affected topic has doubled or tripled, your consumers may simply be facing a legitimate surge, and the fix is to scale consumer capacity rather than tune existing instances. You can sample the broker metric directly with the JMX tool that ships with Kafka:
kafka-run-class kafka.tools.JmxTool \
--object-name "kafka.server:type=BrokerTopicMetrics,name=MessagesInPerSec,topic=orders" \
--jmx-url service:jmx:rmi:///jndi/rmi://kafka-broker-1:9999/jmxrmi \
--reporting-interval 1000The object name is quoted because it contains shell-significant characters. Note that the class moved: kafka.tools.JmxTool is deprecated as of Kafka 3.5 and now delegates to org.apache.kafka.tools.JmxTool (also reachable via the kafka-jmx wrapper script). The old invocation above still works on current releases but prints a deprecation warning; use the new class name if you are on 3.5+.
If the producer rate is stable, shift focus to the consumer. The most common consumer-side bottleneck is slow processing inside the poll loop — every millisecond spent on a record is a millisecond not spent fetching. Profile the application to classify the bottleneck as CPU-bound (heavy deserialization or transformation), I/O-bound (synchronous writes to a database), or network-bound (high-latency downstream calls).
A quick diagnostic is the ratio of fetch-rate to records-consumed-rate, both on the consumer-side consumer-fetch-manager-metrics MBean. If fetches are frequent but the consumed rate is low, the consumer is pulling data faster than it can process it. You can read these from the running consumer rather than through JMX:
// Consumer metrics are available via KafkaConsumer.metrics().
// Look up by metric name rather than constructing a MetricName with tags,
// since the MetricName key includes the full client-id tag set.
for (Map.Entry<MetricName, ? extends Metric> e : consumer.metrics().entrySet()) {
MetricName name = e.getKey();
if (!"consumer-fetch-manager-metrics".equals(name.group())) continue;
if ("fetch-rate".equals(name.name()) || "records-consumed-rate".equals(name.name())) {
System.out.printf("%s = %.2f%n", name.name(), (double) e.getValue().metricValue());
}
}Another subtle cause of slowdown is a max.poll.records / max.partition.fetch.bytes combination that hands the consumer more data than it can process before max.poll.interval.ms elapses. When the application fails to call poll() again within that interval, the consumer proactively leaves the group, triggering a rebalance that pauses consumption for the affected members; the cycle repeats and produces a sawtooth lag pattern. The fix is to reduce the per-poll batch (max.poll.records) or raise max.poll.interval.ms, but the durable solution is to cut processing time per record.
Immediate Remediation Tactics for Active Incidents
When lag is actively growing and downstream systems are starved, you need levers that take effect in seconds to minutes. Restore equilibrium first, even with a temporary fix, then follow up with permanent corrections.
The fastest lever is horizontal scale: add consumer instances to the group. Because each consumer owns a subset of partitions, added consumers raise aggregate throughput — up to the partition count of the subscribed topics. A topic with 32 partitions caps useful parallelism at 32 consumers; an 33rd sits idle. If your group has 8 instances against 32 partitions, scaling to 32 quadruples capacity. This is the single most effective immediate action for a producer-driven backlog.
Scaling does trigger a rebalance, and how disruptive that is depends entirely on your assignment strategy. Incremental cooperative rebalancing lets consumers keep processing the partitions they retain instead of revoking everything, which sharply reduces the stop-the-world window. It is provided by CooperativeStickyAssignor (available since Kafka 2.4). Be precise about the default, because this is a common misconception: as of Kafka 3.0 the default partition.assignment.strategy is [RangeAssignor, CooperativeStickyAssignor], and a group started with that default still uses the eager RangeAssignor — the cooperative protocol does not activate until RangeAssignor is removed from the list. The intended upgrade is a two-step rolling bounce: first deploy with both assignors, then deploy again with only CooperativeStickyAssignor. To actually get cooperative rebalancing, set:
# Consumer configuration — step 2 of the rolling upgrade,
# after every member already advertises the cooperative assignor.
partition.assignment.strategy=org.apache.kafka.clients.consumer.CooperativeStickyAssignorSkipping the two-step sequence and jumping straight to the cooperative-only config across a group that previously used eager assignment can corrupt assignment state, so roll it out in order.
If you cannot scale out because you are already at the partition count, raise per-consumer throughput. A consumer tripping max.poll.interval.ms benefits from a smaller batch, which completes the poll loop sooner and avoids self-eviction:
# Smaller batches complete the poll loop faster and avoid eviction.
max.poll.records=250The trade-off is more frequent fetch requests and slightly higher overhead. Treat this as a stabilizer, not a throughput win — it stops the rebalance sawtooth, but the real throughput fix is faster per-record processing.
For I/O-bound consumers, offload work to a thread pool so the poll loop stays fast: deserialize, hand records to a bounded queue, and poll again. Worker threads drain the queue and commit only after successful processing. This decouples fetching from processing:
ExecutorService workers = Executors.newFixedThreadPool(16);
BlockingQueue<ConsumerRecord<String, String>> queue = new LinkedBlockingQueue<>(1000);
// Poll thread: keep the loop fast; the bounded queue applies backpressure.
while (running) {
ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100));
for (ConsumerRecord<String, String> record : records) {
queue.put(record); // blocks when the queue is full
}
}
// Worker threads: process, then commit.
for (int i = 0; i < 16; i++) {
workers.submit(() -> {
while (running) {
ConsumerRecord<String, String> record = queue.take();
processRecord(record);
// KafkaConsumer is NOT thread-safe: commits must be funneled back
// to the poll thread, and offsets committed in order per partition.
}
});
}Two hazards make this pattern dangerous if implemented naively. First, KafkaConsumer is not thread-safe — only the poll thread may call commitSync/commitAsync and poll, so worker threads must signal completed offsets back to the poll thread rather than committing themselves. Second, committing offsets out of order (e.g., a fast worker commits offset 110 while offset 105 is still in flight) silently advances the committed position past unprocessed records, so on restart those records are skipped. Track the highest contiguously completed offset per partition and commit that. Because this is at-least-once delivery, a crash between processing and commit can reprocess records; if downstream systems are not idempotent, add deduplication. The idempotency patterns are covered in Idempotent Producers and Exactly-Once Semantics, which, while producer-focused, establishes the guarantees consumers must respect.
Permanent Architectural Fixes for Recurring Backlogs
Incident response buys time; architecture prevents the next incident. A group that repeatedly falls behind under predictable load needs a redesigned processing topology.
The most effective pattern for sustained throughput is to decouple ingestion from processing with an intermediate topic. Rather than have one group read raw events and write directly to a slow sink, split the pipeline: a lightweight stage reads the source topic, does minimal transformation, and produces to an intermediate topic with a higher partition count; one or more heavy-processing stages then read the intermediate topic. This “fan-out” lets you scale the expensive stage independently and tune each stage’s consumer configuration separately.
The intermediate topic also acts as a buffer that absorbs producer bursts: if the heavy stage falls behind, the intermediate topic accumulates data without back-pressuring producers (assuming retention is sized for it). The topology, implementable as Kafka Streams or plain producer/consumer code:
Source Topic (32 partitions)
|
v
Stage 1 Consumer Group (32 instances, lightweight transform)
|
v
Intermediate Topic (64 partitions, retention=7 days)
|
v
Stage 2 Consumer Group A (heavy processing, 64 instances)
Stage 2 Consumer Group B (real-time analytics, 32 instances)
A second permanent fix is to attack slow processing at the source. For a consumer that is I/O-bound on a database, batch the writes. Instead of one statement per record, accumulate records and flush in batches — most databases offer a batch API that is far faster than single-row inserts:
PreparedStatement stmt = connection.prepareStatement(
"INSERT INTO orders (id, amount, timestamp) VALUES (?, ?, ?)"
);
int batchSize = 0;
for (ConsumerRecord<String, Order> record : records) {
Order order = record.value();
stmt.setString(1, order.getId());
stmt.setBigDecimal(2, order.getAmount());
stmt.setTimestamp(3, Timestamp.from(order.getTimestamp()));
stmt.addBatch();
if (++batchSize >= 500) {
stmt.executeBatch();
batchSize = 0;
}
}
if (batchSize > 0) {
stmt.executeBatch();
}Commit Kafka offsets only after the batch (and its transaction) succeeds, so a failed flush does not advance the committed position past unwritten records. For CPU-bound consumers, profile the processing path for the usual culprits: repeated deserialization of the same payload, excessive allocation, and cryptographic work in the hot loop. Caching parsed schemas — as the Confluent Schema Registry client does for Avro — avoids re-parsing on every record and is often a large, cheap win.
Tuning Kafka Broker and Topic Configuration for Backlog Resilience
Consumer-side tuning addresses only half the equation. Broker and topic configuration shape how backlogs accumulate and how a lagging consumer interacts with retention. A misconfigured retention policy can turn a transient backlog into real data loss.
The configuration most directly tied to backlog safety is retention.bytes. When a partition reaches its size limit, the broker deletes its oldest segments regardless of whether any consumer has read them — retention is enforced on time and size, never on consumer progress. If a lagging consumer’s committed offset points into a deleted segment, the next fetch throws an out-of-range error, and the consumer’s auto.offset.reset decides what happens: latest jumps to the end and silently skips the gap (data loss), earliest reprocesses from the oldest retained record, and none raises an exception. Size retention so it comfortably exceeds your worst-case backlog: a topic ingesting 100 MB/s with a 24-hour tolerable backlog needs at least 100 MB/s × 86,400 s ≈ 8.64 TB, before replication. Apply it per topic:
kafka-configs --bootstrap-server kafka-broker-1:9092 \
--entity-type topics --entity-name orders \
--alter --add-config retention.bytes=8640000000000Segment granularity matters too. Kafka reclaims space one log segment at a time, not per message, and the active segment is never deleted. A large segment.bytes (e.g., 1 GB) means disk stays occupied until an entire 1 GB segment ages out of retention, even after consumers have caught up. A smaller value lets the broker reclaim space more granularly, at the cost of more open file handles and more frequent segment rolls:
kafka-configs --bootstrap-server kafka-broker-1:9092 \
--entity-type topics --entity-name orders \
--alter --add-config segment.bytes=268435456On the broker, num.replica.fetchers sets how many fetcher threads each broker uses to replicate from leaders (default 1). If consumers are exposed to under-replicated or recovering partitions — for example during a leadership change while a broker catches up — raising this to 4 or 8 on brokers with spare CPU improves replication throughput and shortens the window of degraded availability. It is a replication tuning, not a direct consumer-lag fix, but it reduces the disruptions that indirectly stall consumers:
# broker config
num.replica.fetchers=4Finally, weigh compaction. On a compact topic the broker retains only the latest value per key, so a consumer that falls far behind may never observe intermediate values for a key — fatal for logic that depends on every state transition. If you need full history, use cleanup.policy=delete with time- or size-based retention rather than compact (or compact,delete if you need both behaviors).
Building a Backlog-Resistant Operational Culture
Technology alone does not prevent large backlogs; operational discipline is the final layer. The strongest teams treat backlog prevention as continuous engineering, not a reactive firefight.
Start by defining SLOs for lag that mean something to downstream consumers. “99.9% of messages are processed within 60 seconds of production” is more actionable than “lag must stay below 10,000,” because it maps to a measurable quantity: the wall-clock age of the oldest unprocessed message. You can alert when the 99.9th percentile of that age exceeds your target. Burrow’s progress-based evaluation complements this well, since it distinguishes a draining backlog from a stalled one.
Run regular backlog fire drills. Simulate a producer surge on a non-production topic and watch how your consumers, dashboards, and on-call engineers respond. Capture a playbook for scaling consumers, adjusting fetch parameters, and rolling back recent deployments that introduced regressions — with the exact commands, the order to run them, and the expected recovery time for each.
Invest in chaos experiments that target the group protocol directly. Kill a consumer instance to induce a rebalance and measure the stop-the-world duration. If it exceeds your SLO, address it with cooperative rebalancing and, where appropriate, by tuning session.timeout.ms, heartbeat.interval.ms, and max.poll.interval.ms so transient pauses do not evict healthy members. The official Apache Kafka consumer configuration reference is the definitive source for these parameters and their interactions.
Finally, build a feedback loop between producer and consumer teams. A producer change that doubles message rate without warning the consumer team is a recurring cause of incidents. A lightweight change process — notify consumer owners and confirm downstream capacity before a throughput- or size-affecting producer change ships — is a communication channel, not a heavyweight gate, and it prevents surprises.
The patterns here — partition-level lag inspection, progress-based monitoring, correct cooperative-rebalancing rollout, fan-out architectures, and retention sized to your worst case — form a layered defense against large backlogs. Combined with disciplined incident response and proactive capacity planning, they turn consumer lag from a recurring emergency into a routine operational signal.