Visualizing Consumer Lag with Confluent Control Center

Consumer lag is the delta between the latest offset produced to a partition (the log end offset) and the last offset committed by a consumer group. Sustained lag signals processing bottlenecks, stalled consumers, or infrastructure saturation. For platform engineers and SREs running production Kafka clusters, the question is rarely “is there lag?” but “which partition, and why?” — and Confluent Control Center (C3) earns its place by collapsing the distance between those two.

C3 presents lag in a consumer-centric hierarchy — group, topic, partition — and overlays production and consumption throughput on the same timeline. This guide is operational: where the lag data actually comes from, how to configure it, how to read the patterns without being misled by sampling artifacts, how to alert, and how to drive root-cause analysis. It assumes familiarity with Producers, Consumers & Delivery Semantics and builds on Handling Large Backlogs and Consumer Lag.

Where Control Center’s Lag Numbers Come From

Be precise about the data sources, because the source determines what you can and cannot trust on the dashboard. C3 fuses two independent feeds with different origins, cadences, and failure modes:

The emitter computes, per consumer-group/client/topic/partition, the difference between the last offset the broker stored and the last offset that group committed, then publishes it. It is disabled by default and must be turned on with confluent.consumer.lag.emitter.enabled=true. These are broker properties — they belong in the broker config, not in control-center.properties.

The two feeds are decoupled, and that decoupling is the single most common source of confusion: without interceptors, the throughput charts stay empty even though lag still renders from the emitter — and vice versa, lag can be blank while throughput is live. If half your dashboard is missing, identify which feed is dark before touching anything else.

Per partition, lag is simply:

Consumer Lag (offsets) = Log End Offset - Last Committed Offset

Control Center rolls per-partition values into group- and topic-level summaries. The consumer view exposes:

• Total lag — sum of lag across all partitions owned by a group.
• Max lag — the largest lag on any single partition (the number that exposes skew; watch this, not just the total).
• Consumption / production rate — messages per second, derived from interceptor reports, not from offset commits.

The practical consequence: lag granularity is bounded by the emitter interval. At the 60-second default you detect sustained backlogs reliably but will miss sub-minute spikes entirely — a 200 ms stall that recovers before the next sample never appears. Lower the interval for finer resolution at the cost of more broker work; for anything that must be observed in true real time, read the broker JMX directly rather than waiting on the emitter.

Enabling Lag Monitoring

Two configuration surfaces are involved: the brokers (lag emitter) and the Control Center process itself.

On each broker, enable the lag emitter:

# Broker properties (server.properties)
confluent.consumer.lag.emitter.enabled=true
confluent.consumer.lag.emitter.interval.ms=60000

On the Control Center host, a minimal control-center.properties. Note that the cluster C3 connects to uses the plain bootstrap.servers key, not a confluent.controlcenter.kafka.* key:

# Kafka cluster Control Center connects to
bootstrap.servers=broker1:9092,broker2:9092,broker3:9092

# REST endpoints C3 uses to reach the brokers (Metadata Service)
confluent.controlcenter.streams.cprest.url=http://broker1:8090,http://broker2:8090,http://broker3:8090

# Replication/partitioning for Control Center's own internal topics
confluent.controlcenter.internal.topics.replication=3
confluent.controlcenter.internal.topics.partitions=3
confluent.controlcenter.command.topic.replication=3
confluent.monitoring.interceptor.topic.replication=3
confluent.metrics.topic.replication=3

# Control Center web UI listener (default port 9021)
confluent.controlcenter.rest.listeners=http://0.0.0.0:9021

Set the internal-topic replication factor to match your real cluster (three for production, one for a single-broker dev box), or C3 will fail to create its topics and start.

Three things that reliably catch people:

• There is no C3-side “lag sampling interval.” Lag cadence is owned by the broker emitter (confluent.consumer.lag.emitter.interval.ms); throughput cadence is owned by the client interceptors (confluent.monitoring.interceptor.publishMs). Do not hunt control-center.properties for a single monitoring-interval knob — it does not exist.
• History depth is bounded by topic retention. By default C3 keeps roughly three days of monitoring/metrics data (on _confluent-monitoring and _confluent-metrics) and longer for its other internal topics. This is ordinary Kafka topic retention — adjust it with topic-level retention.ms. Your alert-baselining window (below) can only look back as far as this retention allows.
• Consumer monitoring is on by default. confluent.controlcenter.consumers.view.enable (default true) controls whether the consumers view is shown; it does not “enable” lag collection, and there is no confluent.controlcenter.consumer.group.enable property.

Reading Lag Patterns in the UI

The consumer view is hierarchical: cluster → group → topic → partition. The top-level table lists each group with total lag and consumption rate, colored (healthy / elevated / over threshold) by whatever triggers you have defined. Drilling into a group reveals per-topic breakdowns, where most investigations start.

Three shapes tell you most of what you need:

• Continuously rising — the consumer cannot keep pace with production. Throughput-bound.
• Flat at a high value — the consumer has stalled: committing nothing, or no active members. This is the dangerous one, because a stalled group looks stable, not alarming.
• Saw-tooth between samples — usually a sampling artifact, not a real signal. Because lag is sampled at the emitter interval, judge the trend across several intervals, not the wobble between two.

Partition-level views expose skew, and Max lag is the field that surfaces it. If one partition shows 50,000 messages of lag while the rest sit near zero, you have a hot partition — typically a high-volume topic keyed on a small set of values, so one or two keys land all the traffic on one partition. Remedies are to fix the producer’s partitioning (key choice or a custom partitioner) or to add partitions. Adding partitions only helps future records, changes key-to-partition mapping for default hashing, and forces a group rebalance — plan it deliberately, not mid-incident.

Overlaying throughput on the same timeline disambiguates cause in seconds:

• Production spike coincident with rising lag → the consumer is under-provisioned for the new load. Scale consumers or partitions.
• Production steady, lag growing → the problem is consumer-side: GC pauses, network timeouts, or slow downstream calls inside the poll loop.

Alerting with Control Center Triggers

Visualization is not alerting. Control Center’s triggers fire when a metric crosses a condition you define; you then attach one or more actions. For lag, create a trigger on a specific consumer group with a condition on its lag metric, and add a duration so a single transient sample does not page anyone. Actions include email and generic webhooks — forward to PagerDuty, Slack, or Prometheus Alertmanager via their inbound webhook endpoints to fold Kafka alerts into existing on-call routing.

Triggers are created and managed in the Control Center UI. C3’s internal API is not published as a stable external contract, so configure through the UI rather than scripting against undocumented endpoints that can change between releases.

One practice matters more than the threshold value itself: pair every lag-value trigger with a consumer-presence trigger. A group that has stopped committing can show static, below-threshold lag while being effectively dead — the most common way a real outage hides behind a green dashboard. Alert on “no active members” or “no commit progress within N minutes” so a frozen consumer cannot pass as healthy.

Troubleshooting from Control Center Data

When a trigger fires, work from the partition distribution outward. The first question — answered immediately by the partition heatmap — is concentrated or uniform?, because the two point at opposite halves of the system.

Concentrated lag (one or a few partitions). The bottleneck is broker- or partition-side. Inspect the leader broker for the affected partition in C3’s broker view: request latency, I/O, queue depth. A slow or overloaded leader throttles every consumer assigned to its partitions. If the leader has drifted off its preferred replica after a restart, restore balance with a preferred-leader election. The kafka-leader-election tool (Kafka 2.4+) is the supported path:

kafka-leader-election --bootstrap-server broker1:9092 \
  --election-type preferred \
  --topic orders-topic \
  --partition 3

--election-type preferred acts only when the current leader is not the preferred replica; use --all-topic-partitions instead of --topic/--partition to rebalance the whole cluster at once.

Uniform lag (all partitions roughly equal). The bottleneck is consumer-side processing. The usual cause is heavy synchronous work inside the poll loop — a database write or external API call per record. Move that work off the poll thread, or tune the batch. max.poll.records caps records per poll(); fetch.min.bytes / fetch.max.wait.ms shape how much each fetch returns. The trap: raising max.poll.records without also raising max.poll.interval.ms. If a larger batch takes longer to process than max.poll.interval.ms, the consumer is evicted and the group rebalances — which is the next cause.

Repeated rebalancing. Every rebalance pauses consumption for the whole group, so a flapping group produces uniform lag spikes that recur on a regular beat. If lag spikes line up with rebalance events, review the interaction of session.timeout.ms, heartbeat.interval.ms, and max.poll.interval.ms; the Confluent consumer configuration reference documents how they combine.

Getting C3-Equivalent Lag into Prometheus and Grafana

Most shops want Kafka lag sitting next to application and infrastructure metrics. C3 bundles its own Prometheus (governed by confluent.controlcenter.prometheus.enable / confluent.controlcenter.prometheus.url, and disabled in management mode), but that instance is an internal C3 dependency, not a general-purpose exporter to scrape from your own Prometheus.

To pull the same lag signal C3 shows into your own Prometheus, scrape the broker JMX that feeds it. The emitter exposes the consumer-lag-offset MBean on each broker; export it with the Prometheus JMX Exporter (a Java agent on the brokers) and match it in your scrape config:

# jmx_exporter rules (broker side)
rules:
  - pattern: 'kafka.server<type=ConsumerLagEmitter, name=consumer-lag-offset, group=(.+), topic=(.+), partition=(.+)><>Value'
    name: kafka_consumer_lag_offset
    labels:
      group: "$1"
      topic: "$2"
      partition: "$3"
    type: GAUGE

# prometheus.yml
scrape_configs:
  - job_name: 'kafka-broker-jmx'
    scrape_interval: 30s
    static_configs:
      - targets: ['broker1:7071', 'broker2:7071', 'broker3:7071']

Confirm the exact pattern against the ObjectName your broker version publishes — open it in a JMX console such as jconsole before relying on the rule, because ObjectName layouts differ across Confluent Platform releases. A label or type mismatch yields a silently empty metric, not an error.

With the gauge in Prometheus you can alert on lag velocity, which catches problems earlier than any absolute ceiling:

sum by (group) (rate(kafka_consumer_lag_offset[5m])) > 100

This pages when a group’s lag is climbing faster than ~100 messages/second averaged over five minutes — an early signal before an absolute backlog threshold is breached. One caveat: scrape at 30 s but remember the underlying gauge only refreshes every 60 s by default, so do not set the rate() window below two emitter intervals or you will compute slope across stale samples.

For broker-health context, the Apache Kafka monitoring documentation defines the underlying metrics (UnderReplicatedPartitions, ActiveControllerCount, and others) whose degradation can indirectly drive consumer lag.

Operational Practices

Set per-group lag budgets, not one global threshold. An hourly batch ETL consumer can sit on millions of messages of lag between runs; a real-time fraud path cannot tolerate a few hundred. One global threshold either pages constantly for batch jobs or stays silent through a real outage. Encode the budget per group in the trigger.

Baseline against history before setting thresholds. Lag follows diurnal traffic, so a threshold tuned at 3 a.m. is usually wrong at 3 p.m. Review a week of data (bounded by your monitoring-topic retention) to set time-aware thresholds and to suppress alerts during known maintenance windows.

Correlate lag with producer behavior. A jump in production rate — visible in the topic throughput view — can swamp a consumer that was previously keeping up. Check whether the producer changed batch.size, linger.ms, or compression, or whether an upstream system started emitting larger payloads. The Producers, Consumers & Delivery Semantics pillar covers these throughput levers.

Treat persistent lag as an architecture signal. If a group accumulates lag despite adequate resources, the design — not the dashboard — is the problem. Parallelize across more partitions, push filtering and aggregation upstream, or replace bespoke consumer logic with a stream-processing topology in Kafka Streams or ksqlDB. Handling Large Backlogs and Consumer Lag lays out that decision framework.

Control Center’s value is not that it shows lag exists — kafka-consumer-groups --describe does that for free. It is the partition-, time-, throughput-, and broker-level context that tells you why lag exists, provided you keep one distinction straight: which signal comes from the broker emitter and which from the client interceptors, and never mistake one for the other.