Monitoring Consumer Lag with JMX and Prometheus

Consumer lag is the single most important operational metric for an Apache Kafka deployment. It is the delta between the latest produced offset (the log-end offset) and the latest committed offset for a given consumer group and partition. Sustained lag growth means consumers cannot keep pace with producers, which translates into stale data, delayed processing, and missed SLOs. This article shows how to instrument, expose, collect, and alert on consumer lag using JMX and Prometheus, and is written for platform and SRE engineers who need a production-grade pipeline rather than a quick demo.

A crucial point up front, because the rest of the article depends on it: Kafka does not expose a pre-computed “consumer lag” metric anywhere. The consumer client exposes its own view of lag (records-lag-max) as a client-side JMX metric. The broker exposes per-partition log-end offsets but never the committed offset of a group as a ready-made lag number. To get authoritative, per-group, per-partition lag you must either compute it from log-end offset minus committed offset yourself, or run a dedicated exporter that does exactly that by querying the cluster with the Admin API. Much of the confusion in Kafka monitoring comes from assuming a “broker-side lag MBean” exists. It does not.

For a refresher on Kafka’s core components, see Producers, Consumers & Delivery Semantics. For how group coordination affects lag during rebalances, see Consumer Group Rebalancing: Protocols and Tuning.

Understanding Consumer Lag and Its Operational Impact

Lag is defined per partition as the log-end offset minus the consumer group’s last committed offset for that partition. In a healthy system it stays near zero or within a defined SLO. Growth signals a consumption bottleneck: insufficient consumer parallelism, slow downstream processing, network saturation, or a producer throughput spike. Lag is never a single number — it is a time series that you must observe per partition and aggregate per group, because a single hot partition can be badly behind while the group average looks fine.

The operational consequences are direct. High lag delays event processing and erodes data freshness. In pipelines that rely on Idempotent Producers and Exactly-Once Semantics, a consumer that falls far enough behind can read across compaction or retention boundaries, or simply violate the freshness contract downstream systems assume. Lag also interacts with rebalancing: a consumer that pauses long enough to miss max.poll.interval.ms is evicted from the group, triggering a rebalance during which the whole group stops processing and lag climbs further.

Two sources of lag-related telemetry matter, and they live in different places:

  • Client-side, in the consumer JVM: kafka.consumer:type=consumer-fetch-manager-metrics,client-id=<client-id> exposes records-lag-max (and per-partition records-lag). This is the consumer’s own view, limited to the partitions currently assigned to that instance.
  • Cluster-wide, computed externally: log-end offset (from the broker) minus committed offset (from the __consumer_offsets topic, read via the Admin API). This is the authoritative, per-group, per-partition view, and it is what a dedicated lag exporter produces.

There is no third option where the broker hands you finished lag numbers.

Instrumenting Kafka Clients for JMX Exposure

The Kafka client libraries register their metrics as MBeans in the kafka.consumer and kafka.producer domains automatically; JMX is enabled at the JVM level, not via a Kafka config property. The minimal flags to open a JMX endpoint on a consumer application are:

-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.port=9999
-Dcom.sun.management.jmxremote.rmi.port=9999
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false
-Djava.rmi.server.hostname=<consumer-host-ip>

In production, enable authentication and TLS (com.sun.management.jmxremote.authenticate=true, com.sun.management.jmxremote.ssl=true) and supply password/access files — an unauthenticated RMI port is a remote-code-execution risk. The java.rmi.server.hostname flag is required when the process runs behind NAT or in a container so the RMI stub advertises a reachable address.

The key MBean for client-side lag is:

kafka.consumer:type=consumer-fetch-manager-metrics,client-id=<client-id>

with the attribute records-lag-max — the maximum lag, in records, across the partitions assigned to that consumer instance. There is also a per-partition variant, kafka.consumer:type=consumer-fetch-manager-metrics,client-id=<client-id>,topic=<topic>,partition=<partition>, exposing records-lag and records-lag-avg.

Set a meaningful, unique client.id per instance so MBean names (and the resulting Prometheus series) do not collide:

# Consumer configuration snippet
client.id=order-processor-${HOSTNAME}

You normally do not need to set metric.reporters. JmxReporter is registered by default in Kafka clients, so MBeans appear as soon as the JVM has a JMX endpoint. You would only add metric.reporters to plug in an additional reporter implementation; setting it to org.apache.kafka.common.metrics.JmxReporter alone is redundant.

A caveat worth internalizing: records-lag-max is computed only from the consumer’s last fetch responses, so it reflects lag for currently assigned partitions and only updates while the consumer is actively polling. A stalled or evicted consumer can show stale or zero lag precisely when you most want a real number. For alerting on whether a group is falling behind overall, prefer externally computed lag (next sections) and treat records-lag-max as a per-instance health signal.

Why There Is No “Broker-Side Lag” MBean

It is tempting to look for consumer-group metrics on the broker, but the coordinator-side metrics you actually want (consumer-coordinator-metrics, consumer-fetch-manager-metrics) are client-side: they live in the kafka.consumer domain inside each consumer’s JVM, not on the broker. The broker has no kafka.consumer domain and exposes no per-group lag.

What the broker does expose, useful for computing lag, is the log-end offset per partition it leads:

kafka.log:type=Log,name=LogEndOffset,topic=<topic>,partition=<partition>

Read via JMX this MBean’s attribute is Value. Note two gotchas: a broker reports LogEndOffset only for partitions for which it is the leader, so to cover an entire topic you must scrape every broker and union the results; and this gives you the numerator of lag only — you still need each group’s committed offset to finish the calculation.

The committed offsets live in the internal __consumer_offsets topic. The supported way to read them is the Admin API (AdminClient.listConsumerGroupOffsets), which is exactly what kafka-consumer-groups.sh --describe and dedicated exporters use. Do not attempt to parse __consumer_offsets directly; its binary format is internal and version-dependent. This is why, in practice, you run a lag exporter rather than wiring offset math together by hand.

Deploying the Prometheus JMX Exporter for Kafka

The Prometheus JMX Exporter is a Java agent that attaches to a JVM, scrapes MBeans, and serves them as Prometheus metrics over HTTP. It is the standard bridge from Kafka’s JMX surface to Prometheus and is normally run as an in-process Java agent.

Attach it to a consumer (or broker) JVM with:

-javaagent:/opt/prometheus/jmx_prometheus_javaagent-1.0.1.jar=8080:/opt/prometheus/kafka-consumer.yml

A version note that bites people on upgrade: in jmx_exporter 1.0.0 and later, metrics are served on /metrics, not on the root path / as in the 0.x line. Point your scrape config and any curl checks at http://<host>:8080/metrics.

A minimal config that maps the consumer’s records-lag-max to a Prometheus gauge:

lowercaseOutputName: true
lowercaseOutputLabelNames: true
rules:
  - pattern: 'kafka.consumer<type=consumer-fetch-manager-metrics, client-id=(.+)><>records-lag-max'
    name: kafka_consumer_records_lag_max
    labels:
      client_id: "$1"
    type: GAUGE

jmx_exporter regexes are unanchored, so a pattern matching records-lag would also capture records-lag-avg and records-lag-max. Match the exact attribute (records-lag-max) to avoid unintended series. To expose the broker’s per-partition log-end offset, scrape the kafka.log MBean on every broker:

lowercaseOutputName: true
lowercaseOutputLabelNames: true
rules:
  - pattern: 'kafka.log<type=Log, name=LogEndOffset, topic=(.+), partition=(.+)><>Value'
    name: kafka_log_end_offset
    labels:
      topic: "$1"
      partition: "$2"
    type: GAUGE

This still leaves you to subtract committed offsets in PromQL — and Prometheus has no clean way to join a per-group committed-offset series against a per-partition log-end series, because the JMX route never gives you committed offsets in the first place. For real per-group lag, use a dedicated exporter.

Dedicated lag exporter

A lag exporter runs alongside the cluster, periodically lists every consumer group’s committed offsets via the Admin API, fetches the matching log-end offsets, computes lag, and serves it to Prometheus. Two widely used options:

  • Kafka Lag Exporter (seglo) — note this project is archived/unmaintained (final release v0.8.2); still functional but evaluate before adopting for new systems.
  • Kafka Exporter (danielqsj) — an actively used alternative exposing kafka_consumergroup_lag and related series.

A Kubernetes Deployment for Kafka Lag Exporter:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: kafka-lag-exporter
spec:
  replicas: 1
  selector:
    matchLabels:
      app: kafka-lag-exporter
  template:
    metadata:
      labels:
        app: kafka-lag-exporter
    spec:
      containers:
        - name: exporter
          image: seglo/kafka-lag-exporter:0.8.2
          ports:
            - containerPort: 8000

Kafka Lag Exporter reads its cluster connection from an application.conf (typically mounted via ConfigMap) rather than a single bootstrap env var; consult its README for the exact configuration. It serves /metrics on port 8000 and exposes, among others, kafka_consumergroup_group_lag (lag in offsets) with labels including cluster_name, group, topic, partition, member_host, consumer_id, and client_id, plus kafka_consumergroup_group_lag_seconds (estimated time lag). The exact metric name matters: it is kafka_consumergroup_group_lag, not kafka_consumer_group_lag. Every recording rule, alert, and dashboard query below uses the real name.

Building Prometheus Recording Rules and Dashboards

With the exporter feeding Prometheus, add recording rules so dashboards and alerts query pre-aggregated series rather than recomputing on every evaluation.

groups:
  - name: kafka_consumer_lag
    rules:
      - record: kafka:consumergroup_lag:sum_by_group
        expr: sum by (cluster_name, group) (kafka_consumergroup_group_lag)
      - record: kafka:consumergroup_lag:sum_by_group_topic
        expr: sum by (cluster_name, group, topic) (kafka_consumergroup_group_lag)

For alerting, threshold on sustained lag rather than instantaneous spikes. This rule fires when any partition’s lag stays above 1000 for 10 minutes:

groups:
  - name: kafka_alerts
    rules:
      - alert: HighConsumerLag
        expr: kafka_consumergroup_group_lag > 1000
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "High lag for group {{ $labels.group }} on {{ $labels.topic }}/{{ $labels.partition }}"
          description: "Consumer lag is {{ $value }} records on partition {{ $labels.partition }}."

Absolute offset thresholds are blunt: 1000 records is trivial on a high-throughput topic and catastrophic on a low one. Where the exporter provides it, alert on kafka_consumergroup_group_lag_seconds (time lag) instead — a time-based SLO (“no group more than 5 minutes behind”) generalizes across topics far better than a record count.

For dashboards (Grafana is standard), focus on a small set of panels that answer real questions:

  1. Total lag per group over time — trend and direction.
  2. A per-partition heatmap — to spot skew where one partition dominates.
  3. Top-N partitions by lag — for fast drill-down during an incident.
  4. Consumer fetch rate and producer send rate overlaid — to attribute lag to a producer surge versus a consumer slowdown.

The total-lag-per-group time series:

sum by (group) (kafka_consumergroup_group_lag)

And the per-partition view that feeds a heatmap:

sum by (topic, partition) (kafka_consumergroup_group_lag)

Correlating Lag with Rebalancing and Producer Behavior

Lag is frequently a symptom, not a root cause. The two most common drivers are rebalances and producer surges.

Rebalance telemetry comes from the consumer-side coordinator metrics (again, in the consumer JVM, not the broker):

kafka.consumer:type=consumer-coordinator-metrics,client-id=<client-id>

Useful attributes include rebalance-rate-per-hour (rebalances participated in per hour), rebalance-latency-avg/rebalance-latency-max (time per rebalance), rebalance-total, and failed-rebalance-rate-per-hour. Scrape them with a JMX exporter rule analogous to the lag one, then overlay rebalance rate on your lag dashboards. If lag steps up exactly when rebalance rate spikes, you have a coordination problem, not a throughput one.

A JMX exporter rule for the rebalance rate:

rules:
  - pattern: 'kafka.consumer<type=consumer-coordinator-metrics, client-id=(.+)><>rebalance-rate-per-hour'
    name: kafka_consumer_rebalance_rate_per_hour
    labels:
      client_id: "$1"
    type: GAUGE

and a query to surface active rebalancing:

kafka_consumer_rebalance_rate_per_hour > 0

Common root causes for rebalance storms are processing time exceeding max.poll.interval.ms, heartbeat/session timeouts from GC pauses, or churning group membership. Tuning these is covered in Consumer Group Rebalancing: Protocols and Tuning.

Producer behavior is the other lever. Producer throughput is reported client-side at kafka.producer:type=producer-metrics,client-id=<client-id> via the record-send-rate attribute. Comparing aggregate produce rate against consume rate tells you whether producers are simply outrunning consumers:

sum(rate(kafka_producer_record_send_total[5m]))
  /
sum(rate(kafka_consumer_records_consumed_total[5m]))

(Use the cumulative *-total counters with rate() rather than the pre-averaged *-rate gauges, which Kafka already smooths over a window and which do not compose cleanly under rate().) When this ratio sits above 1 for a sustained period, producers are winning and lag will grow — your options are scaling out consumers, speeding up downstream processing, or throttling producers.

Tiered Alerting and Operational Playbooks

Threshold-on-a-single-number alerting produces both noise and blind spots. A tiered scheme with distinct severities and a rate-of-change signal works better in production:

  • Warning — lag above a moderate level for several minutes; notify the channel, do not page.
  • Critical — lag high for a sustained window, or lag increasing faster than the consumer can plausibly recover; page on-call.
  • Emergency — lag extreme, or the group has stopped committing entirely; trigger automated remediation.
groups:
  - name: kafka_lag_tiered
    rules:
      - alert: ConsumerLagWarning
        expr: max by (group, topic) (kafka_consumergroup_group_lag) > 500
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Consumer lag warning for group {{ $labels.group }}"
      - alert: ConsumerLagCritical
        expr: max by (group, topic) (kafka_consumergroup_group_lag) > 5000
        for: 10m
        labels:
          severity: critical
        annotations:
          summary: "Consumer lag critical for group {{ $labels.group }}"
      - alert: ConsumerLagStalled
        # Group present but not advancing: lag high and offset unchanged for 10m.
        expr: |
          max by (group, topic) (kafka_consumergroup_group_lag) > 1000
          and
          max by (group, topic) (delta(kafka_consumergroup_group_offset[10m])) == 0
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Consumer group {{ $labels.group }} stalled (lag not decreasing)"

A note on absent(): do not alert on absent(kafka_consumergroup_group_lag) as an “emergency” — that series disappears when the exporter or scrape breaks, so it conflates a monitoring outage with a consumer outage. Alert on scrape health (up == 0 for the exporter target) separately, and use a stalled condition (offset not advancing while lag is high, as above) to catch a consumer that has stopped committing.

Automated remediation should be conservative and idempotent. A common pattern is an Alertmanager webhook that triggers a Kubernetes Job to scale a consumer Deployment. Note that this only helps when partitions outnumber consumer instances — adding replicas beyond the partition count yields idle consumers, not throughput.

apiVersion: batch/v1
kind: Job
metadata:
  name: scale-consumer
spec:
  template:
    spec:
      containers:
        - name: scaler
          image: bitnami/kubectl:latest
          command: ["/bin/sh", "-c"]
          args:
            - kubectl scale deployment "$CONSUMER_DEPLOYMENT" --replicas="$NEW_REPLICAS"
      restartPolicy: Never

The Job’s ServiceAccount needs RBAC permission to patch/update the target Deployment’s scale subresource.

Validating Your Monitoring Setup

After deployment, confirm metrics flow end to end before you rely on the alerts. Check that the JMX exporter endpoint is reachable and emitting the expected series (remember /metrics on jmx_exporter 1.0+):

curl -s http://<consumer-host>:8080/metrics | grep kafka_consumer_records_lag_max

And the lag exporter:

curl -s http://<lag-exporter-host>:8000/metrics | grep kafka_consumergroup_group_lag

Then exercise the pipeline by generating real lag — produce a burst faster than the consumer can drain it with kafka-producer-perf-test:

kafka-producer-perf-test \
  --topic test-topic \
  --num-records 100000 \
  --record-size 1000 \
  --throughput 100000 \
  --producer-props bootstrap.servers=kafka-broker:9092

(--throughput -1 disables throttling entirely if you want maximum pressure.) While it runs, watch lag climb in Prometheus and confirm the warning/critical alerts move through Alertmanager. You can also verify ground truth independently with kafka-consumer-groups.sh --bootstrap-server kafka-broker:9092 --describe --group <group>, whose LAG column is computed the same way (log-end minus committed) and should track your exporter’s numbers.

For multi-cluster setups, federate or remote-write the per-cluster Prometheus into a central store; see the Prometheus Federation documentation. For long-term retention and cross-cluster trend analysis, Thanos or Cortex/Mimir extend storage beyond a single Prometheus’s local retention.

Consumer lag monitoring is not set-and-forget: thresholds drift as topic throughput changes, time-based SLOs age, and remediation playbooks need rehearsal. But with the client-side records-lag-max signal, an externally computed per-group kafka_consumergroup_group_lag, and correlation against rebalance and producer rates, you have an accurate, actionable view of one of Kafka’s most common failure modes — built on metrics that actually exist.

Back to Producers Consumers Delivery Semantics