Monitoring Kafka with JMX, Prometheus, and Grafana

Effective observability is the bedrock of running Apache Kafka in production. Without visibility into broker, producer, consumer, and cluster-level metrics, your team is flying blind—unable to detect degradation, plan capacity, or troubleshoot failures before they impact downstream services. This guide builds a production monitoring stack using Java Management Extensions (JMX) as the metric source, Prometheus for collection and storage, and Grafana for visualization and alerting. It is part of our broader Operations, Security & Observability framework.

We will enable and expose JMX metrics, configure Prometheus to scrape them efficiently, and build dashboards that surface the signals that matter. We also address the common traps: metric cardinality explosion, securing an exporter endpoint in environments that already enforce Securing Kafka with mTLS and SASL/SCRAM, and correlating observability data with authorization events governed by Kafka ACLs and Authorization Best Practices.

Understanding the JMX-Prometheus-Grafana Pipeline

Kafka exposes hundreds of metrics via JMX—request rates, network throughput, log segment sizes, purgatory sizes, controller state, and more. The pipeline has three stages:

  1. Metric emission: Each Kafka broker and client (producer, consumer, Connect worker, Streams instance) runs a JVM with an MBean server. Kafka registers its metrics as MBeans.
  2. Collection: The Prometheus JMX Exporter, loaded as a Java agent inside the JVM, reads MBeans and exposes them as Prometheus-format text over an HTTP endpoint.
  3. Storage, query, and visualization: Prometheus scrapes the exporter endpoints and stores the time series; Grafana queries Prometheus to render dashboards; Alertmanager fires alerts.

The exporter is the bridge: it translates the JVM-native MBean model into Prometheus’s pull-based, dimensional data model. Crucially, you do not need to open JMX/RMI ports to the network—the agent runs inside the broker process and exposes a plain HTTP endpoint that you can place behind TLS and network controls.

flowchart LR MBean["MBean server (broker/client JVM)"] --> Agent["JMX Exporter (in-process Java agent)"] Agent --> HTTP["HTTP /metrics endpoint"] HTTP --> Prom["Prometheus (scrape + store)"] Prom --> Grafana["Grafana (dashboards)"] Prom --> AM["Alertmanager (alerts)"]

Why Not Direct JMX/RMI Scraping?

The JMX/RMI protocol negotiates a second, often random, ephemeral port after the initial connection, which makes it hostile to firewalls, NAT, and container networking. Securing it with TLS and authentication is possible but awkward. The JMX Exporter sidesteps all of this by running in-process and exposing a single predictable HTTP port that fits Prometheus’s native pull model.

Enabling and Configuring the JMX Exporter

The Prometheus JMX Exporter runs as a Java agent loaded at JVM startup. You configure it with a YAML file that selects which MBeans to scrape and how to map them to Prometheus metric and label names. Kafka ships sample configurations under config/, but you will want to tailor one for production.

Step 1: Download the JMX Exporter Java Agent

Download the jmx_prometheus_javaagent JAR from the Prometheus JMX Exporter releases page or Maven Central. Since the 1.0 release the artifact lives under the io.prometheus.jmx group ID; 1.3.0 is the current stable release at the time of writing.

wget https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/1.3.0/jmx_prometheus_javaagent-1.3.0.jar \
  -P /opt/prometheus/jmx_exporter/

The rules/pattern configuration syntax used below is backward compatible across the 0.x and 1.x lines, so existing configs continue to work after the upgrade.

Step 2: Create the Exporter Configuration

Below is a broker configuration that captures the essential health metrics while avoiding high-cardinality traps such as per-partition series unless you explicitly need them. Note that the exporter sanitizes every emitted metric and label name: any character outside [a-zA-Z0-9:_] (including the - in attribute names) is replaced with an underscore, and runs of underscores are collapsed.

# /opt/prometheus/jmx_exporter/kafka-broker.yml
startDelaySeconds: 0
lowercaseOutputName: true
lowercaseOutputLabelNames: true
rules:
  # Broker throughput counters
  - pattern: kafka.server<type=BrokerTopicMetrics, name=(BytesInPerSec|BytesOutPerSec|MessagesInPerSec)><>Count
    name: kafka_server_brokertopicmetrics_$1_total
    type: COUNTER

  # Request handler idle ratio
  - pattern: kafka.server<type=KafkaRequestHandlerPool, name=RequestHandlerAvgIdlePercent><>OneMinuteRate
    name: kafka_server_requesthandler_avgidlepercent
    type: GAUGE

  # Network processor idle ratio
  - pattern: kafka.network<type=SocketServer, name=NetworkProcessorAvgIdlePercent><>Value
    name: kafka_network_processor_avgidlepercent
    type: GAUGE

  # Under-replicated partitions
  - pattern: kafka.server<type=ReplicaManager, name=UnderReplicatedPartitions><>Value
    name: kafka_server_replicamanager_underreplicatedpartitions
    type: GAUGE

  # Active controller count
  - pattern: kafka.controller<type=KafkaController, name=ActiveControllerCount><>Value
    name: kafka_controller_activecontrollercount
    type: GAUGE

  # Offline partitions
  - pattern: kafka.controller<type=KafkaController, name=OfflinePartitionsCount><>Value
    name: kafka_controller_offlinepartitionscount
    type: GAUGE

RequestHandlerAvgIdlePercent is a Yammer meter; reading its OneMinuteRate attribute gives a stable idle ratio. (MeanRate is also available but covers the whole process lifetime and is less useful for alerting.)

This config deliberately omits per-partition and per-topic dimensions. In a cluster with thousands of partitions, per-partition metrics multiply your active series count and storage requirements fast—see the cardinality section below before enabling them.

Step 3: Attach the Agent to the Broker JVM

Set KAFKA_OPTS (or edit the broker start script) to add the Java agent flag. The agent argument is JAR=PORT:CONFIG; here it serves on port 7071.

export KAFKA_OPTS="-javaagent:/opt/prometheus/jmx_exporter/jmx_prometheus_javaagent-1.3.0.jar=7071:/opt/prometheus/jmx_exporter/kafka-broker.yml"

When running Kafka in containers, set this in the container definition. On Kubernetes, operators such as Strimzi expose JMX-exporter configuration directly through their custom resources (for example the metricsConfig/JmxPrometheusExporterMetrics settings on the Kafka resource), so you supply the rules as a ConfigMap rather than editing KAFKA_OPTS by hand.

Securing the Exporter Endpoint

The JMX Exporter’s HTTP endpoint supports basic authentication and TLS natively when you supply an httpServer configuration, but many teams prefer to terminate TLS and enforce network policy at a reverse proxy (Nginx, Envoy) or a service mesh (Istio mTLS). This matters if your cluster already uses Securing Kafka with mTLS and SASL/SCRAM—an unauthenticated metrics port would undercut that hardening.

A minimal Nginx front for the exporter:

server {
    listen 7072 ssl;
    ssl_certificate     /etc/nginx/certs/metrics.crt;
    ssl_certificate_key /etc/nginx/certs/metrics.key;

    location /metrics {
        allow 10.0.0.0/8;   # internal monitoring subnet only
        deny  all;
        proxy_pass http://127.0.0.1:7071;
    }
}

(allow/deny must precede proxy_pass within the location for the access rules to be evaluated.)

Configuring Prometheus for Kafka Scraping

With exporter endpoints running on every broker, configure Prometheus to discover and scrape them. The config defines scrape targets, intervals, and relabeling rules that attach cluster metadata.

Static vs. Dynamic Discovery

For small static clusters, a static_configs block is enough. For larger or dynamic environments use Prometheus service discovery—DNS, Kubernetes, or Consul—so brokers are picked up automatically as they scale.

The fragment below uses file-based service discovery, which fits well when broker lists are managed by configuration management:

# prometheus.yml
scrape_configs:
  - job_name: 'kafka-brokers'
    scrape_interval: 30s
    scrape_timeout: 10s
    metrics_path: '/metrics'
    scheme: 'https'
    file_sd_configs:
      - files:
          - '/etc/prometheus/targets/kafka-brokers.json'
    relabel_configs:
      - source_labels: [__address__]
        target_label: instance
        regex: '(.+):\d+'
        replacement: '${1}'
      - source_labels: [__meta_filepath]
        target_label: cluster
        replacement: 'prod-us-east'

The matching target file /etc/prometheus/targets/kafka-brokers.json:

[
  {"targets": ["broker1.kafka.internal:7072"], "labels": {"broker_id": "1"}},
  {"targets": ["broker2.kafka.internal:7072"], "labels": {"broker_id": "2"}},
  {"targets": ["broker3.kafka.internal:7072"], "labels": {"broker_id": "3"}}
]

__meta_filepath is a real meta-label exposed by file_sd_configs; the relabel rule above hard-codes a cluster name, but you can also derive it from the file path with a capture-group regex if you split targets per cluster.

Scrape Interval Considerations

A 30-second scrape interval is a sound default for Kafka. Scraping faster than 15s rarely helps: Kafka’s rate metrics are exponentially-weighted moving averages (Yammer meters), so higher-frequency sampling mostly adds load on the exporter and Prometheus without revealing new signal. Reserve sub-15s intervals for specific low-latency alerting needs.

Cardinality Management

The biggest operational hazard in Kafka monitoring is cardinality explosion. Kafka can expose per-topic and per-partition metrics that multiply with partition count. A cluster with 10,000 partitions and 50 per-partition series produces 500,000 unique time series; a single Prometheus instance starts to struggle in the low millions of active series.

Mitigations:

  • Drop high-cardinality metrics at the exporter: only pattern-match the metrics you actually need, and omit topic/partition capture groups where aggregate views suffice.
  • Drop labels with Prometheus relabeling: use metric_relabel_configs with labeldrop to strip dimensions you don’t query.
  • Aggregate at query time: prefer sum by (topic) (...) over storing per-partition series.
  • Federate hierarchically: run a Prometheus per datacenter scraping brokers, then federate aggregated series up to a global instance, or offload long-term storage to Thanos/Mimir.

Building Effective Grafana Dashboards

Once Prometheus is collecting, Grafana provides the visualization layer. Rather than starting from scratch, begin with a community dashboard and adapt it. The Building a Kafka Monitoring Dashboard with Prometheus and Grafana guide walks through construction in detail; here we cover the essential panels and the PromQL behind them. (The metric names below assume the exporter rules from Step 2.)

Essential Dashboard Panels

1. Throughput and traffic

# Cluster-wide bytes-in rate
sum(rate(kafka_server_brokertopicmetrics_bytesinpersec_total[5m]))

# Cluster-wide messages-in rate
sum(rate(kafka_server_brokertopicmetrics_messagesinpersec_total[5m]))

2. Broker health

# Request handler idle ratio across brokers (lower = busier; watch < 0.2)
avg(kafka_server_requesthandler_avgidlepercent)

# Network processor idle ratio
avg(kafka_network_processor_avgidlepercent)

RequestHandlerAvgIdlePercent and NetworkProcessorAvgIdlePercent are ratios in [0, 1]. A sustained drop toward 0 means the request-handler or network threads are saturated. Pair these with JVM heap and GC pause panels, which you can collect from the same exporter via the default java.lang:* MBeans.

3. Replication and durability

# Under-replicated partitions — alert if > 0
kafka_server_replicamanager_underreplicatedpartitions

# Offline partitions — alert if > 0
kafka_controller_offlinepartitionscount

# Active controllers — must sum to exactly 1 across the cluster
sum(kafka_controller_activecontrollercount)

ActiveControllerCount is reported per broker as 0 or 1; exactly one broker should report 1, so alert on the cluster-wide sum, not on any single instance.

4. Consumer lag

Consumer lag is a client-side metric, emitted by the consumer JVM (see the next section), not by the broker. The consumer fetch manager reports it:

# Max records lag across a consumer's partitions
max by (client_id) (kafka_consumer_fetch_manager_records_lag_max)

For most teams, lag is better tracked with a dedicated exporter such as Kafka’s own consumer-group offset tooling or kafka-lag-exporter, which compute committed-offset-to-end-offset lag from the broker side and do not require instrumenting every consumer.

Alerting Rules

Define alerting rules in a file referenced by rule_files in prometheus.yml:

groups:
  - name: kafka_alerts
    rules:
      - alert: KafkaUnderReplicatedPartitions
        expr: kafka_server_replicamanager_underreplicatedpartitions > 0
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Kafka under-replicated partitions detected"
          description: "Broker {{ $labels.instance }} reports {{ $value }} under-replicated partitions."

      - alert: KafkaActiveControllerNotOne
        expr: sum(kafka_controller_activecontrollercount) != 1
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Kafka active controller count is not 1"
          description: "Cluster reports {{ $value }} active controllers. Expected exactly 1."

      - alert: KafkaRequestHandlerSaturation
        expr: avg(kafka_server_requesthandler_avgidlepercent) < 0.2
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "Kafka request-handler pool saturation"
          description: "Average idle ratio is {{ $value }}, indicating request-handler thread pressure."

      - alert: KafkaConsumerLagHigh
        expr: max by (client_id) (kafka_consumer_fetch_manager_records_lag_max) > 10000
        for: 15m
        labels:
          severity: warning
        annotations:
          summary: "High consumer lag detected"
          description: "Consumer {{ $labels.client_id }} has {{ $value }} records of lag."

Route alerts to your incident system—PagerDuty, Opsgenie, or Slack—through Alertmanager.

Monitoring Kafka Clients and Streams

Broker metrics tell only half the story. Producers, consumers, and Kafka Streams applications emit their own JMX metrics. The same agent approach applies: attach the exporter to your client JVMs and add scraping rules.

Producer Metrics

Producer-level metrics live under the producer-metrics MBean type, keyed by client-id:

  • record-send-rate, record-error-rate — throughput and error rates
  • request-latency-avg — average broker request latency
  • buffer-available-bytes — remaining buffer pool, an indicator of back-pressure
  • record-retry-rate — retry frequency, a signal of transient broker issues

An exporter rule for these (attribute hyphens are sanitized to underscores in the output name):

- pattern: kafka.producer<type=producer-metrics, client-id=(.+)><>(record-send-rate|record-error-rate|request-latency-avg|record-retry-rate|buffer-available-bytes)
  name: kafka_producer_$2
  labels:
    client_id: "$1"
  type: GAUGE

This emits, for example, kafka_producer_record_send_rate{client_id="..."}.

Consumer Metrics

Consumer metrics live under the consumer-fetch-manager-metrics and consumer-coordinator-metrics MBean types:

  • records-consumed-rate, bytes-consumed-rate — consumption throughput
  • records-lag-max — maximum lag across the consumer’s assigned partitions (client level); the per-partition records-lag, records-lag-avg, and records-lag-max attributes are also available under kafka.consumer:type=consumer-fetch-manager-metrics,partition=...,topic=...,client-id=...
  • fetch-rate, fetch-latency-avg — fetch behavior
  • rebalance-rate-per-hour / rebalance-total (consumer-coordinator-metrics) — rebalance frequency

Kafka Streams Metrics

Kafka Streams exposes metrics for thread activity, task processing, and state stores, grouped under MBean types such as stream-thread-metrics, stream-task-metrics, and stream-state-metrics. The official Kafka monitoring documentation lists them in full. Most carry a debug recording level, so set metrics.recording.level=DEBUG in the application to capture them. At a minimum, watch:

  • poll-ratio (stream-thread-metrics) — fraction of time the stream thread spent polling records; a high ratio means the thread has idle headroom.
  • put-latency-avg (stream-state-metrics) — average state-store put latency, your RocksDB write-performance signal.
  • dropped-records-rate / dropped-records-total (stream-task-metrics) — records dropped mid-topology, typically from late-arriving data or deserialization/processing errors. This replaced the older thread-level skipped-records metric, which was removed in Kafka 2.5 (KIP-444).

Operational Best Practices and Pitfalls

Metric Retention and Storage

Kafka monitoring produces a lot of samples. A 10-broker cluster with moderate cardinality can emit tens of thousands of samples per scrape. Plan Prometheus storage accordingly:

  • Set the local retention window with --storage.tsdb.retention.time (for example --storage.tsdb.retention.time=15d); push longer-term, downsampled history to Thanos or Mimir.
  • Watch Prometheus’s own prometheus_tsdb_head_series and keep it comfortably below the few-million range for a single instance before sharding or federating.

Exporter Security in Hardened Environments

If your organization mandates mTLS for all inter-service communication, the exporter’s HTTP endpoint is a potential gap. Terminate TLS at a reverse proxy (as above) or enable the exporter’s built-in TLS, and have Prometheus present a client certificate if the proxy enforces mutual TLS. This extends the model described in Kafka ACLs and Authorization Best Practices to the observability plane.

Avoiding the “Everything” Dashboard

A single dashboard that crams in every metric buries signal in noise. Build layered dashboards instead:

  • Overview: top-level health—throughput, under-replicated partitions, active controller, lag summary.
  • Broker detail: per-broker resource use, request queues, network throughput.
  • Topic/partition detail: drill-down for active investigations only.
  • Client view: producer and consumer metrics scoped to application teams.

Correlating Metrics with Logs and Traces

Metrics tell you what; logs tell you why. Ship broker logs (ideally structured) to a centralized store such as Elasticsearch or Loki, and include broker ID and topic in the log context so you can pivot from a metric anomaly to the relevant lines. For request-level visibility, instrument producers and consumers with OpenTelemetry to trace message flows end to end across your Operations, Security & Observability stack.

Testing Your Monitoring Setup

Before trusting the stack in production, exercise it in staging. Simulate broker failures, partition reassignments, and lag spikes, then verify that:

  • every expected metric appears in Prometheus with the right labels;
  • dashboards render without errors;
  • alerts fire within their for windows;
  • Alertmanager routes notifications to the correct channels.

A chaos-engineering pass with tools like Chaos Mesh or Litmus surfaces observability gaps before they become incidents.

Conclusion

Monitoring Kafka with JMX, Prometheus, and Grafana is a proven pattern. The JMX Exporter bridges Kafka’s JVM-native MBeans into Prometheus’s dimensional model, and Grafana turns those series into actionable dashboards and alerts. Get the high-value signals right—active controller, under-replicated and offline partitions, request-handler saturation, and consumer lag—secure the exporter endpoint, manage cardinality deliberately, and layer your dashboards.

Monitoring is not a one-time setup; it evolves with the cluster. Review dashboards and alerts with the team, prune unused series, and fold lessons from each incident back in. For a deeper dive into dashboard construction, see Building a Kafka Monitoring Dashboard with Prometheus and Grafana, and keep your observability practice aligned with the security and automation patterns across the Operations, Security & Observability pillar.

In this section

1 guide in this area.

Back to Operations Security Observability