Building a Kafka Monitoring Dashboard with Prometheus and Grafana

Effective observability is the bedrock of reliable Operations, Security & Observability for any Apache Kafka deployment. When a cluster handles millions of messages per second, you need to see broker health, consumer lag, and throughput in real time. This guide builds a Kafka monitoring dashboard with Prometheus and Grafana for platform, SRE, and backend engineers who need metrics they can act on.

We move past generic JMX scraping and focus on the operational signals that matter: request latency percentiles, under-replicated and offline partitions, active controller count, network and request-handler idle ratios, and consumer lag. By the end you will have a working dashboard layout, the exact PromQL behind each panel, and the gotchas that quietly break Kafka dashboards.

Understanding the Metrics Pipeline

Kafka exposes metrics via JMX (Java Management Extensions). Prometheus does not speak JMX, so you run an exporter — the Prometheus JMX Exporter as a Java agent inside each broker process. The agent reads JMX MBeans, maps them to Prometheus metric names, and serves them over an HTTP endpoint that Prometheus scrapes. Grafana then queries Prometheus to render the dashboards.

Run the exporter as an in-process Java agent rather than a standalone JMX-over-RMI scraper. The agent reads MBeans locally, which avoids the extra network hop and the JMX RMI port-negotiation problems that plague remote scraping.

A common pitfall is scraping every JMX bean. Kafka exposes thousands of MBeans, and matching them all produces high-cardinality series that bloat Prometheus storage and slow queries. Curate a focused rule set — the configuration below maps roughly a dozen broker MBeans, which is enough to answer the questions that page you at 3 a.m. For the foundational setup, see Monitoring Kafka with JMX, Prometheus, and Grafana.

Deploying the Prometheus JMX Exporter

Download the JMX Exporter Java agent JAR and attach it to each broker. The examples use jmx_prometheus_javaagent-0.20.0.jar (a stable 0.x release; the 1.x line works identically for agent attachment but changes some default metric naming).

Place the JAR somewhere readable by the Kafka process, for example /opt/prometheus/, then write the MBean-to-metric mapping.

Create /opt/prometheus/kafka-broker.yml:

startDelaySeconds: 0
lowercaseOutputName: true
lowercaseOutputLabelNames: true
rules:
  # Broker throughput counters (cumulative; use rate() in PromQL)
  - pattern: kafka.server<type=BrokerTopicMetrics, name=(BytesInPerSec|BytesOutPerSec|MessagesInPerSec)><>Count
    name: kafka_broker_$1_total
    type: COUNTER
  # Request handler pool idle fraction (0.0-1.0)
  - pattern: kafka.server<type=KafkaRequestHandlerPool, name=RequestHandlerAvgIdlePercent><>OneMinuteRate
    name: kafka_request_handler_idle_percent
    type: GAUGE
  # Per-network-processor idle fraction (0.0-1.0)
  - pattern: kafka.network<type=Processor, name=IdlePercent, networkProcessor=(.+)><>Value
    name: kafka_network_processor_idle_percent
    type: GAUGE
    labels:
      processor: "$1"
  # Under-replicated partitions
  - pattern: kafka.server<type=ReplicaManager, name=UnderReplicatedPartitions><>Value
    name: kafka_under_replicated_partitions
    type: GAUGE
  # Active controller count (1 on the active controller, 0 elsewhere)
  - pattern: kafka.controller<type=KafkaController, name=ActiveControllerCount><>Value
    name: kafka_active_controller_count
    type: GAUGE
  # Offline partitions
  - pattern: kafka.controller<type=KafkaController, name=OfflinePartitionsCount><>Value
    name: kafka_offline_partitions_count
    type: GAUGE
  # Request latency, 99th percentile, per request type
  - pattern: kafka.network<type=RequestMetrics, name=TotalTimeMs, request=(.+)><>99thPercentile
    name: kafka_request_latency_p99_ms
    type: GAUGE
    labels:
      request: "$1"

Two things to note. First, there is no hostPort key here. hostPort in the JMX Exporter config tells the exporter where to find a remote JMX RMI endpoint; it does not bind the HTTP server. The HTTP listen port comes from the agent argument below, so adding hostPort for an in-process agent is a no-op at best and a misconfiguration at worst. Second, throughput is exported only from the cumulative Count attribute as a COUNTER — you compute the rate in PromQL. Do not export the broker’s OneMinuteRate attribute and then apply rate() to it; that double-rates a value that is already a rate and produces nonsense.

Attach the agent in your broker startup. In kafka-server-start.sh or your systemd unit, set KAFKA_OPTS before the broker launches. The agent argument is [host:]<port>:<config-file>; with no host it binds 0.0.0.0:

export KAFKA_OPTS="$KAFKA_OPTS -javaagent:/opt/prometheus/jmx_prometheus_javaagent-0.20.0.jar=7071:/opt/prometheus/kafka-broker.yml"

Restart brokers one at a time, waiting for under-replicated partitions to return to zero between restarts, to avoid availability loss. Verify the endpoint:

curl -s http://broker-host:7071/metrics | grep kafka_active_controller_count

You should see kafka_ series. A connection refused means the port is firewalled or the agent did not attach — check the broker log for the agent line and confirm the JAR path.

A note on KRaft

These MBean names are identical in ZooKeeper and KRaft mode. ActiveControllerCount and OfflinePartitionsCount remain under kafka.controller:type=KafkaController on the broker/controller that holds the active controller role. In KRaft the metric is emitted by the controller nodes, so if you run dedicated controllers, scrape them too (attach the same agent on the controller process) or the controller panels will be empty.

Configuring Prometheus Scraping

Add a scrape job to prometheus.yml:

scrape_configs:
  - job_name: 'kafka-brokers'
    scrape_interval: 30s
    scrape_timeout: 10s
    metrics_path: '/metrics'
    static_configs:
      - targets:
          - 'broker-1:7071'
          - 'broker-2:7071'
          - 'broker-3:7071'
        labels:
          cluster: 'production-us-east'
          environment: 'prod'
    relabel_configs:
      - source_labels: [__address__]
        target_label: instance
        regex: '([^:]+):\d+'
        replacement: '${1}'

The relabel_configs block strips the port off __address__ so Grafana shows clean hostnames as instance. For multiple clusters, add one scrape job per cluster with a distinct cluster label so dashboards can aggregate across clusters or drill into one.

Prometheus pulls, so the broker endpoints must be reachable from the Prometheus server. On Kubernetes, use the Prometheus Operator’s PodMonitor/ServiceMonitor or pod annotations for discovery; on bare metal, generate the static targets with a config-management tool such as Ansible. The Prometheus configuration reference documents every option.

Designing the Grafana Dashboard

Lay the dashboard out by question, top to bottom: cluster health, throughput and latency, consumer lag, resource saturation. Every query below filters on a $cluster template variable so one dashboard serves every cluster.

Cluster Health Overview

The top row answers “is the cluster healthy?” with stat panels.

Active controller count — there must be exactly one. Zero means no controller; more than one means split brain.

sum(kafka_active_controller_count{cluster="$cluster"})

Threshold the stat panel: green at 1, red otherwise.

Under-replicated partitions — any non-zero value is a replication problem and a leading indicator of reduced durability.

sum(kafka_under_replicated_partitions{cluster="$cluster"})

Offline partitions — more severe than under-replication; these partitions have no available leader and are unavailable for reads and writes.

sum(kafka_offline_partitions_count{cluster="$cluster"})

Throughput and Latency

Bytes in/out — compute the rate from the cumulative counters. Apply rate() to the _total counter; never to a pre-computed OneMinuteRate:

sum(rate(kafka_broker_BytesInPerSec_total{cluster="$cluster"}[5m])) by (instance)
sum(rate(kafka_broker_BytesOutPerSec_total{cluster="$cluster"}[5m])) by (instance)

Request latency p99 — plot produce and consumer-fetch latency separately. Kafka labels request types Produce, FetchConsumer, and FetchFollower:

kafka_request_latency_p99_ms{cluster="$cluster", request="Produce"}
kafka_request_latency_p99_ms{cluster="$cluster", request="FetchConsumer"}

Base alert thresholds on your SLOs. A well-tuned cluster often holds produce p99 under ~10ms; treat sustained tens-of-milliseconds p99 as a signal to investigate disk and request load rather than a hard universal limit.

Consumer Lag

Consumer lag is the most important signal for stream-processing workloads: how far each consumer trails the log end. The broker’s JMX exporter does not expose consumer group offsets, so you need a dedicated source. Two common options:

  • Kafka Lag Exporter (or a kafka_exporter-style sidecar) that reads committed offsets from the cluster and exposes lag per group/partition.
  • The JMX Exporter attached to each Java consumer JVM, exposing the client’s own fetch metrics.

The consumer-client MBean for the worst-case lag across assigned partitions is records-lag-max:

  - pattern: kafka.consumer<type=consumer-fetch-manager-metrics, client-id=(.+)><>records-lag-max
    name: kafka_consumer_records_lag_max
    type: GAUGE
    labels:
      client_id: "$1"

Note the scope: records-lag-max is a client-side gauge keyed by client-id, not by consumer group or partition. It tells you the worst partition lag a given client currently sees, which is fine for a quick “who is falling behind” view but is not a per-group, per-partition lag total. For group-level lag totals, prefer the Kafka Lag Exporter, which exposes kafka_consumergroup_group_lag and similar series.

max(kafka_consumer_records_lag_max{cluster="$cluster"}) by (client_id)

A table panel sorted by descending lag surfaces the struggling clients first.

Resource Saturation

The network-processor idle fraction shows whether socket-handling threads have headroom; values consistently near 0.2 (20% idle) mean the network threads are saturated:

kafka_network_processor_idle_percent{cluster="$cluster"}

The request-handler pool idle fraction is the equivalent signal for the I/O threads that do the actual request work:

kafka_request_handler_idle_percent{cluster="$cluster"}

Plot both as time series with a warning line at 0.2. Both are fractions in the range 0.0–1.0, where 1.0 is fully idle. The Apache Kafka monitoring documentation lists the authoritative MBean for each metric.

Alerting Rules That Matter

Dashboards are for humans; alerts are for machines. Add a rule file (referenced from rule_files in prometheus.yml) that alerts on symptoms:

groups:
  - name: kafka-cluster-health
    rules:
      - alert: KafkaNoActiveController
        expr: sum(kafka_active_controller_count) != 1
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Kafka cluster has {{ $value }} active controllers"
          description: "Expected exactly 1 active controller. The cluster may be electing or split-brained."

      - alert: KafkaOfflinePartitions
        expr: sum(kafka_offline_partitions_count) > 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Kafka has {{ $value }} offline partitions"
          description: "Partitions have no available leader and are unavailable for produce/consume."

      - alert: KafkaUnderReplicatedPartitions
        expr: sum(kafka_under_replicated_partitions) > 0
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Kafka has {{ $value }} under-replicated partitions"
          description: "Replicas are lagging the leader. Durability is reduced if more brokers fail."

      - alert: KafkaHighProduceLatencyP99
        expr: kafka_request_latency_p99_ms{request="Produce"} > 100
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Kafka produce p99 latency is {{ $value }}ms on {{ $labels.instance }}"
          description: "Produce p99 exceeds the 100ms threshold. Check broker load and disk I/O."

      - alert: KafkaConsumerLagHigh
        expr: max(kafka_consumer_records_lag_max) by (client_id) > 10000
        for: 10m
        labels:
          severity: critical
        annotations:
          summary: "Consumer {{ $labels.client_id }} max lag is {{ $value }} records"
          description: "Consumer is falling behind. Check consumer throughput and processing time."

These alert on outcomes, not root causes: an under-replicated-partition alert tells you something is wrong, not why. Tune each for window to your tolerance for transient blips — controller re-elections and broker restarts cause brief, self-healing spikes you do not want to page on.

Multi-Cluster and Production Hardening

Use a Grafana dashboard variable so one dashboard covers every cluster. Define $cluster as a query variable:

label_values(kafka_active_controller_count, cluster)

Filter every panel on cluster="$cluster" and the dashboard becomes reusable across all environments.

For monitoring-stack high availability, run a pair of Prometheus servers with identical configs that scrape targets independently, then use Thanos or Grafana Mimir for deduplicated global queries and long-term storage. This keeps history and a global view even when one Prometheus is down.

Secure the metrics endpoints. The JMX Exporter supports HTTPS and HTTP basic auth via an httpServer configuration block, so do not leave the endpoint unauthenticated on an untrusted network. Pair that with firewall rules or Kubernetes NetworkPolicies that allow only the Prometheus servers to reach port 7071.

Finally, treat dashboard JSON as code. Keep it in Git next to your infrastructure-as-code and deploy it with Grafana provisioning (or a tool like grafanalib) so dashboard changes are reviewed, versioned, and reproducible.

You now have a monitoring foundation that surfaces the metrics that prevent outages. The dashboard answers the questions that matter — is the cluster healthy, are consumers keeping up, is latency within SLO — without burying you in noise, and it scales with your Kafka footprint while keeping your Operations, Security & Observability practice grounded in data.

Back to Monitoring Kafka With JMX Prometheus And Grafana