Monitoring Cluster Health and Performance Metrics

Effective monitoring is the operational backbone of any production Apache Kafka deployment. Without deep visibility into cluster health and performance, platform and SRE teams cannot preempt failures, diagnose latency spikes, or validate that their Cluster Architecture & Provisioning decisions are delivering as expected. This guide covers instrumenting, collecting, and acting on the metrics that matter most for Kafka clusters, moving past “is it up?” checks into the layered telemetry required to operate a distributed streaming platform at scale. We cover broker-level resources, topic and partition performance, producer and consumer observability, and the health signals from the metadata consensus layer.

All MBean names and CLI flags below are taken from the Apache Kafka monitoring documentation. Where behavior is version-specific, that is called out explicitly.

The Four Golden Signals of Kafka Cluster Health

Borrowing from Google’s SRE practice, we can map the four golden signals—latency, traffic, errors, and saturation—directly onto Kafka. They give you a triage framework before you drill into granular metrics.

Latency is most meaningful end to end: the time from a producer sending a record to a consumer reading it. Internal broker latency is the leading indicator, though. The kafka.network:type=RequestMetrics,name=TotalTimeMs,request=Produce metric exposes the total time a broker spends servicing a produce request, including queue, local processing, remote (replication) wait, response queue, and send time. Elevated values, especially at the 99th percentile, usually point to disk contention or under-provisioned hardware—a direct consequence of choices made during Broker Sizing and Hardware Selection for Kafka Clusters. Drill into the per-request breakdown (RequestQueueTimeMs, LocalTimeMs, RemoteTimeMs, ResponseQueueTimeMs, ResponseSendTimeMs) to localize the bottleneck.

Traffic is throughput. Track kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec and BytesOutPerSec cluster-wide and per topic, against the theoretical limits of your network and storage I/O. A sudden, unplanned drop in throughput is as much a red flag as a spike.

Errors span both client-facing and internal broker failures. The kafka.network:type=RequestMetrics,name=ErrorsPerSec,request=Produce metric (and its Fetch counterpart) is the primary indicator; note that this MBean is further keyed by an error= attribute, so aggregate across error codes when alerting. Internally, the log flush metric kafka.log:type=LogFlushStats,name=LogFlushRateAndTimeMs and replica-fetcher failures signal durability risk.

Saturation measures how full a constrained resource is. In Kafka the usual suspects are CPU, network bandwidth, and disk I/O. The kafka.network:type=SocketServer,name=NetworkProcessorAvgIdlePercent metric directly measures network-thread headroom; a value approaching 0 means the network processors are the bottleneck. Correlate with OS-level disk utilization and, on virtualized instances, CPU steal time.

Broker-Level Resource Monitoring

A broker’s health is tied to its host’s resources. Watch them through a mix of JVM metrics, OS telemetry, and Kafka’s own instrumentation.

JVM and Garbage Collection

Kafka brokers run on the JVM, so garbage collection (GC) behavior is a first-class health signal. When you scrape the broker JVM through the Prometheus JMX Exporter, GC is exposed from GarbageCollectorMXBean as the jvm_gc_collection_seconds summary, with jvm_gc_collection_seconds_count (number of collections) and jvm_gc_collection_seconds_sum (total time), labeled per collector via a gc label. A steadily rising rate of old-generation collection time is the classic symptom of memory pressure, often from oversized fetch/produce batches or bloated consumer-group metadata.

Alert on average GC time per collection over a 5-minute window:

rate(jvm_gc_collection_seconds_sum{job="kafka"}[5m])
  / rate(jvm_gc_collection_seconds_count{job="kafka"}[5m]) > 0.5

This fires when the mean collection time exceeds 500 ms. Sustained pauses at that level cause client timeouts and ISR shrink events. (If you instrument with Micrometer instead of the JMX Exporter, the equivalent metric is jvm_gc_pause_seconds—the names are not interchangeable, so match the query to your exporter.)

OS and Disk I/O

Kafka’s write path leans on sequential disk I/O, but random consumer reads can saturate poorly provisioned drives. Key node-exporter metrics: node_disk_io_time_seconds_total for utilization and node_disk_read_bytes_total / node_disk_written_bytes_total for throughput. Utilization consistently above 80% warrants investigation or scaling.

To check disk utilization on a broker in real time:

iostat -x 1 | grep -E 'Device|sd[a-z]'

Watch the %util column. As it approaches 100% the disk subsystem is saturated. That often means the active data set has outgrown the page cache, forcing consumer reads to hit disk, or that a consumer is reading far behind the log tail.

Network Thread Pool Saturation

kafka.network:type=SocketServer,name=NetworkProcessorAvgIdlePercent reports how idle the network threads are, expressed as a fraction from 0 to 1. Kafka sizes this pool with num.network.threads. When it saturates, clients see added latency and connection timeouts.

A Prometheus rule for the condition:

groups:
  - name: kafka_broker
    rules:
      - alert: KafkaNetworkThreadSaturation
        expr: avg by(instance) (kafka_network_socketserver_networkprocessoravgidlepercent) < 0.2
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Network thread pool saturation on {{ $labels.instance }}"
          description: "Average idle percent below 20% for 5 minutes; network threads are exhausted."

Topic and Partition Performance Metrics

Broker metrics give the macro view; topic and partition granularity isolates noisy neighbors and uneven workload distribution.

Under-Replicated Partitions and ISR Shrinks

Under-replicated partitions (URPs) are arguably the single most important health metric. A non-zero count means data is not fully replicated and durability is at risk. Monitor kafka.server:type=ReplicaManager,name=UnderReplicatedPartitions cluster-wide and per broker.

A common cause is a slow follower that cannot keep up with the leader’s replication traffic. To find lagging followers, correlate URPs with kafka.server:type=ReplicaFetcherManager,name=MaxLag,clientId=Replica, the maximum lag (in messages) of any follower fetcher on that broker.

To list every under-replicated partition across the cluster:

kafka-topics.sh --bootstrap-server localhost:9092 --describe --under-replicated-partitions

This prints only the topics/partitions whose replicas are under-replicated, so you can quickly scope the problem.

Partition Size and Log Segment Management

Unbounded partition growth turns reassignment and recovery into incidents. Monitor on-disk size per partition with kafka.log:type=Log,name=Size,topic=*,partition=* (both the topic and partition attributes are required to identify a partition). Alerting as a partition approaches a defined ceiling (say 100 GB) lets you intervene proactively by adding partitions or tightening retention.

To tighten retention on a runaway topic without a restart:

kafka-configs.sh --bootstrap-server localhost:9092 \
  --entity-type topics --entity-name my_topic \
  --alter --add-config retention.bytes=107374182400

This applies a 100 GB per-partition size limit (retention.bytes is enforced per partition, not per topic), triggering segment deletion once a partition exceeds it.

Producer and Consumer Observability

Client-side metrics expose what broker metrics cannot: serialization overhead, client-side throttling, and application backpressure.

Producer Metrics

Watch record-send-rate, record-error-rate, request-latency-avg, and buffer-available-bytes (all under kafka.producer:type=producer-metrics,client-id=*). buffer-available-bytes is especially telling: if it regularly hits zero, the producer’s buffer.memory is exhausted because records are accumulating faster than the broker accepts them. From there, sends block for up to max.block.ms and then throw, so a too-low max.block.ms turns broker-side slowness into immediate producer errors.

Producer metrics are exposed via JMX automatically. As of Kafka 4.0 the default value of metric.reporters is org.apache.kafka.common.metrics.JmxReporter; on earlier versions the JmxReporter is always injected regardless. Setting it explicitly is harmless and makes intent clear (the older auto.include.jmx.reporter property is deprecated):

Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("metric.reporters", "org.apache.kafka.common.metrics.JmxReporter");
KafkaProducer<String, String> producer = new KafkaProducer<>(props);

The exposed JMX beans can then be scraped by the Prometheus JMX Exporter.

Consumer Lag and Group Health

Consumer lag is the delta between the log-end offset and the consumer group’s committed offset—the definitive measure of whether consumers keep up. The client-side metric records-lag-max on kafka.consumer:type=consumer-fetch-manager-metrics,client-id=* reports the maximum lag across that consumer’s assigned partitions. Its weakness: it only reflects the worst partition and disappears when the consumer is down—exactly when you most want it.

For that reason, alert on broker-side lag computed from committed offsets, exported by tools such as Kafka Lag Exporter or Burrow:

max by(consumergroup, topic) (kafka_consumergroup_lag) > 10000

Tune thresholds per topic against business impact: 10,000 records of lag on a high-volume clickstream topic is noise, while the same lag on a payments topic is an incident.

Metadata Consensus Layer Health

The metadata layer is the brain of the cluster, and what you monitor depends on whether you run ZooKeeper or KRaft. For the trade-offs between them, see KRaft vs ZooKeeper: Choosing the Right Metadata Management.

KRaft is now the default and only mode in current releases: Apache Kafka 4.0 (March 2025) removed ZooKeeper entirely. You cannot upgrade a ZooKeeper-based cluster directly to 4.0—migrate to KRaft on a 3.x bridge release (3.9 is recommended) first. Treat the ZooKeeper section below as relevant only to clusters still on 3.x or earlier.

KRaft Controller Health

In KRaft mode the controller quorum manages metadata through a Raft log. The must-watch metrics are kafka.controller:type=KafkaController,name=ActiveControllerCount (exactly 1 across the cluster; 0 or >1 is an incident) and kafka.controller:type=ControllerStats,name=UncleanLeaderElectionsPerSec. A non-zero unclean-election rate means a leader was elected from outside the ISR without full data sync—potential data loss.

kafka.controller:type=ControllerEventManager,name=EventQueueTimeMs measures how long events wait in the controller’s queue before processing. A rising value means the active controller is overloaded—commonly from a very high partition count or frequent broker state changes.

To inspect the quorum:

kafka-metadata-quorum.sh --bootstrap-server localhost:9092 describe --status

This reports LeaderId, LeaderEpoch, HighWatermark, CurrentVoters, CurrentObservers, and MaxFollowerLag/MaxFollowerLagTimeMs—a clear snapshot of consensus health. A growing MaxFollowerLag means a controller (or observing broker) is falling behind on metadata replication.

ZooKeeper Health (Legacy, 3.x and earlier)

For ZooKeeper-backed clusters, the key ensemble metrics are zk_avg_latency, zk_max_latency, zk_outstanding_requests, and zk_znode_count. A sustained non-zero zk_outstanding_requests means the ensemble is falling behind. A large zk_znode_count—driven by partition count—slows leader elections and metadata propagation.

A quick health snapshot uses the stat four-letter word over nc:

echo stat | nc localhost 2181 | grep -E 'Latency|Outstanding|Node'

Gotcha: since ZooKeeper 3.5.3 the four-letter-word commands are disabled by default—only srvr is whitelisted. To use stat, add it to 4lw.commands.whitelist in zoo.cfg (for example 4lw.commands.whitelist=stat, ruok, srvr, mntr) and restart, or scrape ZooKeeper’s own Prometheus endpoint instead.

Building a Unified Monitoring Dashboard

Collecting metrics is half the job; the other half is presenting them so an on-call engineer can diagnose fast. Structure dashboards in layers: a cluster summary up top, per-broker drill-downs, then client-centric views.

Dashboard Architecture with Prometheus and Grafana

Prometheus for collection and Grafana for visualization is the de facto open-source stack. The Prometheus JMX Exporter bridges Kafka’s JMX beans to Prometheus. Run it as a Java agent on each broker via KAFKA_OPTS:

flowchart LR K["Kafka broker (JMX MBeans)"] -->|"javaagent"| E["JMX Exporter (/metrics on 7071)"] E -->|scrape| PR["Prometheus"] PR --> G["Grafana dashboards"] PR --> AL["Alerting rules"] AL -->|page / notify| OC["On-call"]
export KAFKA_OPTS="-javaagent:/opt/prometheus/jmx_prometheus_javaagent.jar=7071:/opt/prometheus/kafka.yml"

The YAML config selects which beans to expose and how to name them. Because the JMX Exporter walks every bean by default, restrict it with explicit rules (and consider whitelistObjectNames) to keep scrape cost down on brokers with many topics:

lowercaseOutputName: true
rules:
  - pattern: kafka.server<type=(.+), name=(.+)PerSec\w*><>Count
    name: kafka_server_$1_$2_total
  - pattern: kafka.server<type=(.+), name=(.+)PerSec\w*, topic=(.+)><>Count
    name: kafka_server_$1_$2_total
    labels:
      topic: "$3"
  - pattern: kafka.network<type=SocketServer, name=NetworkProcessorAvgIdlePercent><>Value
    name: kafka_network_socketserver_networkprocessoravgidlepercent

This captures per-second rates as Prometheus counters and the network-processor idle gauge.

Alerting Strategy

Tier alerts into three buckets: critical (page immediately), warning (investigate in hours), and informational (trend only). Critical: UnderReplicatedPartitions > 0, ActiveControllerCount != 1, OfflinePartitionsCount > 0, and consumer lag over threshold on business-critical topics. Warning: resource saturation such as NetworkProcessorAvgIdlePercent < 0.2 or disk utilization above 80%.

Validate every threshold against baseline traffic. The common pitfall is static thresholds that fire on normal fluctuation. Use dynamic baselining where you can, and at minimum review thresholds quarterly. The Apache Kafka monitoring documentation lists every available metric with its description and is the authoritative reference for building rules.

Automated Health Checks and Remediation

Beyond passive monitoring, active checks catch issues before alerts fire. Run them as scheduled jobs to continuously validate cluster integrity.

End-to-End Latency and Availability Probe

A producer/consumer pair that writes and reads a heartbeat on a dedicated topic gives you a true client-perspective measure of availability and round-trip latency.

A Python probe using confluent_kafka:

from confluent_kafka import Producer, Consumer
import time, uuid

topic = "health-check"
bootstrap_servers = "localhost:9092"

producer = Producer({"bootstrap.servers": bootstrap_servers})
consumer = Consumer({
    "bootstrap.servers": bootstrap_servers,
    "group.id": f"health-check-{uuid.uuid4()}",
    "auto.offset.reset": "latest",
})
consumer.subscribe([topic])
# Force an assignment so "latest" is positioned before we produce.
consumer.poll(5.0)

test_key = str(uuid.uuid4()).encode("utf-8")
start = time.time()
producer.produce(topic, key=test_key, value=b"ping")
producer.flush()

deadline = start + 30
while time.time() < deadline:
    msg = consumer.poll(1.0)
    if msg is None or msg.error():
        continue
    if msg.key() == test_key:
        print(f"End-to-end latency: {(time.time() - start) * 1000:.2f} ms")
        break
else:
    print("Health check FAILED: no echo within 30s")
consumer.close()

Two gotchas this version handles: with auto.offset.reset=latest you must trigger a partition assignment (poll) before producing, or the consumer may start past your message and never see it; and a fixed deadline keeps the probe from hanging forever when the cluster is unhealthy—precisely the case you are testing for. Use a unique group.id per run so a previous committed offset doesn’t skip the probe message.

Configuration Drift Detection

Configuration drift is a quiet source of instability. A scheduled job that diffs current configuration against a version-controlled baseline catches it early.

# Dynamic (cluster-wide / per-broker) configs set via kafka-configs
kafka-configs.sh --bootstrap-server localhost:9092 \
  --entity-type brokers --describe > current_broker_configs.txt

# Per-topic configs
kafka-configs.sh --bootstrap-server localhost:9092 \
  --entity-type topics --describe > current_topic_configs.txt

# Compare against a committed baseline
diff baseline_broker_configs.txt current_broker_configs.txt

Note the scope: --entity-type brokers --describe without --entity-name returns the dynamic broker configs (those set at runtime via kafka-configs.sh, plus cluster-wide defaults), not the full static server.properties. To capture a broker’s complete effective configuration, add --all (and an --entity-name <brokerId>), which also lists static and default values. Reconcile any unexpected diff, or intentionally promote it to the baseline—essential when multiple teams hold admin access to the cluster.

Back to Cluster Architecture Provisioning