Key Metrics to Monitor for Kafka Cluster Health

Operating Apache Kafka in production demands a data-driven approach to observability. Platform and SRE engineers need to move beyond binary liveness checks and watch the metrics that reveal the true state of a cluster before users feel the impact. This guide is an actionable reference: the key indicators worth monitoring, the exact JMX MBean names and commands to retrieve them, and how to interpret what they tell you. Sound monitoring starts with sound Cluster Architecture & Provisioning, because most of these metrics are a direct reflection of your provisioning and partition-placement decisions.

A note on the JMX examples below: the curl ... /jolokia/... snippets assume you have deployed a JMX-to-HTTP bridge such as Jolokia as a Java agent on the broker (a common pattern). With a plain JMX port you would instead use kafka-run-class.sh kafka.tools.JmxTool, a JMX exporter (e.g. the Prometheus JMX exporter), or JConsole.

1. Host Foundations: CPU, Memory, and Network

A Kafka broker is a JVM process whose performance is tied to the OS, the page cache, and the hardware underneath it. Before looking at Kafka-specific metrics, the host signals are non-negotiable.

CPU utilization. Distinguish user time from iowait. Sustained high iowait means the broker is blocked waiting on disk, which surfaces as elevated produce and fetch latency. CPU user saturation, by contrast, often points at TLS termination, compression, or an undersized broker.

JVM heap and garbage collection. Kafka serves reads primarily from the OS page cache, but the JVM heap still drives replication, request handling, client connection state, and (in ZooKeeper mode) controller bookkeeping. The metric that matters most is GC pause time. Long or frequent stop-the-world pauses can make a broker miss its ZooKeeper session timeout (or its KRaft fetch deadline), triggering needless ISR shrink/expand cycles, leader re-elections, and client timeouts.

Inspect GC behavior on a running broker:

# JVM GC summary for a Kafka broker process (replace <PID>); 10 samples, 1s apart
jstat -gcutil <PID> 1000 10

Watch the FGC (full GC count) and FGCT (cumulative full GC seconds) columns. A steadily climbing FGCT, or full GCs occurring at all under steady state, is a red flag. Kafka’s bundled bin/kafka-server-start.sh already sets G1GC as the default collector via KAFKA_JVM_PERFORMANCE_OPTS; the defaults are equivalent to the long-standing LinkedIn/Confluent recommendation, and you only need to override them to tune:

export KAFKA_JVM_PERFORMANCE_OPTS="-server -XX:+UseG1GC -XX:MaxGCPauseMillis=20 \
  -XX:InitiatingHeapOccupancyPercent=35 -XX:+ExplicitGCInvokesConcurrent \
  -Djava.awt.headless=true"

A broker heap of roughly 6 GB is sufficient for most workloads; oversizing the heap steals memory from the page cache, which usually hurts more than it helps.

Network throughput and errors. Track inbound and outbound bytes against the NIC’s physical limit, and watch TCP retransmissions. A rising retransmit rate points to congestion, a faulty link, or MTU/offload misconfiguration, and shows up downstream as produce latency and fetch failures. Inspect TCP-level counters directly on the host:

netstat -s | grep -i retrans   # cumulative retransmit counters
ss -ti                         # per-socket RTT, cwnd, and retrans stats

2. Request Handler and Network Thread Saturation

Kafka processes client and inter-broker requests through two distinct thread pools, and conflating them is a common monitoring mistake.

• Network threads accept connections and read/write bytes on the socket. They are sized by num.network.threads.
• Request handler (I/O) threads dequeue requests and do the actual work, including disk reads and writes. They are sized by num.io.threads.

The clearest saturation signals are the per-pool idle ratios, each reported as a fraction between 0 (fully busy) and 1 (fully idle):

• Network threads: kafka.network:type=SocketServer,name=NetworkProcessorAvgIdlePercent
• Request handler threads: kafka.server:type=KafkaRequestHandlerPool,name=RequestHandlerAvgIdlePercent

As a rule of thumb, a RequestHandlerAvgIdlePercent below 0.2 indicates a loaded broker and below 0.1 a genuine performance problem — typically disk write pressure, GC stalls, or simply too few I/O threads for the workload. A low NetworkProcessorAvgIdlePercent instead points at connection churn or raw throughput pushing the network pool.

The next signal is the request queue depth, the backlog of requests waiting for a handler thread:

# Request queue size from a broker exposing Jolokia (e.g. on port 9999)
curl -s 'http://broker-host:9999/jolokia/read/kafka.network:type=RequestChannel,name=RequestQueueSize' | jq .

A healthy broker keeps RequestQueueSize near zero. Persistent queuing means the broker can’t drain requests as fast as they arrive — investigate disk latency, CPU starvation, or thread-pool sizing, and revisit your Capacity Planning for Kafka Clusters: Throughput and Storage. (Since KIP-291, controller-plane requests are queued separately, so RequestQueueSize reflects the data plane only.)

Localize latency before you blame the broker. When latency rises, a single queue gauge won’t tell you where the time goes. Kafka decomposes every request’s TotalTimeMs into named phases under kafka.network:type=RequestMetrics, tagged by request (Produce, FetchConsumer, or FetchFollower). Reading the breakdown points you straight at the cause:

All five are exposed with each attribute as the name, e.g. kafka.network:type=RequestMetrics,name=RemoteTimeMs,request=Produce. A high RemoteTimeMs on Produce is the classic fingerprint of acks=all waiting on a struggling follower — a fix that lives in replication, not in this broker’s threads.

Related to the queue, requests parked waiting on a condition (a Produce request awaiting acks, a Fetch request awaiting fetch.min.bytes) sit in purgatory, not the request queue:

kafka.server:type=DelayedOperationPurgatory,name=PurgatorySize,delayedOperation=Produce
kafka.server:type=DelayedOperationPurgatory,name=PurgatorySize,delayedOperation=Fetch

A large fetch purgatory is normal and expected — it is just consumers long-polling — so don’t alert on it. A growing produce purgatory, however, means acknowledgements are stalling and pairs naturally with a rising RemoteTimeMs,request=Produce.

3. Replication and Partition Availability

Two metrics belong on every Kafka dashboard and alert path: under-replicated partitions and offline partitions. Critically, they live in different JMX domains — a frequent source of broken dashboards.

An under-replicated partition has fewer in-sync replicas than its replica count: some follower has fallen behind the leader, usually because of network pressure, a slow disk, or a struggling broker. A sustained nonzero count means the cluster has lost its fault tolerance margin.

kafka.server:type=ReplicaManager,name=UnderReplicatedPartitions

An offline partition has no leader at all — it is unwritable and unreadable, and every offline partition is an active outage for the data on it. This metric is owned by the controller, not the replica manager:

kafka.controller:type=KafkaController,name=OfflinePartitionsCount

You can also surface both states from the CLI, which is handy for triage and is leader-aware across the whole cluster:

kafka-topics.sh --bootstrap-server localhost:9092 --describe --under-replicated-partitions
kafka-topics.sh --bootstrap-server localhost:9092 --describe --unavailable-partitions

Both metrics should be zero in a healthy cluster. Alert on OfflinePartitionsCount > 0 immediately, and on UnderReplicatedPartitions > 0 sustained for more than a minute or two. (Note: partitions being added by a reassignment are deliberately excluded from UnderReplicatedPartitions.) When either fires, check broker logs, network connectivity, and disk health on the replicas involved.

UnderReplicatedPartitions tells you the cluster is currently degraded; the ISR churn rate tells you it is flapping. Watch the rate at which replicas drop out of and rejoin the in-sync set:

kafka.server:type=ReplicaManager,name=IsrShrinksPerSec
kafka.server:type=ReplicaManager,name=IsrExpandsPerSec

Per the Kafka docs, when a broker goes down ISR shrinks for some partitions and then expands once replicas catch up; other than that, the expected value for both the shrink and expand rate is 0. A steady drumbeat of shrink-then-expand without any broker actually failing is the signature of intermittent GC pauses, network blips, or a replica.lag.time.max.ms set too tight for your fetch latency — a slow leak of availability margin that an UnderReplicatedPartitions snapshot can miss between samples.

4. Producer and Consumer Performance

Client-side metrics are the ultimate measure of user experience. Kafka’s Java clients expose them under the kafka.producer/kafka.consumer JMX domains, tagged by client-id.

Producers. Watch record-error-rate (records that failed permanently) alongside record-retry-rate. Errors rising together with retries means producers can’t get records acknowledged — often a broker-side symptom such as a saturated request queue or an unavailable leader. Track request-latency-avg for the round-trip time of produce requests. The official producer monitoring reference lists the full set.

Consumers. records-lag-max is the headline metric: the maximum offset lag across the partitions a consumer is reading. A steadily growing value means consumers can’t keep up with the production rate. Confirm and break it down per group from the CLI:

kafka-consumer-groups.sh --bootstrap-server localhost:9092 --group my-consumer-group --describe

The LAG column shows per-partition lag (log-end-offset minus committed offset). Large, growing lag calls for more consumer instances — up to the partition count, since a partition is consumed by at most one member of a group — or faster per-record processing. Also track fetch-rate and bytes-consumed-rate to understand consumed volume. See the consumer monitoring reference for the complete list.

Two gotchas decide whether you can trust lag in an alert. First, the client-side lag MBeans only report while a consumer is connected and actively fetching — a crashed or rebalancing consumer simply stops emitting, so absence of a high-lag signal is not proof of health. Second, records-lag-max is a count of records, not a time; 50,000 records of lag could be two seconds or two hours depending on throughput. For an authoritative, externally observable view — and one that survives a dead consumer — scrape kafka-consumer-groups.sh or a dedicated lag exporter that can also surface lag as a time delay.

5. Disk I/O and Log Segments

Kafka’s throughput rests on sequential disk I/O. Reads are usually served from the page cache, but the write path and segment housekeeping can still bottleneck a broker.

Monitor per-device latency and utilization with iostat:

iostat -x 2

Focus on await (average time to service an I/O request, including queue time) and %util for the devices backing your log.dirs. A high await points to a saturated storage subsystem, which stalls request handler threads and backs up the queue from Section 2 — you will typically see it mirrored as a rising LocalTimeMs in the request breakdown. Sustained %util near 100% means the disk is the bottleneck — a cue to move to faster media (NVMe), spread partitions across more brokers, or revisit capacity planning. (Avoid relying on svctm: it is deprecated and unreliable on modern Linux kernels.)

On the Kafka side, note that durability normally comes from replication, not fsync: brokers default to letting the OS flush the page cache, and the log.flush.interval.ms / log.flush.interval.messages settings — which force application-level flushes — are left unset on purpose. If you do enable them, watch kafka.log:type=LogFlushStats,name=LogFlushRateAndTimeMs; high flush times translate directly into write-latency spikes. Independently, keep an eye on segment counts: many tiny segments strain filesystem metadata and lengthen log recovery on restart, usually a sign of an undersized segment.bytes or an aggressive segment.ms relative to your throughput.

6. Controller and Metadata Health

The controller manages partition leadership and replica assignment. In ZooKeeper mode it is one elected broker; in KRaft mode it is a member of a dedicated controller quorum. Either way, its health gates every leadership change in the cluster.

The cornerstone metric is ActiveControllerCount:

curl -s 'http://broker-host:9999/jolokia/read/kafka.controller:type=KafkaController,name=ActiveControllerCount' | jq .

Each broker reports 0 or 1; the sum across the cluster must equal exactly 1. A cluster-wide sum of 0 means no controller is active and no leadership changes can happen; a sum greater than 1 indicates a split-brain condition. Alert on any value other than 1.

The rate and latency of leader elections is the next signal. In ZooKeeper mode it is exposed as:

kafka.controller:type=ControllerStats,name=LeaderElectionRateAndTimeMs
kafka.controller:type=ControllerStats,name=UncleanLeaderElectionsPerSec

A spike in LeaderElectionRateAndTimeMs means brokers are dropping out of the ISR or crashing, and frequently correlates with the GC pauses and network issues described earlier. UncleanLeaderElectionsPerSec should be zero — any nonzero value means a leader was elected from an out-of-sync replica and acknowledged data was lost.

Be aware of a KRaft caveat: the kafka.controller:type=ControllerStats and ControllerChannelManager MBeans are ZooKeeper-mode constructs and are not present in KRaft mode. On a KRaft cluster, derive controller health from the quorum and metadata metrics instead. Track the broker’s metadata lag — how far its applied metadata trails the active controller — and the raft quorum’s leadership:

kafka.server:type=broker-metadata-metrics,name=last-applied-record-lag-ms
kafka.server:type=raft-metrics

A rising last-applied-record-lag-ms means a broker is falling behind on metadata updates and may act on a stale view of leadership; the raft-metrics MBean carries the quorum’s CurrentLeader and HighWatermark, the KRaft analogues of “who is the controller” and “how far has the metadata log committed.” For the architectural background, see the Apache Kafka KRaft documentation.

Bringing It Together

No single metric tells the whole story, but they layer into a clear diagnostic order. Host signals (CPU iowait, GC pauses, TCP retransmits) explain why a broker is slow; the RequestMetrics time breakdown and RequestQueueSize localize where the slowness lives — in the queue, on local disk, or waiting on a follower; UnderReplicatedPartitions, the ISR churn rate, and OfflinePartitionsCount quantify the erosion and loss of redundancy; client lag and error rates measure the user-visible impact; and the controller metrics confirm the cluster’s brain is intact.

Wire these into a layered dashboard, but page only on the handful of hard invariants — everything else is for diagnosis:

Get those four right and the rest of the metrics in this guide become the toolkit for explaining why an alert fired — which is the difference between operating Kafka and merely watching it.