Alerting on Consumer Lag with Burrow and Grafana

In production Apache Kafka environments, consumer lag is a critical signal for stream-processing health. Lag is the delta between the last offset produced to a partition (the broker’s log-end offset) and the last offset committed by a consumer group for that partition. Unbounded lag means consumers cannot keep pace with incoming data, threatening real-time SLAs and backing up downstream systems.

Raw lag monitoring via JMX (kafka.consumer:type=consumer-fetch-manager-metrics,...records-lag-max) or the broker-side kafka.server:type=BrokerTopicMetrics gives you numbers, but numbers alone are noisy: a high-throughput consumer can carry 50,000 messages of standing lag and still be perfectly healthy, while a low-throughput consumer sitting at 500 messages that has stopped committing is in real trouble. LinkedIn’s Burrow was built to resolve exactly this ambiguity. Instead of comparing lag against a fixed threshold, it evaluates each consumer group’s offset history and emits a discrete status, which is what you actually want to page on.

This guide covers deploying Burrow, bridging it into Prometheus and Grafana, and building alerts that catch real incidents without paging on every traffic burst. It complements the broader strategies in Handling Large Backlogs and Consumer Lag.

Understanding Burrow’s Lag Evaluation Model

Burrow does not estimate a “time to clear lag” and does not compute a consumption rate from the production rate. Its model is simpler and more robust than that: for each partition a consumer group owns, Burrow keeps a sliding window of the last N committed offsets (and the lag observed at each commit), then applies a small set of rules to that window. The window size is set by intervals in the [storage] section and defaults to 10 stored offsets per partition. With a typical consumer commit interval of 60 seconds, that window spans roughly 10 minutes of history.

The evaluation rules are applied per partition:

• If any lag value in the window is zero, the partition is OK — the consumer caught up at least once recently.
• If the committed offset is not advancing and the lag is non-zero, the partition is STALL — the consumer is committing (or its commit is still being read) but making no progress.
• If the consumer has not committed at all for longer than the window’s time span and the lag is non-zero, the partition is STOP — Burrow believes the consumer has died.
• If the offset is advancing but lag is the same or larger at every step across the window, the partition is WARN — the consumer is moving but losing ground.
• If the committed offset moves backwards, the partition is REWIND (an offset reset or a misbehaving client).

Burrow then rolls partition results up into a group status. Group status uses only four values: NOTFOUND, OK, WARN, and ERR. A group reports ERR when one or more of its partitions is in a STOP, STALL, or REWIND state; it reports WARN when partitions are in the warning state but none are worse. STOP, STALL, and REWIND are partition-level statuses only — you will never see them as a group status, and ERR/NOTFOUND are never used at the partition level. The full set, as defined in Burrow’s core/protocol package, maps to these integer constants:

These integer codes matter because the Prometheus exporter publishes status as a number, and your Grafana value mappings and alert thresholds must use this exact ordering (higher = worse).

Deploying and Configuring Burrow

Burrow is a single Go binary, also published as an official container image on the GitHub Container Registry. Pull it from ghcr.io/linkedin/burrow/burrow:

docker pull ghcr.io/linkedin/burrow/burrow:latest

Pin a real tag in production (for example v1.9.6) rather than latest. Community images on Docker Hub under names like linkedin/burrow are unofficial and unmaintained.

Create a burrow.toml. The configuration below mirrors Burrow’s documented section structure: a [client-profile] holds the Kafka client settings (version, client ID, and optional TLS/SASL references), and the [cluster] and [consumer] sections reference it.

[general]
pidfile = "/var/run/burrow/burrow.pid"
stdout-logfile = "/var/log/burrow/burrow.out"

[logging]
filename = "/var/log/burrow/burrow.log"
level = "info"

[zookeeper]
servers = [ "zk1:2181", "zk2:2181", "zk3:2181" ]
timeout = 6
root-path = "/burrow"

[client-profile.local]
kafka-version = "2.0.0"
client-id = "burrow-monitoring"

[storage.default]
class-name = "inmemory"
intervals = 10
expire-group = 604800

[cluster.local]
class-name = "kafka"
client-profile = "local"
servers = [ "kafka1:9092", "kafka2:9092", "kafka3:9092" ]
topic-refresh = 60
offset-refresh = 10

[consumer.local]
class-name = "kafka"
cluster = "local"
servers = [ "kafka1:9092", "kafka2:9092", "kafka3:9092" ]
client-profile = "local"
offsets-topic = "__consumer_offsets"
start-latest = true

[httpserver.default]
address = ":8000"

A few things that commonly trip people up:

• The [cluster] section connects directly to Kafka brokers. It does not take offsets-topic, offsets-refresh, or offsets-retention keys — those are not valid here. Cluster refresh cadence is controlled by topic-refresh (default 60s) and offset-refresh (default 10s).
• Consumer offsets are read by the [consumer] section. offsets-topic (default __consumer_offsets) and start-latest (default false) belong here. Setting start-latest = true means Burrow only sees commits made after it starts, which avoids replaying the entire offsets topic on a cold start.
• The [zookeeper] section is used only for Burrow’s own coordination/notifier state, not for reading consumer offsets. Modern consumers commit to __consumer_offsets, so ZooKeeper here is optional in most deployments.

Run Burrow with the config mounted at the path the image expects (/etc/burrow/burrow.toml):

docker run -d \
  --name burrow \
  -v /path/to/burrow.toml:/etc/burrow/burrow.toml \
  -v /path/to/logs:/var/log/burrow \
  -p 8000:8000 \
  ghcr.io/linkedin/burrow/burrow:latest

Verify the process via the health-check endpoint:

curl http://localhost:8000/burrow/admin

List the consumer groups Burrow has detected:

curl http://localhost:8000/v3/kafka/local/consumer

Retrieve a specific group’s evaluated status:

curl http://localhost:8000/v3/kafka/local/consumer/my-group/status

The status response contains a top-level status.status (one of NOTFOUND/OK/WARN/ERR), status.complete (the fraction of partitions with a full window of data), status.maxlag (the partition with the worst lag), and a partitions array giving per-partition status, current_lag, start, and end offsets.

Integrating Burrow with Prometheus and Grafana

Grafana does not speak Burrow’s API directly, so the usual path is Burrow → exporter → Prometheus → Grafana. The widely used exporter is jirwin/burrow_exporter, which polls Burrow’s HTTP API and republishes the data as Prometheus gauges. There is no official Docker Hub image; build it from source (or use a vetted community image such as solsson/burrow-exporter):

git clone https://github.com/jirwin/burrow_exporter.git
cd burrow_exporter
docker build -t burrow_exporter .

Run it pointed at Burrow. Note API_VERSION defaults to 2; set it to 3 so the exporter calls the same /v3 endpoints shown above:

docker run -d \
  --name burrow-exporter \
  -e BURROW_ADDR="http://burrow:8000" \
  -e METRICS_ADDR="0.0.0.0:8080" \
  -e INTERVAL="30" \
  -e API_VERSION="3" \
  -p 8080:8080 \
  burrow_exporter

INTERVAL is the poll interval in seconds (default 30). Add a Prometheus scrape job for the exporter’s /metrics endpoint:

scrape_configs:
  - job_name: 'burrow'
    scrape_interval: 30s
    static_configs:
      - targets: ['burrow-exporter:8080']

The exporter publishes these gauges (verified against its source). Every metric carries cluster, group, topic, and partition labels except where noted:

• kafka_burrow_partition_lag — current lag for a partition.
• kafka_burrow_partition_current_offset — the consumer group’s last committed offset for a partition.
• kafka_burrow_partition_max_offset — the broker log-end offset for the partition.
• kafka_burrow_partition_status — per-partition status as the numeric code (0–6) from the table above.
• kafka_burrow_total_lag — summed lag across the group (labels: cluster, group).
• kafka_burrow_status — the group-level status code (labels: cluster, group).
• kafka_burrow_topic_partition_offset — the latest produced offset per topic partition.

Note the kafka_burrow_ prefix, not burrow_. Getting the metric names wrong is the single most common reason a “working” dashboard shows no data.

In Grafana, build a table panel on kafka_burrow_status with a value mapping that translates the numeric codes (1→OK, 2→WARN, 3→ERR) into colored cells, and a time-series panel on kafka_burrow_partition_lag grouped by group and topic for trend visibility.

Building Actionable Alerts on Burrow Status

Alert on Burrow’s evaluated status, not on raw lag thresholds. Because the status is already ordered (higher = worse), alert rules are simple comparisons. The examples below use Grafana’s unified alerting (Grafana 8+), where a rule is a query plus a threshold/expression; the same logic applies if you alert directly in Prometheus with for: durations.

A sensible tiering maps cleanly onto Burrow’s codes:

• WARN (group status = 2) — notify the on-call channel; do not page. Lag is growing but the consumer is still advancing. Often self-resolves after a burst.
• ERR (group status = 3) — page after it persists. A partition has stopped, stalled, or rewound. Sustained ERR needs a human.
• STOP / STALL (partition status = 4 or 5) — page quickly. The consumer is dead or wedged (a poison-pill message, a crashed instance, or a network partition).
• REWIND (partition status = 6) — alert and investigate. An offset reset is sometimes intentional (a replay) and sometimes a bug; either way it deserves eyes.

For a fast STOP/STALL detector, alert when the max partition status is at or above 4 over a short window, sustained for one minute:

A = max(kafka_burrow_partition_status) by (cluster, group)
Condition: A >= 4   for: 1m

For ERR conditions, use a longer for: to ride out transient rebalances and deployments:

A = max(kafka_burrow_status) by (cluster, group)
Condition: A >= 3   for: 5m

Requiring the group status to stay at ERR for five consecutive minutes filters out brief spikes during rebalances without delaying response to a genuinely stuck consumer.

Two practical refinements: alert per group (not globally) so a single bad consumer does not silence everything, and if you run more than one exporter (below), aggregate with max() by (cluster, group) so duplicate scrapes do not produce duplicate alerts.

Tuning Burrow for Your Workload

The single most impactful tuning knob is the window size, set by intervals in the [storage] section (default 10). The window’s time span is intervals multiplied by how often your consumers commit. A consumer that commits every 60 seconds gets a ~10-minute window with the default; a consumer that commits every 5 seconds gets only ~50 seconds. For low-frequency committers that would otherwise look “stopped,” widen the window:

[storage.default]
class-name = "inmemory"
intervals = 30
expire-group = 604800
min-distance = 1

min-distance (default 0) tells Burrow to ignore commits that arrive within this many seconds of the previous one, which prevents a chatty consumer from filling the window with near-identical offsets in a fraction of a second. expire-group (default 604800 seconds = 7 days) controls when an inactive group is purged from memory.

The cluster’s offset polling cadence is offset-refresh (default 10s); lowering it increases load on the brokers, so keep it modest on large clusters. If your brokers require TLS or SASL, attach those to the [client-profile] rather than the cluster or consumer section:

[tls.mytls]
certfile = "/etc/burrow/client-cert.pem"
keyfile = "/etc/burrow/client-key.pem"
cafile = "/etc/burrow/ca-cert.pem"
no-verify = false

[client-profile.local]
kafka-version = "2.0.0"
client-id = "burrow-monitoring"
tls = "mytls"

The kafka-version profile key must be set appropriately — Burrow’s bundled client library supports brokers up through the 2.x protocol, so set it to the lowest broker version in your cluster (for example 2.0.0) to negotiate a compatible protocol.

Operationalizing Burrow in Production

Burrow keeps its sliding windows in memory (the inmemory storage module). On restart it loses that history and must rebuild the window from fresh commits, so every group briefly evaluates as OK (insufficient data) until enough commits accumulate. Watch status.complete in the API response — it reports the fraction of the window that is populated — and suppress paging on groups below, say, 1.0 right after a restart. Plan restarts for low-traffic windows and expect a short monitoring gap.

For high availability, run two or more Burrow instances; each evaluates groups independently, so you can query any of them. Pair each Burrow with its own exporter and have Prometheus scrape all of them, then collapse the duplicates with max() by (cluster, group) in your alert expressions so two instances reporting the same ERR page you once, not twice.

Resource usage is modest: a single instance watching ~100 groups across ~1,000 partitions typically fits in well under 1 GB of RAM with negligible CPU. The real constraint is network round-trips to Kafka, so deploy Burrow close to the brokers (same region/AZ) to minimize jitter on offset refreshes.

Burrow logs as JSON via its [logging] section. Configure the file destination and level directly in burrow.toml:

[logging]
filename = "/var/log/burrow/burrow.log"
level = "info"
maxsize = 100
maxbackups = 30
maxage = 10
use-localtime = true
use-compression = true

Watch these logs for repeated Kafka connection errors or offset-fetch failures, which usually point at broker or network problems rather than Burrow itself.

Finally, monitor Burrow’s own liveness. A dead Burrow means you are blind to consumer lag, which is itself an incident. Add a Prometheus blackbox or up{job="burrow"} alert so an unreachable exporter or a crashed Burrow pages you. Correlate Burrow’s status with producer throughput and broker I/O in the same dashboards so you can tell whether lag originates from a producer spike, a slow consumer, or a broker bottleneck — the same proactive posture described in Handling Large Backlogs and Consumer Lag.

Related Pages

• Producers, Consumers & Delivery Semantics