Debugging Log Compaction Issues and Tombstone Records

Log compaction retains the latest value for each key in a topic, which makes it the foundation for changelog topics, materialized state, and event-sourced systems. When it misbehaves—deleted keys reappear, segments never shrink, or tombstones outlive their welcome—the cause is rarely visible from a single metric. This guide walks through how the Log Cleaner actually works, then gives a repeatable procedure for diagnosing stalled compaction and tombstone problems, grounded in the broker internals and the real CLI/JMX surface.

How Log Compaction Works

The Log Cleaner runs as a pool of background threads (log.cleaner.threads, default 1). For each eligible partition it builds an in-memory offset map of the latest offset per key over the “dirty” portion of the log, then rewrites the older segments, keeping only the most recent record for each key. The original segments are replaced atomically once the cleaned copy is written, so a crash mid-clean never loses committed data.

Two boundaries matter:

• The active segment (the one currently being appended to) is never compacted. Everything from the last cleaned offset up to the start of the active segment is the cleanable (“dirty”) range.
• A partition becomes eligible when its dirty ratio—dirty bytes divided by total bytes in the cleanable range—exceeds min.cleanable.dirty.ratio (default 0.5). You can also force cleaning on a time basis with max.compaction.lag.ms, or delay it with min.compaction.lag.ms.

The cleaner persists its progress in a cleaner-offset-checkpoint file at the root of each log directory (log.dirs). Each line is <topic> <partition> <offset>, recording the first dirty offset (last cleaned offset + 1) for that partition. This file—not any CLI flag—is the authoritative record of how far the cleaner has gotten:

cat /var/lib/kafka/data/cleaner-offset-checkpoint

If a partition’s checkpoint offset stays far behind its log-end offset over time, the cleaner is not keeping up with that partition.

The offset map sets a hard per-pass key ceiling

The single most useful number to know is how many distinct keys one cleaner pass can hold. The offset map is a SkimpyOffsetMap: each entry stores a 16-byte MD5 of the key plus an 8-byte offset, so 24 bytes per key. Usable capacity per cleaner thread is therefore:

keys_per_pass ≈ log.cleaner.dedupe.buffer.size × log.cleaner.io.buffer.load.factor / 24

With the defaults (dedupe.buffer.size=134217728, i.e. 128 MiB, and io.buffer.load.factor=0.9), that is 134217728 × 0.9 / 24 ≈ 5.03 million distinct keys per pass. The buffer is shared across threads, so with log.cleaner.threads=4 each thread gets roughly a quarter of it (~1.26 M keys). When the cleanable portion of a partition contains more unique keys than fit, the cleaner cannot dedup it in one shot: it cleans the prefix it can map, advances the checkpoint partway, and the remaining keys are only reconciled on a later pass. This shows up directly as max-buffer-utilization-percent pinned near 100. The fix is to raise log.cleaner.dedupe.buffer.size until a single partition’s live key set fits—not to keep adding threads, which only divides the same buffer into smaller shares.

Diagnosing Stalled Compaction

Start at the JMX metrics the Log Cleaner exposes. The most important gauge is the maximum dirty percentage across all partitions on the broker:

# Kafka 4.0+: JmxTool lives in org.apache.kafka.tools (kafka.tools.JmxTool was removed)
kafka-run-class.sh org.apache.kafka.tools.JmxTool \
  --jmx-url service:jmx:rmi:///jndi/rmi://localhost:9999/jmxrmi \
  --object-name "kafka.log:type=LogCleanerManager,name=max-dirty-percent"

A max-dirty-percent that climbs and stays high means at least one partition’s dirty ratio is well past min.cleanable.dirty.ratio but is not being cleaned—the cleaner is behind, or wedged. Pair it with these kafka.log:type=LogCleaner metrics:

• cleaner-recopy-percent — bytes written divided by bytes read during the last clean, as a percentage. Lower is better (a well-deduplicated log recopies little). A persistently high value means the cleaner is doing a lot of I/O for little reduction.
• max-clean-time-secs — longest cleaning pass observed; a sudden jump points at a partition that has grown very large.
• max-buffer-utilization-percent — how full the cleaner’s dedup buffer got. If this is pinned near 100%, the offset map is too small to hold all keys in one pass (see the capacity formula above and log.cleaner.dedupe.buffer.size / log.cleaner.io.buffer.size), and the cleaner is doing extra passes.

Symptom-to-cause table

A frequently missed failure mode: a single corrupt or oversized partition can throw an exception and kill a cleaner thread. Once a thread dies it is not restarted, so the partitions it owned simply stop compacting while the rest of the cluster looks healthy. Watch for dead-thread-count > 0 and grep the broker log for [kafka-log-cleaner-thread] stack traces. The classic recovery is to stop the broker, back up or delete the offending partition’s entry from cleaner-offset-checkpoint, and restart so the cleaner re-derives its starting offset.

The other common stall is purely a sizing problem: a partition receiving steady updates to a small key set keeps almost all of its bytes in the active segment, so the cleanable range never crosses the dirty-ratio threshold. Roll the active segment more aggressively so older data becomes eligible:

# Smaller segments roll sooner, making data cleanable sooner
kafka-configs.sh --bootstrap-server localhost:9092 \
  --entity-type topics --entity-name my-compacted-topic \
  --alter --add-config segment.bytes=104857600

# Or lower the eligibility threshold for this topic
kafka-configs.sh --bootstrap-server localhost:9092 \
  --entity-type topics --entity-name my-compacted-topic \
  --alter --add-config min.cleanable.dirty.ratio=0.1

Note that there is no built-in CLI that prints a per-partition dirty ratio. kafka-log-dirs.sh --describe reports only partition, size, offsetLag, and isFuture per replica—useful for spotting a partition whose on-disk size never drops, but it does not expose dirtyRatio or a “cleanable offset.” For dirtiness, rely on max-dirty-percent and the checkpoint file above.

Tombstone Lifecycle and Deletion Delays

A tombstone is a record with a non-null key and a null value: it marks a key as logically deleted. Tombstones are not removed immediately, because a consumer that has read the old value must still get a chance to read the deletion. Retention is governed by delete.retention.ms (default 86400000, i.e. 1 day).

The mechanism is subtler than “keep the tombstone for delete.retention.ms after it was written.” As of KIP-534, when a compaction pass first encounters a tombstone it stamps a delete horizon into the record (an internal header). The tombstone is only physically removed on a later pass that runs after deleteHorizon = cleanTime + delete.retention.ms. So the clock starts when the cleaner first processes the tombstone, not when the producer wrote it—a tombstone in a rarely-cleaned partition can survive far longer than delete.retention.ms wall-clock.

This timing is exactly why delete.retention.ms must be sized against the slowest reader, not against intuition. The window a consumer has to observe a deletion is, in the worst case, only delete.retention.ms after the cleaner stamps the horizon—and the cleaner may stamp it almost immediately on a busy partition. A consumer (or a restoring Kafka Streams state store) that lags by more than delete.retention.ms can read past a tombstone after it has been removed, see the old value still physically present earlier in the log, and resurrect the key in its local state. Size the retention to comfortably exceed your worst restore or catch-up time.

To inspect what is actually on disk for a key, dump a segment:

kafka-dump-log.sh \
  --files /var/lib/kafka/data/my-compacted-topic-0/00000000000000000000.log \
  --print-data-log \
  | grep -A1 "payload: my-key"

A tombstone shows as a record with the key set and payload: empty (null). If you see a tombstone and older values for the same key in a closed segment, that segment has not been cleaned yet—check the checkpoint offset and max-dirty-percent. If the tombstone is gone but a stale value remains, the dirty ratio for that partition likely never crossed the threshold, so the dedup pass that would have dropped the old value hasn’t run.

To diff what a consumer would actually observe, read the topic with key, value, and offset printing:

kafka-console-consumer.sh \
  --bootstrap-server localhost:9092 \
  --topic my-compacted-topic \
  --from-beginning \
  --property print.key=true \
  --property print.value=true \
  --property print.offset=true \
  | grep "my-key"

Debugging Tombstone Reappearance and Ghost Keys

When a deleted key reappears, work through the causes in order of likelihood.

Unclean leader election. This is the most common cause of true resurrection. With unclean.leader.election.enable=true, an out-of-sync replica can be elected leader and serve data that predates a deletion already applied on the old leader. For a compacted topic backing stateful processing this is a correctness bug, not just an availability event. Check the broker default and the topic override:

kafka-configs.sh --bootstrap-server localhost:9092 \
  --entity-type topics --entity-name my-compacted-topic --describe

For compacted topics that back state, set unclean.leader.election.enable=false and accept that the partition stays offline until an in-sync replica returns, rather than serving stale keys.

Consumer/restore lag past the delete horizon. A reader slower than delete.retention.ms (measured from when the cleaner stamped the horizon, per the section above) can miss the tombstone entirely while still encountering an earlier physical copy of the key. This is the quiet, configuration-independent resurrection path: nothing fails, no leader changes, the consumer simply never sees the delete. Detect it by comparing the lagging group’s consumer-lag against delete.retention.ms expressed in records; fix it by widening the retention or by speeding up restore.

Cleaner timing across replicas. Each broker runs its own cleaner against its own replicas, so a tombstone can already be physically gone on one broker while still present on another that has cleaned less. On its own this does not resurrect a key—followers replicate the leader’s log byte-for-byte—but combined with an unclean election it widens the window in which a stale value can be promoted. Eliminating unclean elections closes this gap.

compact,delete plus short retention.ms. With cleanup.policy=compact,delete, time- and size-based retention runs alongside compaction and can delete whole segments—including the segment holding the latest value for a key—before that value is ever superseded. The key vanishes, then “reappears” when a producer next writes it. This is a configuration mismatch: keep retention.ms comfortably larger than the longest interval between updates for any live key, or drop delete from the policy if you never want segments aged out by time. See the Kafka log compaction docs for how the policies interact.

Tuning the Log Cleaner

The defaults are deliberately conservative. For high-throughput compacted topics, raise parallelism and lift the I/O throttle:

# server.properties
log.cleaner.threads=4
log.cleaner.io.max.bytes.per.second=200000000
log.cleaner.backoff.ms=1000

• log.cleaner.threads (default 1): cleaning is I/O-bound; one thread per data disk is a reasonable starting point. Remember each added thread divides the shared dedupe.buffer.size, so scale the buffer up in step if your partitions have large key sets.
• log.cleaner.io.max.bytes.per.second (default effectively unlimited—Double.MAX_VALUE): the combined read+write throttle for all cleaner threads. Set it explicitly to keep compaction from starving foreground produce/fetch I/O on shared disks; set it too low and cleaning falls behind.
• log.cleaner.backoff.ms (default 15000): how long the cleaner sleeps when no partition is dirty enough to clean. Lowering it makes the cleaner pick up newly-eligible partitions sooner, at the cost of slightly more idle CPU.
• log.cleaner.dedupe.buffer.size (default 134217728): size this from the capacity formula—the largest single partition’s live key count × 24 bytes ÷ io.buffer.load.factor, with headroom—before reaching for more threads.

Watch these kafka.log MBeans in steady state:

• kafka.log:type=LogCleanerManager,name=max-dirty-percent — the dirtiest partition on the broker. Trending toward 100 means the cleaner is losing ground; add threads or raise the throttle.
• kafka.log:type=LogCleaner,name=cleaner-recopy-percent — recopy efficiency. A sustained high value suggests low duplication (re-examine key cardinality) or thrashing passes.
• kafka.log:type=LogCleaner,name=max-compaction-delay-secs — how long compaction has been overdue for the most-delayed partition; useful when you rely on max.compaction.lag.ms.

These are listed in the Apache Kafka monitoring guide.

Preventing Compaction Pitfalls Through Design

1. Bound key cardinality. Unbounded keys (raw UUIDs, one per event) defeat compaction—every key is unique, so there is nothing to deduplicate and the log only grows. Worse, an unbounded live key set eventually overruns the offset map (see the per-pass ceiling above), so even closed segments stop fully compacting. Pick keys with bounded cardinality and a realistic update frequency.

2. Treat changelog topics deliberately. Kafka Streams creates and configures changelog topics for you; hand-rolled stores must do the same. Keep the changelog compacted, and align delete.retention.ms with how long a restoring instance might take to read past a deletion, so a slow restore never misses a tombstone.

3. Use explicit event types. Carrying an event_type field that distinguishes upserts from deletes—rather than relying solely on null-value tombstones—decouples downstream consumers from the tombstone retention window and survives a stale read more gracefully.

4. Enforce policy as code. A compacted topic silently switched to cleanup.policy=delete will lose data with no error. Kafka does not expose topic configuration as a JMX gauge, so guard it externally: manage topic configs through infrastructure-as-code, and periodically diff live config (kafka-configs.sh --describe) against your declared baseline, alerting on drift.

Related Pages

• Retention vs Compaction: When to Use Each
• Topics, Partitions & Data Modeling