Retention vs Compaction: When to Use Each

Apache Kafka stores data durably, but durability is not one-size-fits-all. The two cleanup policies that govern a log’s lifecycle—time/size-based retention (delete) and log compaction (compact)—solve fundamentally different problems. Choosing wrong leads to ballooning storage bills, silent data loss, or stale application state. For platform engineers and SREs, knowing the precise operational boundaries of each is the difference between a predictable cluster and a recurring incident.

This article covers the internal mechanics of both policies, a concrete decision framework, the configuration keys that actually exist (with their real defaults), and the failure modes that surprise even experienced operators. It connects these policies to partition leadership, consumer-group semantics, and the broader Topics, Partitions & Data Modeling design that underpins every deployment.

The Mechanics of Time-Based Retention

cleanup.policy=delete is the default. Its premise is simple: once a segment is older than a configured age, or the partition exceeds a configured size, whole segments are deleted. The implementation details are where operators get tripped up.

Segment-Level Deletion

Kafka never deletes individual messages under the delete policy. Each partition’s log is split into segments—immutable, sequentially written files. log.retention.hours (or log.retention.ms / log.retention.minutes for finer granularity) sets the maximum message age; log.retention.bytes caps total partition size. A segment becomes eligible only when its newest message is older than the retention period (for time) or the partition is over its size cap.

Deletion is triggered by a periodic check, not continuously. log.retention.check.interval.ms (default 300000, i.e. 5 minutes) controls how often the broker scans for eligible segments. A segment that ages out at T+0 may not be removed until the next scan. In environments with tight disk headroom, lowering this to 60000 reduces the worst-case lag at the cost of more frequent scans.

# Broker-level defaults (can be overridden per topic)
log.retention.hours=168
log.retention.bytes=-1
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000

Note that the active segment is never deleted, and the active segment must roll before the segment behind it can be cleaned. A low-throughput partition can therefore exceed retention.ms substantially if no new writes force a roll. segment.ms bounds this by rolling the active segment after a fixed time even without writes.

Time-Based Retention and Consumer Lag

A classic incident: a consumer group falls behind, the retention window passes, and the segments it still needed are deleted. Kafka does not warn the consumer—it just removes the data. On the next fetch the consumer hits an OffsetOutOfRangeException, and auto.offset.reset decides what happens next: earliest resets to the oldest surviving offset (reprocessing a gap), latest jumps to the log end (silently skipping every record that was deleted), and none throws. With latest, this looks exactly like silent data loss.

There is no single formula for “safe” retention, but the window must comfortably exceed the worst expected consumer downtime:

retention.ms > max_expected_consumer_lag (time) + recovery_headroom

If consumers normally lag minutes but can lag 6 hours during incident response, a multi-day retention window gives ample buffer. The cost is storage: a topic ingesting 1 TB/hour into a single partition holds 168 TB after 7 days. This is where Partitioning Strategies for High-Throughput Topics matters—spreading load across more partitions reduces per-partition footprint, at the cost of more segments and open file handles to manage.

# Check consumer lag for a specific group
kafka-consumer-groups.sh --bootstrap-server localhost:9092 \
  --group my-consumer-group --describe

# Output columns include CURRENT-OFFSET, LOG-END-OFFSET, and LAG

Size-Based Retention and Its Trade-offs

retention.bytes is a per-partition cap, not a per-topic one. It deletes by total partition size regardless of age, so under heavy write load it can evict data that is only minutes old. Use it as a hard ceiling alongside a time-based primary policy, so that time-sensitive data is protected by retention.ms while retention.bytes only guards against runaway growth.

# Topic-level configuration combining time and size retention
kafka-configs.sh --bootstrap-server localhost:9092 \
  --entity-type topics --entity-name orders \
  --alter --add-config retention.ms=259200000,retention.bytes=107374182400

This retains messages for 3 days but caps each partition at 100 GB; whichever limit is hit first wins. Be deliberate about this when a compliance regime mandates a minimum retention period—retention.bytes can violate it under load. For more, see Configuring Retention Policies for Compliance and Cost Control.

The Mechanics of Log Compaction

Log compaction (cleanup.policy=compact) is a different model. Rather than deleting by time or size, it guarantees that the log retains at least the last known value for every unique message key. It turns a topic into a durable, replayable snapshot of current state.

How the Cleaner Works

Compaction is performed by the log cleaner, a pool of threads sized by log.cleaner.threads (default 1). The log is conceptually split into a “clean” tail (already compacted) and a “dirty” head (new writes since the last clean). Cleaning proceeds in two steps:

1. Eligibility by dirty ratio. The cleaner picks the log with the highest ratio of dirty bytes to total bytes, and only cleans logs whose ratio exceeds log.cleaner.min.cleanable.ratio (broker default 0.5; the per-topic override is min.cleanable.dirty.ratio).

2. Building the offset map and recopying. The cleaner scans the dirty head to build an in-memory map of key → latest offset, then recopies the segments, dropping any record whose key has a later offset. Compaction never reorders records and never changes offsets; surviving records keep their original offsets, so the compacted log has gaps.

Crucially, the cleaner never cleans past the last stable offset (LSO)—the boundary of the most recent committed/aborted transaction. On a transactional topic, a hanging (never committed or aborted) transaction therefore pins the LSO and blocks compaction indefinitely.

# Compaction-related broker configurations (defaults shown)
log.cleaner.min.cleanable.ratio=0.5
log.cleaner.max.compaction.lag.ms=9223372036854775807
log.cleaner.min.compaction.lag.ms=0
log.cleaner.delete.retention.ms=86400000

log.cleaner.delete.retention.ms (topic override delete.retention.ms, default 86400000 = 1 day) controls how long a tombstone (a record with a non-null key and a null value) is retained after the segment it lives in becomes eligible for cleaning. If a consumer is offline longer than this, it can miss the tombstone entirely and never learn the key was deleted.

Guarantees and Limitations

Compaction’s core guarantee: a consumer that reads to the head of the log will observe the latest value for every key that has not been tombstoned. The caveats:

• Per-partition only. The guarantee holds within a partition, never across partitions. If the same key lands in two partitions, you get two surviving values. Design keying with this in mind—see Keying Strategies for Ordered Message Processing.

• Eventually consistent. Cleaning is periodic; an old value can persist long after a newer one is written. log.cleaner.min.compaction.lag.ms keeps a record uncompacted for a minimum time, and log.cleaner.max.compaction.lag.ms (topic override max.compaction.lag.ms, default effectively infinite) forces cleaning within an upper bound even if the dirty ratio is low.

• I/O and CPU heavy. The cleaner reads whole segments, decompresses them, builds the offset map, and rewrites segments. On clusters with thousands of compacted partitions this can dominate broker load.

Monitoring Compaction Health

The log cleaner exposes its state through LogCleanerManager and LogCleaner MBeans:

# Cleaner health via JMX
kafka.log:type=LogCleanerManager,name=max-dirty-percent
kafka.log:type=LogCleanerManager,name=time-since-last-run-ms
kafka.log:type=LogCleanerManager,name=uncleanable-partitions-count,logDirectory="..."

max-dirty-percent is the worst dirty ratio across all logs, scaled to 0–100. When it stays high, the cleaner is falling behind and consumers reading from the start will replay large amounts of superseded data. time-since-last-run-ms climbing without bound means the cleaner thread has likely died (often after an IOException on a corrupt segment), and uncleanable-partitions-count (reported per log directory) becoming non-zero confirms partitions the cleaner has given up on. Alert on all three.

Decision Framework: Retention vs Compaction

This is not strictly binary—Kafka also supports the combined compact,delete policy. Use the guidance below.

Use delete (time/size) when:

1. You need the full, ordered event history. Audit logs, clickstreams, and transaction ledgers must preserve every event for a defined period. Compaction would discard intermediate states needed for replay.

2. Consumers care about events, not just latest-per-key state. If downstream logic depends on seeing every update to a key, compaction will silently drop the superseded ones.

3. Storage is predictable. A known retention window and throughput make capacity planning and debugging straightforward.

# Pure time-based retention topic (note: --config is repeated per setting)
kafka-topics.sh --bootstrap-server localhost:9092 \
  --create --topic audit-logs \
  --partitions 12 --replication-factor 3 \
  --config cleanup.policy=delete \
  --config retention.ms=7776000000 \
  --config segment.ms=86400000

This retains audit logs for 90 days, rolling a new segment daily so deletion granularity is at most one day—no single oversized segment blocks cleanup.

Use compact when:

1. You maintain a changelog for stateful stream processing. Kafka Streams state stores and ksqlDB materializations back their state with compacted changelog topics; on restart, an instance rebuilds local state by replaying the compacted log.

2. Only current state per key matters. A user-profile or device-shadow service needs the latest record per ID, not the full edit history. Compaction turns the topic into a durable key-value snapshot.

3. You implement deletes via tombstones. Producing a null-valued record for a key removes it from the compacted view after delete.retention.ms.

# Pure compaction topic
kafka-topics.sh --bootstrap-server localhost:9092 \
  --create --topic user-profiles \
  --partitions 12 --replication-factor 3 \
  --config cleanup.policy=compact \
  --config min.cleanable.dirty.ratio=0.5 \
  --config delete.retention.ms=604800000

Here delete.retention.ms=604800000 (7 days) gives consumers a week to observe tombstones before they are cleaned away—a far safer default than the 1-day broker default if any consumer can be offline for days.

Use compact,delete when:

You want the latest value per key and a bound on how long any record—including superseded values awaiting compaction—can live. The combined policy compacts as usual but also deletes whole segments older than retention.ms.

# Hybrid policy. cleanup.policy contains a comma, so it MUST be bracketed.
kafka-configs.sh --bootstrap-server localhost:9092 \
  --entity-type topics --entity-name orders-snapshot \
  --alter --add-config 'cleanup.policy=[compact,delete],retention.ms=2592000000,delete.retention.ms=86400000'

kafka-configs takes a single --add-config with a comma-separated list, so any value that itself contains a comma must be wrapped in square brackets: cleanup.policy=[compact,delete]. Splitting into multiple --add-config flags does not work—the tool accepts only one. In this example the topic keeps the latest value per key but deletes anything older than 30 days, preventing a write-once-never-updated key from occupying storage forever. The 1-day tombstone window is aggressive and only safe if consumers are continuously online.

Operational Deep-Dive: Tuning at Scale

Configuring a handful of topics is trivial. Doing it across thousands of topics with differing requirements demands automation and governance.

Topic-Level Overrides and Precedence

Precedence is: per-topic override > broker config > Kafka built-in default. A single misconfigured topic override silently defeats a carefully tuned broker default. Gate topic creation behind a pipeline that validates configs against a schema.

# Example topic configuration schema for governance
topic_config_schema:
  type: object
  properties:
    cleanup.policy:
      enum: [delete, compact, "compact,delete"]
    retention.ms:
      type: integer
      minimum: 3600000
      maximum: 31536000000
    min.cleanable.dirty.ratio:
      type: number
      minimum: 0.1
      maximum: 0.9
  required: [cleanup.policy]

Enforce the schema in CI for topic provisioning. The Java AdminClient (createTopics, alterConfigs, incrementalAlterConfigs) or a topic-management controller can apply the rules programmatically.

Scaling the Cleaner: Threads and I/O

On compaction-heavy clusters the cleaner is often the bottleneck. log.cleaner.threads (default 1) sets the thread count. log.cleaner.io.max.bytes.per.second throttles the cleaner’s combined read+write throughput; its default is effectively unlimited (Double.MAX_VALUE), so you set it only to protect foreground traffic, not to speed compaction up.

# Cleaner tuning for a compaction-heavy broker
log.cleaner.threads=8
log.cleaner.io.max.bytes.per.second=1500000000
log.cleaner.backoff.ms=15000
log.cleaner.dedupe.buffer.size=134217728

log.cleaner.dedupe.buffer.size (134217728 = 128 MB, which is already the default) is the total memory shared across all cleaner threads for the key→offset map. This sizing matters: if a log’s dirty section has more unique keys than the buffer can hold, the cleaner must clean it in multiple passes, doing extra I/O. For topics with very high key cardinality, raise this above the default—the value above merely states the default explicitly. Both dedupe.buffer.size and io.max.bytes.per.second are dynamically reconfigurable without a broker restart.

Segment Configuration

Segment size and roll interval affect both policies. Smaller segments give finer deletion granularity and faster individual compaction passes, but more open files, more frequent cleaner scans, and larger index overhead. The active segment is never compacted or deleted, so a slow-rolling segment delays both.

# Segment tuning for a high-throughput compacted topic
segment.bytes=536870912
segment.ms=600000
segment.index.bytes=10485760

A 512 MB segment with a 10-minute roll keeps segments large enough to amortize compaction passes but small enough to roll out of the active position quickly. segment.index.bytes=10485760 (10 MB) is the default; it’s stated here only as a reminder that high-key-cardinality topics rely on a large offset index, and shrinking it risks premature segment rolls.

Debugging Compaction and Retention Failures

The Dirty-Ratio Spiral

When the cleaner falls behind, the dirty ratio climbs, each pass gets more expensive, and the cleaner falls further behind. The root cause is almost always too few cleaner threads or insufficient I/O headroom.

Diagnosis:

1. Watch kafka.log:type=LogCleanerManager,name=max-dirty-percent. A sustained high value means the cleaner can’t keep up.
2. Check time-since-last-run-ms. If it grows without bound, the cleaner thread has probably died—inspect the broker log for LogCleaner IOExceptions and check uncleanable-partitions-count.
3. Raise log.cleaner.threads incrementally, watching broker CPU and disk utilization.
4. If disk I/O is saturated, raise (or set) log.cleaner.io.max.bytes.per.second, or move compacted topics to brokers with faster storage.

# Identify the largest partition directories (size in bytes)
kafka-log-dirs.sh --bootstrap-server localhost:9092 \
  --describe --topic-list compacted-topic | \
  tail -n +3 | \
  jq '.brokers[].logDirs[].partitions[] | select(.size > 1000000000)'

kafka-log-dirs --describe returns per-partition size, offsetLag, and future fields—there is no per-partition dirty-ratio in this output (the dirty ratio is exposed only via the JMX gauge above), so filter on .size to find heavy partitions. The first two lines of output are human-readable text, so skip them before piping to jq.

Tombstone Removal and State Resurrection

A tombstone (non-null key, null value) marks a key as deleted. If a consumer is offline longer than delete.retention.ms, the tombstone can be cleaned away before the consumer sees it. The consumer then never learns the key was deleted and keeps stale state. In Kafka Streams, an instance rebuilding its state store from a compacted changelog after a long outage can resurrect keys that should be gone, corrupting joins and aggregates.

Prevention:

• Set delete.retention.ms to at least the worst expected consumer downtime plus margin. For critical applications, a week or more is reasonable.
• Alert when any consumer group’s lag (in time) approaches the tombstone window.

See Debugging Log Compaction Issues and Tombstone Records.

Disk-Full Scenarios

When a log directory fills, Kafka does not enter a global read-only mode. On an IOException, the broker takes that specific log directory offline: it stops fetchers, checkpointing, compaction, and retention for logs on that directory, and reports the affected replicas as offline to the controller (KAFKA-8733 / KIP-112), which elects new leaders on healthy brokers. On older versions a full disk could halt the broker process outright; KIP-928 (Kafka 3.7+) makes brokers more resilient by keeping retention running on the offline directory so space can be reclaimed.

Immediate mitigation:

1. Find the largest partitions with kafka-log-dirs.
2. Temporarily lower retention.ms (or retention.bytes) on the offending topics to make segments eligible for deletion at the next check interval.
3. Only as a true last resort, and never on the active segment, manually remove old segment files—this risks corrupting the partition and should be done with the broker stopped.

# Emergency retention reduction to free disk space
kafka-configs.sh --bootstrap-server localhost:9092 \
  --entity-type topics --entity-name large-topic \
  --alter --add-config retention.ms=3600000
# Deletion happens on the next log.retention.check.interval.ms scan.

Long-term prevention:

• Alert on disk usage at ~70% and ~85%.
• Apply retention.bytes (or log.retention.bytes) as a ceiling so no single topic can consume an entire disk.
• Use tiered storage for topics that need long retention but have low read velocity (see below).

Retention, Compaction, and Data Governance

These policies are governance controls, not just operational knobs—they affect compliance, lineage, and cost attribution.

Compliance-Driven Retention

GDPR, CCPA, and PCI-DSS impose retention and deletion requirements. Time-based retention satisfies “delete after N days” at the topic level, but it cannot selectively erase a single key. Right-to-erasure on a keyed dataset is implemented by producing a tombstone to a compacted topic:

// Producing a tombstone for a GDPR erasure request
ProducerRecord<String, byte[]> tombstone = new ProducerRecord<>(
    "user-data",
    userId,        // key
    null           // null value = tombstone
);
producer.send(tombstone);

This only erases the value once the tombstone is compacted and then aged out per delete.retention.ms. For pure-delete topics you cannot target a single key; you must run a separate read-filter-republish pipeline, which is complex and error-prone.

Cost Attribution

In multi-tenant clusters, storage cost should follow topic ownership. Retention and compaction settings drive storage directly, so they belong in the chargeback model. Joining broker Size metrics (kafka.log:type=Log,name=Size) with topic configs lets you report per-team consumption:

-- Storage attribution (assuming metrics in a time-series store)
SELECT
    topic,
    SUM(segment_size_bytes) AS total_storage_bytes,
    AVG(retention_ms)       AS avg_retention_ms,
    cleanup_policy
FROM kafka_topic_metrics
WHERE timestamp > now() - interval '1 day'
GROUP BY topic, cleanup_policy
ORDER BY total_storage_bytes DESC;

Use this to drive a feedback loop so teams right-size their own retention.

Automated Policy Enforcement

Manual review doesn’t scale. Run a continuous audit that compares live topic configs against governance rules and alerts (or remediates) on drift:

# Simplified policy audit using kafka-python's admin client
from kafka.admin import KafkaAdminClient, ConfigResource, ConfigResourceType

admin = KafkaAdminClient(bootstrap_servers="localhost:9092")

MAX_RETENTION_MS = 7776000000  # 90 days

for topic in admin.list_topics():
    resource = ConfigResource(ConfigResourceType.TOPIC, topic)
    described = admin.describe_configs([resource])
    configs = described[0].resources[0][4]
    retention = next(
        (int(c[1]) for c in configs if c[0] == "retention.ms" and c[1] not in (None, "")),
        -1,
    )
    if retention > MAX_RETENTION_MS:
        print(f"VIOLATION: {topic} retention.ms={retention} exceeds max {MAX_RETENTION_MS}")
        # To remediate: admin.alter_configs(...) with the corrected value.

Run it as a cron job or inside a Kubernetes operator. (The exact describe_configs return shape depends on the client library version—confirm it against your client before relying on the indices above.)

Advanced Patterns and Edge Cases

Compaction After Partition Expansion

Compaction is per-partition, so changing partition count breaks the “one value per key” expectation. A topic grown from 8 to 16 partitions re-hashes keys: a key that mapped to partition 3 may now map to partition 11, leaving its old value compacted in partition 3 and its new value in partition 11. A full-topic reader sees two values for that key. Either avoid expanding compacted topics, or add an application-level last-write-wins dedup, and design keys per Keying Strategies for Ordered Message Processing.

Long Retention with Tiered Storage

For topics needing very long retention without paying for local broker disk, Kafka’s tiered storage (KIP-405) offloads older, “cold” segments to object storage while keeping recent “hot” segments local. It reached early access in Kafka 3.6 and was marked production-ready in Kafka 3.9. In Apache Kafka (OSS) it is configured with remote.log.storage.system.enable=true at the broker, remote.storage.enable=true per topic, and local.retention.ms / local.retention.bytes to bound the local hot set:

# Broker
remote.log.storage.system.enable=true
remote.log.storage.manager.class.name=<your RemoteStorageManager implementation>

# Topic
remote.storage.enable=true
local.retention.ms=86400000        # keep 1 day locally; older segments live in remote storage
retention.ms=-1                    # total retention (here, unbounded)

OSS Kafka ships no built-in object-store RemoteStorageManager—you supply one (e.g. the Aiven open-source plugin) or use a managed platform. The confluent.tier.* keys (confluent.tier.enable, confluent.tier.backend, etc.) are Confluent Platform-specific and are not valid in Apache Kafka.

Critically, Apache Kafka tiered storage does not support compacted topics: a tiered topic’s cleanup.policy must be delete, and you cannot change it to compact afterward. So “infinite retention plus compaction via tiered storage” is not achievable in OSS today—pick tiering or compaction per topic. (Adding compaction support to tiered storage is tracked in KIP-1272.)

Compaction and Exactly-Once Semantics

EOS relies on the idempotent producer and the transactional API. The cleaner never cleans past the last stable offset, so a topic with an open transaction cannot be compacted beyond that point. A hung transaction—producer crashed without committing or aborting—pins the LSO and lets a compacted partition grow unbounded until the transaction is resolved. Watch kafka.server:type=ReplicaManager,name=... LSO-lag style metrics, and use kafka-transactions.sh (KIP-664) to find and abort hanging transactions.

# Transaction-state and timeout settings relevant to compacted topics
transaction.timeout.ms=900000
transaction.state.log.replication.factor=3
transaction.state.log.min.isr=2

transaction.timeout.ms (here 15 minutes) bounds how long the coordinator waits before aborting an idle transaction; a cleanly-aborted transaction unblocks the LSO and lets compaction resume. A truly hung transaction may still need manual intervention via kafka-transactions.sh.

Multi-Cluster Replication

MirrorMaker 2 replicates topic configs only when you ask it to. Replicating a compacted source topic to a target left at cleanup.policy=delete lets the target grow without ever deduplicating keys. Keep policies aligned by enabling config sync:

# MirrorMaker 2: preserve cleanup and retention policy on the target
replication.policy.class=org.apache.kafka.connect.mirror.IdentityReplicationPolicy
sync.topic.configs.enabled=true

sync.topic.configs.enabled=true propagates topic configurations (including cleanup.policy) to the target. Use config.properties.exclude to suppress any properties you do not want mirrored.

Conclusion

Retention and compaction are foundational primitives that shape your cluster’s behavior, cost, and reliability. delete gives a predictable, time-bounded window of full history—right for audit trails and temporal analytics. compact turns Kafka into a durable, eventually-consistent key-value store, enabling stateful stream processing. compact,delete combines bounded history with latest-per-key snapshots.

Operating these at scale means monitoring the cleaner’s max-dirty-percent, time-since-last-run-ms, and uncleanable-partitions-count; guarding tombstone windows against consumer downtime; understanding that a full disk takes a log directory offline rather than the whole broker; and remembering the hard edges—per-partition-only compaction guarantees, the LSO boundary, and the fact that OSS tiered storage and compaction are mutually exclusive per topic. Fold these into your provisioning pipelines and dashboards, and retention and compaction become predictable infrastructure rather than recurring toil.