Configuring Retention Policies for Compliance and Cost Control

Apache Kafka’s ability to store and replay event streams is one of its most useful properties, but that storage is not free. Every byte retained on a broker consumes disk, page cache, and replication bandwidth, and ultimately your infrastructure budget. For platform and SRE engineers, retention is not just a topic setting—it is the lever that balances legal mandates, data availability, and storage cost. Misconfigured retention can quietly violate a compliance order by deleting data too soon, or fill your disks and stall production because data was kept too long.

This guide takes a code-driven approach to configuring retention that satisfies both regulatory requirements and cost constraints. We move past the basic retention.ms knob into time-based, size-based, and hybrid strategies, operationalize them with configuration management, and add monitoring to catch drift. Because retention is tightly coupled to the physical layout of your data, we connect these concepts to the broader discipline of Topics, Partitions & Data Modeling.

The Retention Primitives: Time, Size, and the Log Segment

Before designing a policy, understand the mechanisms that govern data removal in Kafka. The two deletion limits operate concurrently, and the first one reached triggers deletion—but both act on whole log segments, not individual messages.

Time-based retention (retention.ms) controls how long a message may remain in a topic after it is written. The broker checks for eligible segments on a schedule set by log.retention.check.interval.ms (default 300000 ms, i.e. 5 minutes). The key operational detail is that retention is enforced against a segment, not each message: an inactive (closed) segment is deleted only once the timestamp of its newest message is older than retention.ms. The currently active segment is never deleted. So a low-traffic topic with a 1-hour retention.ms and a 1 GB segment can retain data far longer than an hour if the segment never closes.

Size-based retention (retention.bytes) sets the maximum total size of log segments retained in a partition. This is a per-partition limit, not a topic-wide aggregate. If you set retention.bytes to 10 GB on a topic with 10 partitions, the topic can hold up to roughly 100 GB before size-based deletion begins. Size-based retention is essential for cost control because it puts a hard ceiling on storage per partition, independent of throughput.

Log segment management is the engine behind both policies. Kafka deletes whole segments, never individual records. Segment size is controlled by segment.bytes (default 1073741824, i.e. 1 GB) and segment age by segment.ms (default 604800000 ms, i.e. 7 days). A segment becomes a candidate for deletion only after it is closed (rolled), which happens when it reaches the size limit or the time limit. This segment-oriented model is the root of most retention surprises, especially in low-throughput topics.

While this article focuses on deletion-based retention, the same segment machinery underlies log compaction. For when compaction is the right tool instead, see Retention vs Compaction: When to Use Each.

Segment Rolling and Retention Accuracy

Segment rolling directly determines how precisely your time limit is honored. Consider a topic with retention.ms=3600000 (1 hour) and segment.bytes=1073741824 (1 GB). If the topic receives only 10 MB per day, the active segment never reaches 1 GB, so it stays open until segment.ms (default 7 days) forces it to roll. Until that roll happens, the broker will not delete any data, even though the oldest records are well past the 1-hour mark. Your 1-hour retention has silently become a 7-day retention.

To bound this, tune segment.ms to match your retention goal. A practical pattern is to set segment.ms to a fraction of retention.ms so segments close often enough for timely deletion.

# Create a topic with tight segment management for accurate time-based retention
kafka-topics.sh --bootstrap-server localhost:9092 \
  --create --topic compliance-events \
  --partitions 12 --replication-factor 3 \
  --config retention.ms=86400000 \
  --config segment.ms=3600000 \
  --config segment.bytes=536870912

Here segment.ms is 1 hour (3,600,000 ms), one twenty-fourth of the 24-hour retention period. Even under low throughput, segments close hourly and become eligible for deletion shortly after their newest message ages past the retention boundary. The trade-off: smaller, more frequent segments mean more open file handles and index files, so do not push segment.ms to seconds in production without testing.

Designing Retention Policies for Compliance

Compliance adds constraints beyond cost. Regulations such as GDPR, HIPAA, PCI DSS, and SOX impose data-lifecycle rules: minimum retention for auditability, maximum retention for privacy, and deletion guarantees. Kafka can satisfy these, but only if you model them against the segment behavior above. Note that Kafka’s broker-side deletion gives you a time/size ceiling, not record-level erasure of a specific key—important context for “right to erasure” obligations.

Minimum Retention Guarantees

A minimum retention period keeps data available for replay or audit for at least a specified duration. Because the active segment is never deleted and closed segments are only deleted once their newest record passes retention.ms, time-based retention naturally guarantees a minimum: data is never removed early. To make the floor exact rather than open-ended, pair retention.ms with a segment.ms small enough that segments close and age out promptly.

# Topic with a 90-day minimum retention for financial audit compliance
kafka-configs.sh --bootstrap-server localhost:9092 \
  --entity-type topics --entity-name financial-ledger \
  --alter --add-config \
  retention.ms=7776000000,segment.ms=86400000

If you later enable compaction on this topic, min.cleanable.dirty.ratio (default 0.5) controls how dirty a log must be before the cleaner runs. It has no effect on a pure cleanup.policy=delete topic, so do not set it preemptively—configure it only when you actually switch the topic to compact.

Maximum Retention and Deletion Deadlines

GDPR’s “right to erasure” and similar rules require that data not be retained beyond a defined period. Enforcing a maximum age ceiling with broker retention is harder than a minimum, because of the open-segment problem: under a throughput drop, the active segment can stay open and unexpired past your intended ceiling. Two practical mitigations:

  1. Keep segment.ms short relative to the ceiling so the active segment rolls on a predictable schedule rather than waiting on volume.
  2. Add retention.bytes as a backstop so that, even if time-based deletion lags, a full partition forces older segments out.
# Topic with combined time-and-size limits
kafka-topics.sh --bootstrap-server localhost:9092 \
  --create --topic user-sessions \
  --partitions 6 --replication-factor 3 \
  --config retention.ms=2592000000 \
  --config retention.bytes=10737418240 \
  --config segment.ms=43200000 \
  --config cleanup.policy=delete

Here retention.ms is 30 days (2,592,000,000 ms), retention.bytes is 10 GB per partition, and segment.ms is 12 hours so the active segment cannot stay open indefinitely. The size limit is a safety valve: if a spike fills a partition to 10 GB in 10 days, its oldest segments are deleted even though the 30-day time limit has not been reached. Be explicit with your legal team that broker retention enforces a coarse, segment-granular ceiling and cannot erase one specific record on demand; for that, you need key-level erasure (e.g. crypto-shredding or a compacted tombstone topic) on top of broker retention.

Irreversible Deletion and Consumer Safety

Kafka’s deletion is irreversible: once a segment is removed, it cannot be recovered through Kafka APIs. That means you must ensure downstream systems have consumed the data before it ages out. Monitor consumer lag and alert when a group’s lag approaches the retention boundary; if a consumer falls behind and data is deleted before it is processed, you have both a data-loss and a compliance gap.

For sensitive audit trails, use a two-tier strategy: a hot Kafka topic with short retention for operational consumers, plus a cold sink (S3, HDFS) for long-term archival. The Kafka Connect S3 sink connector continuously offloads records to object storage, where bucket lifecycle rules, object lock, and legal holds provide immutability guarantees Kafka itself does not offer.

Cost Control Through Tiered Retention

Storage dominates Kafka infrastructure cost, and retention is the most direct way to control it. The savings depend entirely on your current settings and access patterns—topics that were defaulted to a 7-day retention but only need 24 hours are where the easy wins are. Start by auditing actual per-topic disk usage against business need rather than assuming a fixed percentage.

Time-Size Hybrid Policies

The most effective cost-control pattern combines time and size. Time-based retention gives a predictable availability window; size-based retention caps growth during traffic anomalies. The broker enforces whichever is hit first.

# Hybrid policy for a high-throughput telemetry topic
kafka-configs.sh --bootstrap-server localhost:9092 \
  --entity-type topics --entity-name sensor-readings \
  --alter --add-config \
  retention.ms=604800000,retention.bytes=53687091200,segment.bytes=1073741824

For a topic with 20 partitions, this allows up to ~1 TB total (50 GB × 20) but caps data age at 7 days. In steady state the 7-day limit usually triggers first; during a surge, the size limit stops the topic from consuming all available disk.

Differentiated Retention by Topic Class

Not all topics deserve the same retention. Classify topics into tiers by criticality and access pattern:

  • Tier 1 (critical / compliance): long retention (30–90 days), high replication factor, size limits set generously above steady-state volume.
  • Tier 2 (operational): medium retention (3–7 days), standard replication, size limits modestly above expected volume.
  • Tier 3 (transient / debug): short retention (1–24 hours), minimal replication, aggressive size limits.

Express this as code with your infrastructure-as-code tooling. With Terraform and the Mongey/terraform-provider-kafka provider, topic config values are strings:

resource "kafka_topic" "tier1_audit" {
  name               = "audit-events"
  partitions         = 12
  replication_factor = 3
  config = {
    "retention.ms"    = "7776000000"   # 90 days
    "retention.bytes" = "107374182400" # 100 GB per partition
    "segment.ms"      = "86400000"     # 1 day
    "cleanup.policy"  = "delete"
  }
}

resource "kafka_topic" "tier3_debug" {
  name               = "debug-logs"
  partitions         = 4
  replication_factor = 1
  config = {
    "retention.ms"    = "3600000"      # 1 hour
    "retention.bytes" = "1073741824"   # 1 GB per partition
    "segment.ms"      = "600000"       # 10 minutes
    "cleanup.policy"  = "delete"
  }
}

Monitoring Storage and Retention Drift

Configuration is not enough; monitor actual disk usage and retention behavior. The relevant per-partition JMX MBeans exposed by the broker are:

  • kafka.log:type=Log,name=Size,topic=<topic>,partition=<partition> — current on-disk size of the partition’s segments, in bytes.
  • kafka.log:type=Log,name=LogEndOffset,topic=<topic>,partition=<partition> — offset just past the last message.
  • kafka.log:type=Log,name=LogStartOffset,topic=<topic>,partition=<partition> — earliest offset still retained.
  • kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec,topic=<topic> — ingress throughput (a Meter; use its rate attributes).

A rising LogStartOffset is the direct signal that retention is deleting data, and the gap between LogStartOffset and LogEndOffset is the number of records currently retained. To estimate the time span retained, divide partition Size (bytes) by the ingress rate (bytes/sec): the result is the approximate age of the data on disk. If that estimate diverges from your configured retention.ms by more than your tolerance, investigate segment.ms and log.retention.check.interval.ms.

The following Prometheus rule assumes the standard JMX exporter naming, where kafka.log:type=Log,name=Size becomes kafka_log_log_size and BytesInPerSec becomes a ..._total counter you rate. It is dimensionally consistent—bytes divided by bytes/sec yields seconds:

groups:
  - name: kafka_retention
    rules:
      - alert: KafkaRetentionDrift
        expr: |
          kafka_log_log_size
            / on(topic, partition)
          (sum by(topic, partition) (
             rate(kafka_server_brokertopicmetrics_bytesinpersec_total[1h])
           ) > 0)
          > 2 * 7 * 86400
        for: 1h
        labels:
          severity: warning
        annotations:
          summary: "Topic {{ $labels.topic }} retains ~{{ $value | humanizeDuration }} of data, above the 7-day target"

This divides on-disk bytes by the recent ingress byte rate to estimate the retained time span in seconds, firing when it exceeds twice the 7-day target. Tune the metric label names to your exporter (they vary), the lookback window, and the threshold per tier. Because the estimate uses a recent rate, it is sensitive to throughput changes—treat it as a drift signal, not a precise measurement.

Operationalizing Retention at Scale

Managing retention across hundreds of topics needs automation, validation, and governance. Hand-editing with kafka-configs.sh does not scale and invites drift.

Configuration as Code with GitOps

Store topic configurations in version-controlled YAML or HCL and apply them through CI/CD. A typical pipeline validates against a schema, runs a dry-run diff against current cluster state, and applies changes only after review. If you run on Kubernetes, the Strimzi operator’s KafkaTopic custom resource gives you exactly this declarative model:

# topic-configs/compliance/audit-events.yaml
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaTopic
metadata:
  name: audit-events
  labels:
    strimzi.io/cluster: my-cluster
    tier: "1"
    compliance: "sox"
spec:
  partitions: 12
  replicas: 3
  config:
    retention.ms: "7776000000"
    retention.bytes: "107374182400"
    segment.ms: "86400000"
    cleanup.policy: "delete"
    min.insync.replicas: "2"

A pre-commit or CI check can enforce policy invariants in code—for example, that segment.ms is at most 25% of retention.ms on Tier 1 topics, so the segment-rolling best practice is applied automatically rather than relying on memory.

Capacity-Aware Placement with Cruise Control

For clusters running Apache Kafka Cruise Control, retention feeds directly into capacity planning. Cruise Control’s goal-based optimization includes a disk-capacity goal; when you size broker disk capacity against the storage that your retention settings imply, its rebalancing keeps partition placement within those limits and its anomaly detectors can flag brokers trending toward disk exhaustion.

Cruise Control’s load monitor also gives you per-broker storage trends. By projecting ingress against retention limits you can forecast disk pressure days in advance and act—shifting partitions, or temporarily lowering retention.bytes on non-critical topics—before a broker fills up.

Retention During Cluster Expansion

When you add brokers, partition reassignment moves replica data to the new nodes. A moving replica is rebuilt from the source, so reassignment can transiently increase total disk usage across the cluster and adds replication traffic. Throttle the movement with kafka-reassign-partitions.sh so it does not saturate the network or fill disks:

# Throttled partition reassignment during expansion
kafka-reassign-partitions.sh --bootstrap-server localhost:9092 \
  --reassignment-json-file expand-reassign.json \
  --throttle 52428800 \
  --execute

The --throttle value is the maximum replication bandwidth in bytes per second; 50 MB/s (52,428,608 ≈ 52428800) is a conservative start for HDD-backed clusters. After the move completes, run the same tool with --verify to confirm completion and remove the throttle quota, and watch disk utilization on source and target brokers throughout.

Testing and Validating Retention Behavior

Retention bugs are insidious because they surface slowly—often only when a disk fills or an auditor finds missing data. Test proactively.

Deterministic Retention Testing

Create a test topic with accelerated segment settings, produce a known volume, and verify that the earliest retained offset advances on schedule.

# Create a test topic with 2-minute retention and 30-second segments
kafka-topics.sh --bootstrap-server localhost:9092 \
  --create --topic retention-test \
  --partitions 1 --replication-factor 1 \
  --config retention.ms=120000 \
  --config segment.ms=30000 \
  --config segment.bytes=102400

# Produce records at a steady, throttled rate
kafka-producer-perf-test.sh \
  --topic retention-test \
  --num-records 1000 \
  --record-size 1024 \
  --throughput 10 \
  --producer-props bootstrap.servers=localhost:9092

# Wait a few minutes, then read the EARLIEST retained offset (--time -2)
kafka-get-offsets.sh \
  --bootstrap-server localhost:9092 \
  --topic retention-test --time -2

# After retention kicks in, the earliest offset should be greater than 0,
# reflecting that the oldest segments have been deleted.

kafka-get-offsets.sh (added in Kafka 3.0 by KIP-635 as the wrapper around GetOffsetShell) takes --time -2 for the earliest offset and --time -1 for the latest. On older clusters without the wrapper script, the equivalent is kafka-run-class.sh kafka.tools.GetOffsetShell --bootstrap-server localhost:9092 --topic retention-test --time -2. Automate this check against a staging cluster; if the observed retention window deviates from the configured value beyond your tolerance, fail the pipeline and investigate segment.ms and log.retention.check.interval.ms.

Chaos Engineering for Retention

Inject faults that stress the deletion machinery: fill a disk toward capacity and confirm retention.bytes deletes segments before the broker hits a log.dir failure; pause a broker JVM with SIGSTOP (note this halts the entire process, not just the retention thread—useful for verifying that a stalled broker is detected and alerts fire, not for isolating the cleaner); and simulate a consumer-lag spike to confirm monitoring flags the data-loss risk before deletion occurs.

For the precise semantics of the retention check interval and how it interacts with segment rolling, see the Apache Kafka broker configuration docs. Understanding these internals is what makes a chaos experiment meaningful rather than theater.

Integrating Retention with the Data Lifecycle

Retention does not exist in isolation; it interacts with partitioning, keying, and schema evolution. A topic with 100 partitions and a 10 GB per-partition retention.bytes can hold ~1 TB, but if your key causes skew, hot partitions hit the size limit and start deleting while cold partitions sit nearly empty. The result is premature, uneven deletion that breaks the time-based guarantee for a subset of keys.

Partitioning and Retention Symmetry

Align partitioning with retention goals. If you need uniform retention across keys, ensure the partitioner distributes data evenly; for high-volume keys, consider over-partitioning or a custom partitioner. The Topics, Partitions & Data Modeling guide covers these patterns in depth.

Schema Evolution and Replay Windows

When you evolve schemas, account for the retention period of existing topics. If consumers must replay old records after a schema change, those records must remain available and deserializable. That means retention must exceed your maximum replay window, and your schema registry must use a compatibility mode (e.g. BACKWARD) that lets new code read old records.

For stateful applications built on log compaction rather than deletion, the calculus differs: a compacted topic keeps at least the latest value per key, with different cost and compliance implications. That trade-off is covered in Retention vs Compaction: When to Use Each.

End-to-End Lifecycle Automation

A mature platform automates the full topic lifecycle. When a topic reaches end of life—say its upstream service is deprecated—an automated workflow should:

  1. Verify all registered consumers have stopped and committed their offsets.
  2. Drain the topic by setting retention.ms to a short grace value (e.g. 1 hour).
  3. Export any remaining data to cold storage if compliance requires it.
  4. Delete the topic and remove its config from the GitOps repository.

Streaming pipelines built on Kafka Connect, or a system like LinkedIn’s Brooklin, can orchestrate the drain-and-archive step so no data is lost and storage is reclaimed cleanly.

Conclusion

Retention for compliance and cost is a continuous engineering discipline, not a one-time setup. It rests on three things: a real understanding of Kafka’s segment-oriented deletion model (especially the open-segment trap on low-traffic topics), disciplined configuration management, and monitoring that catches drift before it becomes an incident. Combine time and size limits, differentiate by topic tier, and enforce policy through GitOps and capacity-aware tooling, and you can satisfy auditors and finance at the same time.

Adapt these patterns to your specific regulatory environment and throughput. Start from the examples here, instrument your clusters with the MBeans above, and iterate on observed behavior. Retention is a dial, not a switch—tune it deliberately.

Back to Retention vs Compaction When To Use Each