Tiered Storage in Apache Kafka: Architecture, Configuration, and Operations

Apache Kafka’s traditional storage model couples compute and storage on local broker disks: every byte in a topic’s retention window lives on the brokers that host its partitions. As retention grows, the cost of provisioning local disk for the full window becomes the dominant line item, and broker recovery times stretch because rebuilding a replica means re-replicating the entire log. Tiered storage, introduced as Early Access in Kafka 3.6 (KIP-405) and declared production-ready (GA) in Kafka 3.9, addresses this by offloading older log segments to a remote object store while keeping the active working set on local disk. The result: long retention at object-store prices, smaller broker disks, and faster broker recovery and reassignment.

This article covers the architecture, verified production configuration, capacity planning, monitoring with the real JMX metrics, and troubleshooting. All configuration keys, class names, and metric names below are taken from the Apache Kafka documentation and KIP-405/KIP-963.

Architecture

Tiered storage introduces a two-level hierarchy for each partition’s log:

1. Local tier: Broker disk (typically NVMe SSD) holding active and recently sealed segments — the working set most consumers read.
2. Remote tier: An object store (S3, GCS, Azure Blob, MinIO) holding older, sealed segments offloaded according to per-topic retention.

This is not a backup mechanism. It is an integrated storage layer that preserves Kafka’s per-partition ordering and is transparent to clients: a consumer fetches by offset and never knows whether a segment was served from local disk or the object store.

Core Components

RemoteLogManager (RLM) — A broker component that manages the lifecycle of remote log segments. When a segment is rolled and sealed and falls outside the local retention window, the RLM copies it to the remote store and records metadata. The broker keeps the lightweight offset and timestamp indexes needed to locate remote segments, so it can serve historical fetches transparently over the standard Kafka protocol.

RemoteStorageManager (RSM) — The pluggable interface (org.apache.kafka.server.log.remote.storage.RemoteStorageManager) for the object-store backend. Kafka does not ship a built-in RSM implementation — this is a deliberate KIP-405 design decision to keep the storage backend out of the broker. You supply one via remote.log.storage.manager.class.name. The most widely used open-source implementation is Aiven’s tiered-storage-for-apache-kafka, whose class is io.aiven.kafka.tieredstorage.RemoteStorageManager and which supports S3, GCS, and Azure Blob backends.

RemoteLogMetadataManager (RLMM) — Stores metadata about remote segments (object location, size, offset and timestamp ranges, leader epoch). Kafka ships a default, topic-based implementation, org.apache.kafka.server.log.remote.metadata.storage.TopicBasedRemoteLogMetadataManager, which persists metadata in the internal, replicated topic __remote_log_metadata. Because this topic uses Kafka’s own replication, the metadata inherits Kafka’s durability guarantees.

Segment Lifecycle

A partition’s log is a sequence of segments. The active segment rolls when it reaches segment.bytes or segment.ms. Once sealed, a segment becomes eligible for offloading based on local retention; deletion from the remote tier is governed by total retention:

• Local retention (log.local.retention.ms / log.local.retention.bytes at the broker, or local.retention.ms / local.retention.bytes per topic): how long (or how many bytes of) sealed data stay on local disk before being eligible for offload. Both default to -2, which means “inherit the total retention” (log.retention.ms / retention.bytes). With the default, local retention equals total retention, so nothing is offloaded — you must set local retention below total retention to actually tier data.
• Total retention (retention.ms / retention.bytes): the overall lifetime of data. Once a segment ages past this, it is deleted everywhere, including the remote tier. There is no separate remote.log.retention.* property; remote retention is simply total retention minus the portion still held locally.

The RLM copies an eligible segment to the remote store, commits its metadata to the RLMM, and only then is the local segment file eligible for deletion. Copy is verified before the local copy is removed, so a failed upload does not lose data.

Fetch Path for Remote Data

When a consumer requests an offset that has aged out of local disk, the broker reads the segment from the object store and streams the requested records to the consumer. A dedicated reader thread pool (remote.log.reader.threads) handles these reads, with a bounded backlog (remote.log.reader.max.pending.tasks). The broker caches the remote index files (offset and timestamp indexes) locally so it does not re-download them on every fetch; that cache is sized by remote.log.index.file.cache.total.size.bytes (default 1 GiB). Note that one FetchRequest is served from remote storage for only a single partition at a time.

Configuration

Enabling tiered storage is a broker-level change applied to every broker, combined with per-topic opt-in. The broker properties below assume the Aiven RSM plugin with an S3-compatible backend (MinIO here); consult your RSM’s own documentation for its exact rsm.config.* keys, as they are defined by the plugin, not by Kafka.

Broker-Level Configuration

Set the following in server.properties on every broker:

# Enable tiered storage on the broker
remote.log.storage.system.enable=true

# Pluggable remote storage backend (Aiven RSM shown; Kafka ships no built-in RSM)
remote.log.storage.manager.class.name=io.aiven.kafka.tieredstorage.RemoteStorageManager
remote.log.storage.manager.impl.prefix=rsm.config.
rsm.config.storage.backend.class=io.aiven.kafka.tieredstorage.storage.s3.S3Storage
rsm.config.storage.s3.bucket.name=kafka-tiered-storage
rsm.config.storage.s3.region=us-east-1
rsm.config.storage.s3.endpoint.url=http://minio.example.com:9000
rsm.config.storage.s3.path.style.access.enabled=true

# Default topic-based metadata manager
remote.log.metadata.manager.class.name=org.apache.kafka.server.log.remote.metadata.storage.TopicBasedRemoteLogMetadataManager
remote.log.metadata.manager.impl.prefix=rlmm.config.
remote.log.metadata.manager.listener.name=PLAINTEXT
rlmm.config.remote.log.metadata.topic.replication.factor=3
rlmm.config.remote.log.metadata.topic.num.partitions=50

# Cluster-wide default local retention: keep 24h on local disk
log.local.retention.ms=86400000

Key points:

• remote.log.storage.system.enable=true activates the feature; it defaults to false. This is read at startup, so changing it requires a rolling restart.
• remote.log.storage.manager.class.name defaults to null. You must point it at an RSM implementation on the classpath — there is no built-in S3 manager.
• remote.log.metadata.manager.class.name already defaults to TopicBasedRemoteLogMetadataManager; it is shown explicitly here for clarity. remote.log.metadata.manager.listener.name is mandatory and names the broker listener the RLMM’s internal clients use to reach the cluster.
• log.local.retention.ms sets the cluster default for how long sealed data stays local before offload. It defaults to -2 (inherit total retention), which disables tiering — you must lower it.
• The rsm.config.* and rlmm.config.* prefixes (set by *.impl.prefix) pass implementation-specific configuration through to the RSM and RLMM. Avoid embedding static S3 keys here in production; prefer instance roles (see Securing Remote Storage Access).

Topic-Level Configuration

Tiered storage is opt-in per topic via remote.storage.enable=true. Retention overrides let you tune local versus total retention per workload:

kafka-configs.sh --bootstrap-server localhost:9092 \
  --entity-type topics \
  --entity-name events.payments \
  --alter \
  --add-config remote.storage.enable=true,local.retention.ms=21600000,retention.ms=7776000000

This enables tiering for events.payments, keeps 6 hours of data on local disk, and retains data for 90 days total (the remainder living in the object store). Topic overrides take precedence over broker defaults and are applied dynamically without a broker restart.

Note: enabling tiered storage on a topic is straightforward; disabling it (remote.storage.enable=false) is a distinct operation governed by remote.log.disable.policy and was added in 3.9 via KIP-950. Plan disablement deliberately rather than assuming it is a simple toggle.

Verifying the Setup

After a rolling restart with the configuration applied:

1. Confirm the metadata topic was auto-created:

  kafka-topics.sh --bootstrap-server localhost:9092 \
    --describe --topic __remote_log_metadata

2. Check broker logs for RemoteLogManager startup and the absence of RSM init errors.
3. Produce to a tiered topic, wait past the local retention window, and confirm objects appear in the bucket and that RemoteLogSizeBytes for the topic is non-zero.

For a hands-on walkthrough including IAM policies and bucket lifecycle rules, see the companion guide on Enabling Tiered Storage with S3 and MinIO: A Practical Guide.

Hardware and Capacity Planning

Tiered storage changes the inputs to Broker Sizing and Hardware Selection for Kafka Clusters. Local disk now sizes to the working set plus an offload buffer, not the full retention window.

Sizing Local Storage

Local disk must hold:

• Active and sealed segments within the local retention window
• A buffer for segments queued for upload (copy lag)
• Remote index caches and Kafka/OS overhead

A practical upper bound:

LocalDiskGB ≈ (IngressMBps × LocalRetentionSeconds / 1024) × ReplicationFactor × 1.2

The 1.2 multiplier covers index files, copy-lag buffer, and headroom. For a broker ingesting 50 MB/s with 24-hour local retention and RF=3:

(50 × 86400 / 1024) × 3 × 1.2 ≈ 15,188 GB

This is a worst case. In steady state, offload runs continuously as segments age past local retention, so utilization sits below this figure — provided the copy pipeline keeps up (monitor RemoteCopyLagBytes).

Instance Type Selection

With local storage bounded by the working set, you can shift from storage-dense to compute-optimized instances. On AWS, replacing i3en.6xlarge (2×7,500 GB NVMe) with c6id.8xlarge (1×1,900 GB NVMe) trades bulk local disk for compute while the object store absorbs long-tail retention at a far lower cost per GB. Validate the savings against your own ingress and retention; the win comes from no longer paying SSD prices for cold data.

Tiering also improves elasticity: adding or replacing a broker only re-replicates segments still held locally, not the full window, so reassignment and recovery complete faster.

Operational Best Practices

Monitoring the Offload Pipeline

If offload falls behind, local disk fills and brokers eventually stop accepting produce on affected partitions. The metrics below are the real JMX MBeans from KIP-405/KIP-963; the per-topic ones live under kafka.server:type=BrokerTopicMetrics with a topic= key (aggregate across topics by omitting it):

Alert on sustained growth in RemoteCopyLagBytes and on any RemoteCopyErrorsPerSec. A persistently low RemoteLogManagerTasksAvgIdlePercent means the copy pool is the bottleneck — increase remote.log.manager.copier.thread.pool.size (see Performance Tuning).

Example Prometheus alert (metric name as exported by the JMX exporter, lowercased):

groups:
  - name: kafka_tiered_storage
    rules:
      - alert: KafkaRemoteCopyLagGrowing
        expr: sum(kafka_server_brokertopicmetrics_remotecopylagbytes) > 5368709120
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "Kafka tiered storage copy lag is growing"
          description: "Pending-upload backlog has exceeded 5 GiB for 10 minutes; local disk may fill."

(The exact exported metric name depends on your JMX exporter rules; verify it against /metrics rather than assuming.)

Securing Remote Storage Access

1. Instance roles (AWS): Attach an IAM role to broker instances rather than embedding static keys. Most RSM implementations honor the default AWS credential chain (instance profile, env vars, profile file).
2. Kubernetes: Mount credentials via a Secret or use IRSA / Workload Identity so brokers assume a role without static keys.
3. Credential rotation: Where static credentials are unavoidable, automate rotation. Credential-provider configuration is defined by your RSM plugin’s rsm.config.* keys, not by Kafka core.

Handling Remote Store Outages

If the object store is unavailable, fetches for data older than the local window fail until it recovers. Mitigations:

• Provision the object store for high availability (e.g., S3’s multi-AZ durability; cross-region replication for DR).
• Bound concurrent remote reads with remote.log.reader.max.pending.tasks so a slow store cannot exhaust reader threads and stall local fetches.
• Replicate __remote_log_metadata across racks/AZs (use a rack-aware replica placement). Losing this topic renders all remote data unlocatable.

Pre-Production Validation

Exercise these in staging before enabling tiering in production:

1. Offload and read-back: Produce, wait past local retention, confirm objects in the bucket, then consume from earliest and verify the full window reads back.
2. Broker failure and recovery: Kill and restart a broker; confirm it re-reads remote metadata and resumes serving historical offsets.
3. Partition reassignment: Move partitions and verify the new leader references the existing remote segments rather than re-uploading.

kafka-verifiable-consumer.sh \
  --bootstrap-server localhost:9092 \
  --topic events.payments \
  --group-id test-tiered-storage \
  --max-messages 1000 \
  --reset-policy earliest \
  --verbose

Integration with KRaft

Tiered storage is orthogonal to the metadata consensus protocol, but __remote_log_metadata is an ordinary Kafka topic and benefits from the cluster’s normal replication and leader election. In KRaft vs ZooKeeper: Choosing the Right Metadata Management terms, this topic’s partitions are still led by brokers and managed via the standard log layer in both modes; what differs is where cluster metadata itself lives.

In KRaft mode the controller quorum holds cluster metadata, removing the ZooKeeper dependency and reducing operational surface area. Size the __remote_log_metadata topic’s replication factor for durability (commonly 3) independent of the controller count. As of Kafka 4.0, KRaft is the only supported mode and ZooKeeper has been removed, so new clusters running tiered storage are KRaft by default.

Performance Tuning

Offload Parallelism

Copy and expiration run on separate, configurable thread pools. The single combined pool (remote.log.manager.thread.pool.size, default 2) was deprecated in 4.2; tune the dedicated pools instead:

remote.log.manager.copier.thread.pool.size=10
remote.log.manager.expiration.thread.pool.size=10
remote.log.manager.task.interval.ms=30000

copier.thread.pool.size (default 10) bounds concurrent uploads; expiration.thread.pool.size (default 10) bounds remote deletions; task.interval.ms (default 30000) is how often the RLM scans partitions for work. Raise the copier pool when RemoteCopyLagBytes grows and RemoteLogManagerTasksAvgIdlePercent is low, watching RemoteCopyErrorsPerSec for object-store throttling as you scale up.

Remote Read Throughput

Remote fetches are served by remote.log.reader.threads (default 10) with a bounded queue of remote.log.reader.max.pending.tasks (default 100). Increase the reader thread count for workloads that frequently replay cold data (e.g., backfills); keep the pending-task bound modest so a slow store sheds load rather than queueing unboundedly. The remote index cache (remote.log.index.file.cache.total.size.bytes, default 1 GiB) avoids re-downloading offset/timestamp indexes on repeated reads.

Segment Size

segment.bytes sets offload granularity. Smaller segments offload sooner but multiply remote object and metadata counts; larger segments cut metadata overhead but increase per-object copy time and copy lag. 256–512 MB is a common balance:

kafka-configs.sh --bootstrap-server localhost:9092 \
  --entity-type topics \
  --entity-name events.payments \
  --alter \
  --add-config segment.bytes=268435456

Monitoring Remote Read Latency and Saturation

Remote reads are slower than local reads; watch throughput and reader-pool saturation:

If RemoteLogReaderTaskQueueSize climbs and RemoteLogReaderAvgIdlePercent falls, you are read-bound: add reader threads, or widen the local retention window so latency-sensitive consumers stay within local data and never hit the object store. Sizing local retention to cover typical consumer lag is the most effective latency control.

Troubleshooting

Offload Stalls

Symptom: Local disk grows on tiered topics; RemoteCopyLagBytes shows a steady backlog.

Diagnosis: Check broker logs for RSM errors and inspect RemoteCopyErrorsPerSec. Common causes:

• Incorrect object-store credentials or insufficient IAM permissions
• Network/endpoint reachability to the object store
• Object-store throttling (S3 request-rate limits)

Resolution: Fix credentials/network, then increase copy parallelism to drain the backlog:

kafka-configs.sh --bootstrap-server localhost:9092 \
  --entity-type brokers \
  --entity-name 0 \
  --alter \
  --add-config remote.log.manager.copier.thread.pool.size=16

remote.log.manager.copier.thread.pool.size is a cluster-wide dynamic config; apply it with --entity-type brokers --entity-default to set it for all brokers, or per broker as above. Verify it took effect with a --describe on the same entity.

Consumers Cannot Fetch Historical Data

Symptom: Consumers error or time out fetching offsets older than the local window.

Diagnosis: Confirm objects exist in the bucket for the partition and that __remote_log_metadata is healthy (all partitions have a leader and full ISR). Check broker logs for RSM read errors and RemoteFetchErrorsPerSec.

Resolution: If objects are missing, audit the bucket’s lifecycle rules — an aggressive expiration policy can delete segments Kafka still considers retained. Align the bucket lifecycle with retention.ms plus a safety margin, and let Kafka, not the bucket policy, drive deletion. If __remote_log_metadata is under-replicated, restore replication before anything else.

Metadata Topic Leader Imbalance

Symptom: __remote_log_metadata has most partitions led by one broker, concentrating RLMM client load.

Diagnosis:

kafka-topics.sh --bootstrap-server localhost:9092 \
  --describe --topic __remote_log_metadata

Resolution: Run a preferred-leader election (kafka-leader-election.sh --election-type preferred) or rebalance with kafka-reassign-partitions.sh. Provisioning the topic with enough partitions up front (rlmm.config.remote.log.metadata.topic.num.partitions) spreads metadata load across brokers from the start.

Conclusion

Tiered storage lets Kafka hold long retention at object-store cost while keeping the hot working set on fast local disk. The architecture — the broker-side RemoteLogManager, a pluggable RemoteStorageManager (which you supply; Kafka ships none), and the default topic-based RemoteLogMetadataManager backed by __remote_log_metadata — preserves ordering and serves historical reads transparently to existing clients.

The operational essentials are few but unforgiving: lower local retention below total retention or nothing tiers; watch RemoteCopyLagBytes so disk never fills; replicate __remote_log_metadata durably; and let Kafka, not bucket lifecycle rules, own deletion. Do that, and a cluster can scale retention to petabytes without scaling broker disk in lockstep.

References:

• KIP-405: Kafka Tiered Storage
• Apache Kafka Tiered Storage Operations
• Apache Kafka Tiered Storage Configs
• KIP-963: Additional metrics in Tiered Storage
• Aiven RemoteStorageManager implementation
• Enabling Tiered Storage with S3 and MinIO: A Practical Guide

Related Pages

• Cluster Architecture & Provisioning