Enabling Tiered Storage with S3 and MinIO: A Practical Guide

Tiered storage in Apache Kafka offloads older (closed) log segments to object storage while keeping client access transparent: consumers can still read offloaded data, served on demand from the remote tier. This lets you keep only a short window of data on broker disks and retain the long tail in cheaper object storage. This guide walks through enabling tiered storage against an S3-compatible backend, using MinIO for on-premises and hybrid deployments and AWS S3 for cloud.

For conceptual background on the RemoteStorageManager interface and how the remote tier fits together, see the Tiered Storage in Kafka: Architecture and Configuration guide. This article assumes that background and focuses on implementation.

Prerequisites

Apache Kafka ships the tiered storage framework (KIP-405) but no built-in RemoteStorageManager (RSM) — you supply a plugin. For S3-compatible backends, the open-source tiered-storage-for-apache-kafka plugin from Aiven supports AWS S3 and any S3-compatible API (MinIO, Ceph RADOS Gateway, others), as well as GCS and Azure Blob.

Requirements:

• Kafka 3.9.0 or later. Tiered storage was early access (explicitly not for production) from 3.6 through 3.8 and reached production-ready GA in 3.9. Use 3.9+ for any real deployment. See the Apache Kafka Tiered Storage operations guide.
• A running MinIO cluster or AWS S3 bucket with credentials.
• Network connectivity from every broker to the object storage endpoint.
• A JDK supported by your Kafka release (Kafka 3.9 supports Java 11 for the broker and Java 17 for clients/Streams; later releases raise these floors — check your version’s docs).

Limitation to confirm first: the remote tier does not support compacted topics, and you must keep cleanup.policy=delete on any topic you tier. You also cannot disable broker-level tiered storage until it is disabled on every topic. These constraints are documented in the operations guide above.

Install the plugin

The Aiven plugin is published on GitHub Releases, not Maven Central, and is distributed as two tarballs: a core library and one storage-backend library per backend (here, s3). Each tarball bundles its own dependencies (including the AWS SDK), so you do not download individual SDK JARs by hand.

# Pick a version from the releases page; export it once
VERSION=<latest-release>   # e.g. 2024-11-26-1716; see the GitHub Releases page

mkdir -p /opt/kafka/plugins/tiered-storage/{core,s3}

# Core RemoteStorageManager
curl -fsSL -o /tmp/core.tgz \
  https://github.com/Aiven-Open/tiered-storage-for-apache-kafka/releases/download/v${VERSION}/core-${VERSION}.tgz
tar xzf /tmp/core.tgz -C /opt/kafka/plugins/tiered-storage/core --strip-components=1

# S3 storage backend
curl -fsSL -o /tmp/s3.tgz \
  https://github.com/Aiven-Open/tiered-storage-for-apache-kafka/releases/download/v${VERSION}/s3-${VERSION}.tgz
tar xzf /tmp/s3.tgz -C /opt/kafka/plugins/tiered-storage/s3 --strip-components=1

The exact release tag, artifact names, and bundled dependencies vary between versions. The plugin README is the authoritative source for artifact names and the installation layout for your chosen release.

Make the plugin directories readable by the Kafka process user, and install them identically on every broker.

Configuring MinIO for Kafka Tiered Storage

MinIO provides an S3-compatible object store suitable for on-premises and edge deployments. For production, run MinIO in distributed mode across multiple nodes (a four-node deployment is the practical minimum for erasure-coded redundancy).

Create a dedicated bucket, policy, and service account with the MinIO client (mc):

# Configure the MinIO client alias
mc alias set minio http://minio.example.com:9000 adminuser adminpassword

# Create a dedicated bucket
mc mb minio/kafka-tiered-storage

# Define a least-privilege policy
cat > kafka-tiered-policy.json <<'EOF'
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:PutObject",
        "s3:DeleteObject",
        "s3:ListBucket",
        "s3:AbortMultipartUpload"
      ],
      "Resource": [
        "arn:aws:s3:::kafka-tiered-storage",
        "arn:aws:s3:::kafka-tiered-storage/*"
      ]
    }
  ]
}
EOF

# Create a service account and attach the policy
mc admin user add minio kafka-tiered-user kafka-secret-key-123
mc admin policy create minio kafka-tiered-policy kafka-tiered-policy.json
mc admin policy attach minio kafka-tiered-policy --user kafka-tiered-user

s3:AbortMultipartUpload matters: the plugin uploads large segments via S3 multipart, and an interrupted upload must be abortable to avoid orphaned parts. For AWS S3, create an IAM policy with the same actions scoped to your bucket ARN and attach it to a dedicated IAM user or, preferably, an IAM role assumed via an instance profile.

Broker Configuration

Add the following to each broker’s server.properties. The RSM settings must be identical across all brokers.

The plugin’s own settings are passed through Kafka using a prefix. You set remote.log.storage.manager.impl.prefix (conventionally rsm.config.), and every plugin property is then specified with that prefix.

# --- Enable the tiered-storage framework ---
remote.log.storage.system.enable=true

# Listener the default (topic-based) RemoteLogMetadataManager uses to talk to brokers.
# Must name one of your configured listeners.
remote.log.metadata.manager.listener.name=PLAINTEXT

# --- Plug in the Aiven RemoteStorageManager ---
remote.log.storage.manager.class.name=io.aiven.kafka.tieredstorage.RemoteStorageManager
remote.log.storage.manager.class.path=/opt/kafka/plugins/tiered-storage/core/*:/opt/kafka/plugins/tiered-storage/s3/*
remote.log.storage.manager.impl.prefix=rsm.config.

# --- Plugin (rsm.config.*) settings: select the S3 storage backend ---
rsm.config.storage.backend.class=io.aiven.kafka.tieredstorage.storage.s3.S3Storage
rsm.config.storage.s3.bucket.name=kafka-tiered-storage
rsm.config.storage.s3.region=us-east-1

# Endpoint + path-style access are required for MinIO / non-AWS S3
rsm.config.storage.s3.endpoint.url=http://minio.example.com:9000
rsm.config.storage.s3.path.style.access.enabled=true

# Static credentials (omit on AWS and use the default credential chain instead)
rsm.config.storage.aws.access.key.id=kafka-tiered-user
rsm.config.storage.aws.secret.access.key=kafka-secret-key-123

# --- Optional upload tuning ---
# Total S3 API call timeout incl. retries (ms); per-attempt is s3.api.call.attempt.timeout
rsm.config.storage.s3.api.call.timeout=300000
# Multipart part size in bytes (default 26214400 = 25 MiB; valid 5 MiB..2 GiB)
rsm.config.storage.s3.multipart.upload.part.size=26214400

For AWS S3, omit endpoint.url, omit path.style.access.enabled, and omit the static aws.* keys so the plugin uses the AWS SDK default credential provider chain (instance profile / IRSA / environment). Keep secrets out of server.properties where possible.

After updating server.properties, perform a rolling restart, one broker at a time, waiting for each to rejoin the ISR before proceeding. Enabling the framework and installing the plugin is an online, restart-based change; topics are not tiered until you opt them in (next section). Confirm each broker starts cleanly and that the __remote_log_metadata internal topic is created.

Configuring Topic-Level Retention

Tiering is opt-in per topic and is governed by these topic configs:

Create a topic with tiering enabled:

kafka-topics.sh --bootstrap-server localhost:9092 \
  --create \
  --topic events-tiered \
  --partitions 12 \
  --replication-factor 3 \
  --config remote.storage.enable=true \
  --config local.retention.ms=3600000 \
  --config retention.ms=-1

This keeps roughly the last hour on local disk and retains everything in object storage indefinitely. Enable it on an existing topic dynamically:

kafka-configs.sh --bootstrap-server localhost:9092 \
  --alter \
  --entity-type topics --entity-name events-tiered \
  --add-config 'remote.storage.enable=true,local.retention.ms=3600000'

Size local.retention.ms to your read patterns: consumers that frequently re-read recent data should find it on local disk (fast path), while purely append-and-forget workloads can use a short window. Watch consumer lag and disk utilization to converge on the right value.

Operational Validation and Monitoring

Confirm that segments are being offloaded and that consumers can transparently read them back.

Inspect per-partition remote size with kafka-log-dirs.sh; the output JSON includes a remoteLogSize field per partition (the bytes stored in the remote tier):

kafka-log-dirs.sh --bootstrap-server localhost:9092 \
  --describe \
  --topic-list events-tiered

Metrics

Kafka exposes tiered-storage metrics over JMX. The per-topic copy/fetch/lag metrics live on BrokerTopicMetrics; the reader thread-pool metrics live on a separate MBean. The exact object names (verified against the Kafka monitoring docs):

Scrape these with the Prometheus JMX exporter and alert on sustained RemoteCopyErrorsPerSec/RemoteFetchErrorsPerSec, climbing RemoteCopyLagBytes, and a growing RemoteLogReaderTaskQueueSize.

Troubleshooting Common Issues

Access Denied / signature errors. If brokers log an S3Exception with Access Denied, confirm the IAM or MinIO policy grants GetObject, PutObject, DeleteObject, ListBucket, and AbortMultipartUpload on the exact bucket ARN. For MinIO and other non-AWS endpoints you must set rsm.config.storage.s3.path.style.access.enabled=true; without it the SDK builds virtual-host-style URLs that fail or produce signature mismatches.

Upload timeouts on large segments. Slow links or very large segments can exceed the S3 API timeout. Raise rsm.config.storage.s3.api.call.timeout (and s3.api.call.attempt.timeout per attempt), and/or reduce the topic’s segment.bytes so segments close and upload sooner.

Plugin fails to load (ClassNotFoundException). Ensure both the core and S3 directories exist on every broker and that remote.log.storage.manager.class.path is an absolute, :-separated list of directories each ending in /*. The class name must be io.aiven.kafka.tieredstorage.RemoteStorageManager and rsm.config.storage.backend.class must be io.aiven.kafka.tieredstorage.storage.s3.S3Storage.

First-read latency on offloaded data. The first fetch of a remote segment pays the round-trip to object storage. Increase remote.log.reader.threads for more read parallelism, ensure the endpoint has enough read throughput, and for latency-sensitive consumers keep more data local by raising local.retention.ms.

Capacity Planning with Tiered Storage

Tiering lets you size broker disks for the local retention window plus headroom for upload lag, rather than the full retention period. That makes smaller, faster NVMe drives viable while object storage holds the long tail.

Estimate local disk per broker-replica set with:

LocalDiskGB = (IngressMBps × LocalRetentionSeconds / 1024) × ReplicationFactor × 1.3

The 1.3 multiplier covers segment index/time-index files and a buffer for in-flight upload lag. For 100 MB/s ingress, one hour of local retention, and replication factor 3:

LocalDiskGB = (100 × 3600 / 1024) × 3 × 1.3 ≈ 1371 GB

That is far less than the tens of terabytes the same workload would need to keep multiple weeks on local disk. Budget object storage separately, against total retained volume and API request counts — frequent small reads of cold data can make request charges, not capacity, the dominant cost.

For broker sizing and hardware selection, see the Cluster Architecture & Provisioning pillar.