Configuring Multi-AZ Kafka on AWS with EBS and Placement Groups

Running Apache Kafka in production demands a rigorous approach to infrastructure design, particularly when the goal is to survive the loss of an entire Availability Zone (AZ) without data loss or prolonged downtime. This guide is an operational blueprint for a Multi-AZ Kafka cluster on AWS, focused on the interplay between Elastic Block Store (EBS) volumes and EC2 placement groups. It assumes a foundational understanding of Cluster Architecture & Provisioning and builds on Multi-AZ Kafka Deployment Strategies for High Availability. The objective is to give platform and SRE engineers the precise specifications needed to build a Kafka cluster that is resilient, performant, and cost-aware.

Why EBS and Placement Groups Matter

The default approach to high availability on AWS spreads instances across AZs and relies on the durability of EBS. For a stateful system like Kafka, that abstraction is leaky. A Kafka broker is not a stateless web server; its identity is tied to its log directories on disk. When a broker fails and is replaced, the new instance should re-attach the same EBS volumes that hold its log segments, so it rejoins with its state intact instead of re-replicating terabytes of data from its peers.

There is a hard constraint here: an EBS volume lives in exactly one AZ and can only be attached to an instance in that same AZ. So any recovery automation must launch the replacement instance in the same AZ as the failed broker. If an Auto Scaling group or operator launches the replacement in a different AZ, it cannot attach the original volume at all, and the broker comes up empty and must rebuild from its in-sync replicas.

Placement groups add a second layer. Kafka’s replication is sensitive to network latency between brokers, especially for followers catching up. Inter-AZ latency is a physical constraint you cannot remove, but you can control intra-AZ behavior:

  • A cluster placement group packs instances into a single low-latency, high-bandwidth network segment within one AZ. It maximizes intra-AZ throughput but introduces correlated-failure risk and can return InsufficientInstanceCapacity errors during scale-out.
  • A spread placement group guarantees that each instance runs on distinct underlying hardware (a separate rack), reducing the chance that one hardware fault takes out multiple instances. A spread group supports at most seven running instances per AZ.

The strategic choice between them, and their combination, forms the backbone of a performant and resilient deployment. We use cluster placement groups within each AZ to maximize intra-AZ replication throughput, while relying on Kafka’s built-in rack awareness to spread replicas across AZs.

EBS Volume Strategy: Provisioning for Performance and Durability

Kafka’s workload is dominated by sequential writes and reads, with occasional random reads when consumers fall behind and fetch older segments. The gp3 volume type is the default recommendation for most production Kafka clusters on AWS: it provides a baseline of 3,000 IOPS and 125 MiB/s throughput that you can scale independently of volume size. For latency-sensitive or very high-throughput clusters, io2 and io2 Block Express provide provisioned IOPS at higher durability (99.999% durability, versus 99.8–99.9% for gp3) at a premium price.

The operational detail that matters most is the relationship between volume size, throughput, and recovery time. When a broker restarts and re-attaches an existing EBS volume, it must recover its on-disk logs — and if it did not shut down cleanly, it scans the active segments to rebuild indexes and validate the log end offset. The dominant cost is reading data off the volume, and that is gated by EBS throughput. A 1 TB gp3 volume at the default 125 MiB/s would take roughly 2.2 hours to read end to end (1,000,000 MiB ÷ 125 MiB/s ≈ 8,000 s). Provisioning extra throughput shrinks that window dramatically.

For gp3, you can provision up to 16,000 IOPS and 1,000 MiB/s of throughput, subject to a ratio of at most 0.25 MiB/s per provisioned IOPS. (As of a September 2025 update, gp3 can go further — up to 80,000 IOPS and 2,000 MiB/s — on supported instance types; confirm current limits in the AWS docs.) The following Terraform snippet balances cost against recovery speed:

resource "aws_ebs_volume" "kafka_data" {
  availability_zone = var.availability_zone
  size              = 1000 # GiB
  type              = "gp3"
  iops              = 16000
  throughput        = 500 # MiB/s

  tags = {
    Name      = "kafka-data-${var.broker_id}"
    BrokerID  = var.broker_id
    Cluster   = var.cluster_name
    ManagedBy = "terraform"
  }
}

At 500 MiB/s, a full sequential read of the 1 TB volume drops to about 33 minutes (1,000,000 MiB ÷ 500 MiB/s ≈ 2,000 s). In practice recovery is shorter than that — a clean shutdown lets the broker skip the recovery scan entirely, and even after an unclean stop it only scans the unflushed tail of each log. The headroom matters precisely for the catch-up phase after a restart, when a follower must pull missed segments from the leader as fast as the disk and network allow.

A common pitfall is putting Kafka log directories on instance store. Instance store offers very low latency and high throughput at no extra cost, but the data is ephemeral: if the instance stops, terminates, or the underlying host fails, the data is gone. For a Multi-AZ design where durability is the point, log directories belong on EBS; instance store should be reserved, if used at all, for genuinely disposable data. The AWS documentation on EBS volume types is the authoritative reference for the performance and durability characteristics of each type.

Placement Group Architecture: Cluster Within an AZ, Spread for Controllers

The choice between spread and cluster groups is not binary; a careful deployment uses both. Within each AZ, deploy a cluster placement group for the brokers so they share a low-latency, high-bandwidth segment. This matters for in-sync replica (ISR) traffic: a follower that stops fetching for longer than replica.lag.time.max.ms (default 30,000 ms) is dropped from the ISR, which can force leader changes and degrade tail latency for producers using acks=all.

A cluster placement group does concentrate failure risk — if that network segment degrades, every broker in the group can be affected at once. Mitigate it at the cluster level: deploy the controllers (ZooKeeper nodes, or KRaft controller nodes) into a spread placement group so they land on distinct hardware. The controllers hold the cluster’s metadata quorum; spreading them ensures a single rack or host fault cannot take down a majority and stall the quorum.

Tie the topology together with broker.rack. Kafka’s rack-aware partition assignor uses it to place a partition’s replicas in different racks. Set broker.rack to the AZ identifier (e.g., us-east-1a), because the AZ is the failure domain Kafka should reason about; the placement group is an infrastructure-level optimization Kafka does not need to know about. Note that broker.rack is a read-only (static) config — it is set in server.properties and only takes effect on broker startup, so changing it requires a restart.

Recovery from an instance failure is therefore a same-AZ operation: launch a replacement instance in the original AZ (and placement group) and attach the surviving EBS volume to it. Because run-instances block device mappings can only create volumes (from a snapshot or fresh spec) and cannot reference an existing VolumeId, attaching a pre-existing volume is a two-step workflow — launch, then attach:

# 1. Launch the replacement instance into the same AZ and placement group.
INSTANCE_ID=$(aws ec2 run-instances \
    --image-id ami-0abcdef1234567890 \
    --instance-type r6i.2xlarge \
    --placement "GroupName=kafka-cluster-pg-us-east-1a,AvailabilityZone=us-east-1a" \
    --subnet-id subnet-0abc123def456gh78 \
    --tag-specifications "ResourceType=instance,Tags=[{Key=BrokerID,Value=1},{Key=Cluster,Value=prod-kafka}]" \
    --count 1 \
    --query 'Instances[0].InstanceId' --output text)

aws ec2 wait instance-running --instance-ids "$INSTANCE_ID"

# 2. Attach the surviving data volume (same AZ as the instance).
aws ec2 attach-volume \
    --volume-id vol-0abcd1234efgh5678 \
    --instance-id "$INSTANCE_ID" \
    --device /dev/sdf

The data volume is intentionally separate from the instance lifecycle: it survives instance termination and is re-attached to the replacement, which is what preserves broker state across failures.

Rack Awareness and Multi-AZ Replication Topology

Rack awareness is the linchpin that ties EBS and placement groups into a coherent Multi-AZ strategy. With broker.rack set, the partition assignor distributes a partition’s replicas across racks. In this AWS context each AZ is a rack, so a topic with replication factor 3 across three AZs gets exactly one replica per AZ for each partition. If an entire AZ is lost, every partition still has two replicas in the surviving AZs, and producers using acks=all keep writing as long as min.insync.replicas is satisfied (set it to 2 with RF 3 so a single-AZ outage does not stall producers, while still requiring two acknowledgers).

Inject broker.rack from the instance’s own metadata at launch so the value always matches the actual placement, even when an AMI is reused across AZs. The example below uses IMDSv2, which is the default on current AMIs and requires a session token:

#!/bin/bash
TOKEN=$(curl -s -X PUT "http://169.254.169.254/latest/api/token" \
    -H "X-aws-ec2-metadata-token-ttl-seconds: 21600")
AZ=$(curl -s -H "X-aws-ec2-metadata-token: $TOKEN" \
    http://169.254.169.254/latest/meta-data/placement/availability-zone)
echo "broker.rack=$AZ" >> /opt/kafka/config/server.properties

The replication topology also shapes performance. Inter-AZ traffic traverses the AWS backbone and carries higher latency (typically low single-digit milliseconds within a region) than intra-AZ traffic. Kafka’s replication protocol tolerates this, but a follower in the same AZ as the leader catches up faster than one across an AZ boundary. You can influence leader placement with rack-aware reassignment and preferred-leader election (kafka-leader-election.sh), but treat AZ-affinity tuning of leaders as an advanced step that needs measurement, not a default.

Infrastructure-as-Code: Terraform Module for a Broker

A repeatable module must handle both initial provisioning and recovery, where an existing EBS volume is re-attached to a new instance. The key correctness point for the recovery path: attaching a pre-existing volume is done with a separate aws_volume_attachment resource — the aws_instance ebs_block_device block creates volumes with the instance and has no volume_id argument, so it cannot attach one you already have.

variable "availability_zone" {}
variable "broker_id" {}
variable "cluster_name" {}
variable "subnet_id" {}
variable "placement_group_name" {}
variable "ebs_volume_id" {
  default = null # If null, create a new volume; otherwise attach this one.
}

resource "aws_placement_group" "cluster_pg" {
  name     = var.placement_group_name
  strategy = "cluster"
}

resource "aws_ebs_volume" "data" {
  count             = var.ebs_volume_id == null ? 1 : 0
  availability_zone = var.availability_zone
  size              = 1000
  type              = "gp3"
  iops              = 16000
  throughput        = 500
  tags = {
    Name     = "kafka-data-${var.cluster_name}-${var.broker_id}"
    BrokerID = var.broker_id
    Cluster  = var.cluster_name
  }
}

locals {
  volume_id = coalesce(var.ebs_volume_id, try(aws_ebs_volume.data[0].id, null))
}

resource "aws_instance" "broker" {
  ami               = data.aws_ami.kafka.id
  instance_type     = "r6i.2xlarge"
  subnet_id         = var.subnet_id
  availability_zone = var.availability_zone
  placement_group   = aws_placement_group.cluster_pg.name

  user_data = templatefile("${path.module}/init.sh.tpl", {
    broker_id = var.broker_id
    az        = var.availability_zone
  })

  tags = {
    Name     = "kafka-broker-${var.cluster_name}-${var.broker_id}"
    BrokerID = var.broker_id
    Cluster  = var.cluster_name
  }
}

# Attach the data volume out-of-band so it survives instance replacement.
resource "aws_volume_attachment" "data" {
  device_name = "/dev/sdf"
  volume_id   = local.volume_id
  instance_id = aws_instance.broker.id

  # Do NOT destroy the attachment by force-detaching on every change;
  # let Kafka stop cleanly first. Manage detachment in your runbook.
  skip_destroy = true
}

The ebs_volume_id variable is the switch: provide it to re-attach a surviving volume during recovery, or leave it null to create a fresh one. This pattern is what lets self-healing automation hand a failed broker’s state to its replacement.

The init.sh.tpl template handles rack configuration and mounting. On Nitro-based instances the EBS volume attached as /dev/sdf is presented by the NVMe driver under a name like /dev/nvme1n1, and the index is not guaranteed to match the attachment order. Resolve the device by its volume ID rather than hardcoding nvme1n1:

#!/bin/bash
set -euo pipefail

# Configure broker identity and rack.
echo "broker.rack=${az}"     >> /opt/kafka/config/server.properties
echo "broker.id=${broker_id}" >> /opt/kafka/config/server.properties

# Resolve the data volume's device by serial (the NVMe driver exposes the
# EBS volume ID, with the leading "vol-" dash stripped) instead of guessing.
VOL_SERIAL=$(echo "${ebs_volume_id}" | tr -d '-')
DEV=$(lsblk -dno NAME,SERIAL | awk -v s="$VOL_SERIAL" '$2==s {print "/dev/"$1}')

if ! blkid "$DEV"; then
    mkfs -t xfs "$DEV"
fi
mkdir -p /var/lib/kafka/data
mount "$DEV" /var/lib/kafka/data
echo "$DEV /var/lib/kafka/data xfs defaults,noatime 0 2" >> /etc/fstab

systemctl enable --now kafka

Always validate device naming on the exact instance type you deploy; the /dev/sdf → /dev/nvmeXn1 mapping is a frequent source of operational errors.

Operational Validation: Verifying Multi-AZ Resilience

Deploying the infrastructure is only half the job. Run the following procedure before promoting the cluster.

Step 1: Verify Rack Assignment

The reliable way to confirm each broker’s rack depends on your metadata mode. In a ZooKeeper-backed cluster, read the broker znode, which carries a rack field:

zookeeper-shell.sh zk1:2181 get /brokers/ids/1

The JSON output includes "rack":"us-east-1a" (or whatever AZ you assigned). If the field is missing, that broker did not start with broker.rack set — fix it before proceeding.

For a KRaft cluster there is no ZooKeeper. Confirm the controller quorum is healthy with:

kafka-metadata-quorum.sh --bootstrap-server broker1:9092 describe --status

This prints the leader, voters, observers, and follower lag for the metadata quorum — it validates the controllers, not the per-broker rack. To confirm rack placement in either mode, the authoritative check is the effect of rack awareness on replica placement, which Step 2 verifies directly.

Step 2: Validate Partition Distribution

Create a test topic with replication factor equal to the number of AZs and inspect the assignment:

kafka-topics.sh --bootstrap-server broker1:9092 \
    --create --topic resilience-test \
    --partitions 12 --replication-factor 3

kafka-topics.sh --bootstrap-server broker1:9092 \
    --describe --topic resilience-test

Map each partition’s replica broker IDs back to their AZs. With rack awareness honored, every partition’s three replicas land in three distinct AZs. If any partition has two replicas in the same AZ, rack awareness is not in effect — usually a missing broker.rack or a custom assignor.

Step 3: Simulate AZ Failure

Stop all Kafka processes in one AZ at once. Stop the service rather than terminating instances, so you isolate Kafka’s failover behavior from instance replacement:

# On each broker in the target AZ
sudo systemctl stop kafka

Then watch replication health:

kafka-topics.sh --bootstrap-server broker1:9092 \
    --describe --topic resilience-test \
    --under-replicated-partitions

Partitions whose leader was in the failed AZ briefly appear here while Kafka elects new leaders from the surviving in-sync replicas; that election normally completes within seconds. The remaining partitions stay under-replicated (RF 3 with one AZ down means two replicas instead of three) until the AZ returns — that is expected, not a fault. Keep unclean.leader.election.enable=false (the production default) so Kafka never promotes an out-of-sync replica, which would discard committed data; it is a boolean, not a timeout.

While the AZ is down, drive traffic to confirm the cluster still serves writes:

kafka-producer-perf-test.sh \
    --topic resilience-test \
    --num-records 100000 \
    --record-size 1000 \
    --throughput -1 \
    --producer-props acks=all bootstrap.servers=broker2:9092,broker3:9092

With acks=all and min.insync.replicas=2, this completes despite one AZ being offline, at modestly higher latency. Point bootstrap.servers at brokers in the surviving AZs.

Step 4: Restore the Failed AZ

Restart Kafka in the recovered AZ and watch the followers catch up:

# On each broker in the restored AZ
sudo systemctl start kafka

# Monitor under-replicated partitions trending to zero
watch -n 5 'kafka-topics.sh --bootstrap-server broker1:9092 \
    --describe --under-replicated-partitions | wc -l'

The count should fall steadily as the restarted brokers replay missed segments and rejoin the ISR. Recovery speed tracks EBS throughput and intra-AZ network performance. If it stalls, check EBS throttling via CloudWatch (VolumeThroughputPercentage, BurstBalance) or iostat on the broker.

Monitoring and Alerting

The signals that matter for Multi-AZ Kafka span both JMX and infrastructure metrics:

  1. Under-replicated partitions — kafka.server:type=ReplicaManager,name=UnderReplicatedPartitions. A sustained non-zero value during normal operation means replicas are lagging and durability margin is shrinking.
  2. Active controller count — kafka.controller:type=KafkaController,name=ActiveControllerCount. Summed across the cluster this must equal exactly 1; anything else signals a controller problem.
  3. Offline partitions — kafka.controller:type=KafkaController,name=OfflinePartitionsCount. Any non-zero value means partitions with no leader (unavailable for reads and writes).
  4. ISR shrink/expand rate — kafka.server:type=ReplicaManager,name=IsrShrinksPerSec and IsrExpandsPerSec. Frequent churn points to network instability or disk throughput limits.
  5. Unclean leader elections — kafka.controller:type=ControllerStats,name=UncleanLeaderElectionsPerSec. Should stay at zero when unclean election is disabled.
  6. EBS throughput and queue depth — CloudWatch VolumeReadBytes/VolumeWriteBytes, VolumeQueueLength, and VolumeThroughputPercentage to detect saturation.

A CloudWatch composite alarm can correlate these. For example, fire when under-replicated partitions stay non-zero for more than five minutes and VolumeThroughputPercentage is pinned at 100% — that combination indicates replication is bottlenecked on disk I/O and points you at a throughput increase or a partition-count reduction rather than a network issue.

The Apache Kafka monitoring documentation lists the full set of broker metrics and their interpretations, and the AWS Well-Architected reliability pillar covers the broader resilience design these choices fit into.

Cost Optimization and Performance Tuning

This architecture is resilient but can be expensive if it is not tuned to the workload.

Storage tiering with JBOD. Not every topic needs the same disk. Kafka’s JBOD support lets a broker use multiple log directories on different volumes, configured at the broker level:

log.dirs=/var/lib/kafka/data/gp3,/var/lib/kafka/data/st1

By default Kafka places each new partition in the log directory with the fewest partitions; there is no topic-level config that binds a topic to a specific disk. To deliberately steer specific replicas onto a specific volume, use kafka-reassign-partitions.sh with a reassignment JSON that names the target log_dirs per replica (this drives the AlterReplicaLogDirs API). That gives you intentional placement — e.g., hot topics on gp3, cheap/rebuildable topics on throughput-optimized st1 — at the cost of managing the mapping yourself. Note that on KRaft, full JBOD disk-failure handling landed via KIP-858 (generally available in Kafka 3.7+); confirm your version supports JBOD in KRaft before relying on it.

Placement group strategy. Cluster placement groups are free but restrict instance types and can hit capacity errors on scale-out. If you do not need sub-millisecond intra-AZ latency, a spread group (max seven instances per AZ) is often enough: with RF 3 and acks=all, inter-AZ latency usually dominates end-to-end producer latency, so the intra-AZ network advantage of a cluster group is marginal.

Instance right-sizing. The r6i family suits Kafka because its high memory-to-vCPU ratio feeds the OS page cache that serves recent reads. If your workload is network- or CPU-bound rather than memory-bound, c6i (or network-optimized *n variants) can offer better price-performance. Use kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec and BytesOutPerSec to identify the real bottleneck before resizing.

These choices are part of the broader discipline of Cluster Architecture & Provisioning. By combining EBS volume management, placement-group topology, and rack awareness into one infrastructure-as-code module, you build a cluster that survives an AZ failure with minimal operator intervention — and the validation steps above belong in CI as integration tests, so every infrastructure change is checked against the resilience guarantees your workloads depend on.

Back to Multi AZ Kafka Deployment Strategies For High Availability