Multi-AZ Kafka Deployment Strategies for High Availability

A single availability zone (AZ) failure can halt data ingestion, stall consumers, and cause data loss if your Kafka cluster topology doesn’t explicitly account for it. This guide covers the architecture, configuration, and operational procedures required to run Kafka across multiple AZs with real fault tolerance.

Failure Domains and Kafka’s Resilience Model

Kafka achieves durability through replication. Each partition has a leader and follower replicas distributed across brokers. When a broker fails, the controller—ZooKeeper-based or KRaft—elects a new leader from the in-sync replica (ISR) set. This works transparently within a single AZ, but cross-AZ deployments change the failure characteristics.

An AZ is an isolated failure domain with independent power, cooling, and networking. A zone-level failure can simultaneously remove multiple brokers, potentially including the controller and several replicas of the same partition. The core design goal: no single zone failure compromises data durability or availability.

Three-Zone Replication Model

With a replication factor (RF) of 3 across three zones, place one replica per zone. If zone A fails, partitions whose leaders lived in zone A undergo leader election; the controller selects a new leader from the ISR in zone B or C. Two replicas remain, so the cluster continues serving reads and writes without data loss.

This requires Kafka’s rack awareness feature, covered in detail below.

Metadata Management Implications

ZooKeeper-based clusters require a separate ensemble that must also span zones, adding operational overhead. KRaft mode eliminates this external dependency but imposes its own quorum placement requirements. See KRaft vs ZooKeeper: Choosing the Right Metadata Management for a detailed comparison.

Failure Scenarios

Network Architecture

Cross-AZ communication introduces latency, bandwidth constraints, and cost that don’t exist in single-AZ clusters.

Latency and Replication Impact

Typical inter-AZ round-trip latency is 1–5 ms within a region. With producer acks=all and RF=3 across three zones, the produce request is acknowledged only after min.insync.replicas replicas have appended the record. With min.insync.replicas=2, the producer waits for the leader’s local append plus one follower fetch round trip. If the nearest in-sync follower is in another zone with 2 ms RTT, this adds roughly that RTT to produce latency versus a single-AZ deployment.

For latency-sensitive workloads, consider producer acks=1 or a lower min.insync.replicas, accepting the corresponding durability trade-off.

Topic Configuration for Strong Durability

kafka-topics.sh --bootstrap-server broker1:9092 \
  --create \
  --topic critical-events \
  --partitions 12 \
  --replication-factor 3 \
  --config min.insync.replicas=2 \
  --config retention.ms=604800000

With min.insync.replicas=2, a produce request using acks=all is acknowledged only when at least two replicas (leader plus one follower) have persisted the record. In a three-zone deployment with one replica per zone, data survives any single zone failure.

Note: acks is a producer-side configuration, not a topic-level config. Set acks=all in the producer configuration; it cannot be applied via --config on topic creation. min.insync.replicas is the broker/topic-side counterpart that makes acks=all durable.

Bandwidth Capacity Planning

Replication is the primary consumer of inter-AZ bandwidth. With RF=3 and one replica per zone, each record the leader receives is shipped to two followers, and both followers are in remote zones—so essentially all replication traffic crosses zone boundaries. For a cluster ingesting 100 MB/s, that is ~200 MB/s of sustained inter-AZ replication traffic before accounting for consumer fetches.

Replication BW ≈ Ingress Throughput × (RF - 1) × Cross-Zone Fraction

With one replica per zone and a leader uniformly distributed, the cross-zone fraction of follower fetches approaches 1.0, since both followers of any partition are remote relative to its leader.

Replication Throttling

Throttle replication traffic to prevent saturation during reassignments or broker recovery. These are dynamic broker configs applied with kafka-configs.sh:

kafka-configs.sh --bootstrap-server broker1:9092 \
  --entity-type brokers \
  --entity-name 1 \
  --alter \
  --add-config 'leader.replication.throttled.rate=104857600,follower.replication.throttled.rate=104857600'

This sets a 100 MB/s throttle on both leader and follower replication traffic for broker 1. The rate applies only to replicas listed in the per-topic leader.replication.throttled.replicas / follower.replication.throttled.replicas sets, which kafka-reassign-partitions.sh populates automatically when you pass --throttle during a reassignment.

TLS for Inter-Broker Communication

Inter-AZ traffic traverses shared infrastructure. Enable TLS:

# server.properties
listeners=INTERNAL://0.0.0.0:9092,EXTERNAL://0.0.0.0:9093
inter.broker.listener.name=INTERNAL
listener.security.protocol.map=INTERNAL:SSL,EXTERNAL:PLAINTEXT
ssl.keystore.location=/var/private/ssl/kafka.keystore.jks
ssl.keystore.password=changeit
ssl.key.password=changeit
ssl.truststore.location=/var/private/ssl/kafka.truststore.jks
ssl.truststore.password=changeit
ssl.client.auth=required

Modern instance types with AES-NI acceleration minimize the encryption overhead. If you terminate TLS at a service mesh or proxy, ensure that proxy layer is itself distributed across zones.

Rack Awareness and Partition Placement

Kafka’s rack awareness is the primary mechanism for controlling replica placement across failure domains.

Configuring Broker Rack IDs

Derive the rack ID from instance metadata to avoid manual errors. On AWS, use IMDSv2 (a token-authenticated request) rather than the legacy unauthenticated endpoint:

#!/bin/bash
TOKEN=$(curl -s -X PUT "http://169.254.169.254/latest/api/token" \
  -H "X-aws-ec2-metadata-token-ttl-seconds: 60")
AVAILABILITY_ZONE=$(curl -s -H "X-aws-ec2-metadata-token: ${TOKEN}" \
  http://169.254.169.254/latest/meta-data/placement/availability-zone)
echo "broker.rack=${AVAILABILITY_ZONE}" >> /opt/kafka/config/server.properties

broker.rack is a static broker property read at startup; set it in server.properties:

broker.rack=us-east-1a

Use the cloud provider’s zone identifier (e.g., us-east-1a) for clarity.

Rack-Aware Replica Assignment

When a topic is created or partitions are reassigned and every broker has a broker.rack, Kafka spreads each partition’s replicas across as many distinct racks as possible before reusing a rack. For RF=3 across three racks, this yields exactly one replica per zone per partition. If RF exceeds the number of racks, Kafka minimizes the number of replicas placed in any single rack.

Verify current placement with kafka-topics.sh --describe and map the Replicas/Isr broker IDs to their racks:

kafka-topics.sh --bootstrap-server broker1:9092 \
  --describe --topic critical-events

The output lists Replicas: and Isr: broker IDs per partition. Confirm that no two replicas of the same partition share a rack.

Rebalancing After Cluster Changes

Cluster expansions or broker replacements can skew replica distribution. Use kafka-reassign-partitions.sh to generate a rack-aware plan. First describe the topics to move in JSON:

{
  "topics": [
    {"topic": "critical-events"}
  ],
  "version": 1
}

Then generate a proposed assignment across the target broker set:

kafka-reassign-partitions.sh --bootstrap-server broker1:9092 \
  --topics-to-move-json-file topics.json \
  --broker-list "1,2,3,4,5,6" \
  --generate

--generate prints a proposed reassignment (and the current one). Review it, save the proposed JSON, then apply it with --execute --reassignment-json-file plan.json. Pass --throttle <bytes/sec> on --execute to cap the migration rate and protect inter-AZ links, and run --verify afterward to confirm completion and remove the throttle.

For continuous automated rebalancing, consider Cruise Control or Confluent Self-Balancing Clusters.

Controller Placement and Metadata Resilience

Controller Election Across Zones

ZooKeeper mode: Controller election is first-come-first-served via ephemeral node registration in ZooKeeper. There is no built-in zone preference. If the controller’s zone fails, the cluster must detect the failure and elect a new controller before partition leader elections proceed.

KRaft mode: The controller is a quorum of nodes running the Raft protocol. A majority must be available. With three controllers across three zones, one zone failure leaves two controllers—a majority—so the cluster remains operational. With three controllers placed across only two zones (two in one zone, one in the other), losing the zone that holds two controllers breaks quorum entirely. Always place controller voters so that no single zone holds a majority.

Optimal KRaft placement for three zones—one controller per zone:

controller.quorum.voters=1@controller-az1:9093,2@controller-az2:9093,3@controller-az3:9093
controller.listener.names=CONTROLLER

Controller Failover Timing

After election, the new controller loads cluster metadata and initializes state. For very large clusters, this can take tens of seconds.

Key Controller Metrics

ActiveControllerCount is reported by every broker/controller node and should sum to exactly 1 across the cluster.

Deployment Patterns

Pattern 1: Stretched Cluster Across Three Zones

Brokers distributed evenly across three AZs, RF=3, min.insync.replicas=2, one replica per zone per partition.

Advantages:

• Tolerates complete loss of any single zone without data loss or unavailability.
• Single cluster to manage.
• Transparent client failover.

Disadvantages:

• Cross-AZ replication latency affects acks=all produce latency.
• Inter-AZ data transfer costs scale with throughput.
• Network partitions between zones require careful timeout tuning.

Configuration:

# server.properties (per broker)
broker.rack=us-east-1a
default.replication.factor=3
min.insync.replicas=2

For AWS-specific EBS and placement group details, see Configuring Multi-AZ Kafka on AWS with EBS and Placement Groups.

Pattern 2: Active-Passive Across Two Regions

A stretched cluster in the primary region replicates asynchronously to a secondary region via MirrorMaker 2.

Advantages:

• Tolerates complete regional failure.
• Geographic locality for distributed consumers.

Disadvantages:

• Two clusters plus MirrorMaker to operate.
• Replication lag means seconds to minutes of potential data loss on failover.
• Roughly double the infrastructure cost.

MirrorMaker 2 configuration:

# mm2.properties
clusters = primary, secondary
primary.bootstrap.servers = broker1-primary:9092,broker2-primary:9092
secondary.bootstrap.servers = broker1-secondary:9092,broker2-secondary:9092

primary->secondary.enabled = true
primary->secondary.topics = critical-events, transactions, audit-logs

primary->secondary.replication.factor = 3
primary->secondary.offset-syncs.topic.replication.factor = 3

primary->secondary.emit.heartbeats.enabled = true
primary->secondary.emit.checkpoints.enabled = true

The emit.checkpoints and consumer-offset translation features let consumers resume near their last committed position after a failover; offsets are not byte-for-byte identical across clusters.

Pattern 3: Single-Zone Clusters with Application-Level Multi-Cluster Writes

Separate Kafka clusters per AZ; applications write to multiple clusters.

Advantages:

• Zero cross-AZ replication latency or cost.
• Complete zone isolation.

Disadvantages:

• Application complexity: multi-cluster writes with retry and deduplication.
• No cross-cluster ordering guarantees.
• Consumers must aggregate from multiple clusters and handle duplicates.

Rarely recommended for general-purpose event streaming. Suitable only where cross-AZ latency is unacceptable and eventual consistency is tolerable.

Operational Procedures for Zone Failures

Detecting Zone-Level Failures

Zone failures manifest as correlated symptoms:

1. Multiple simultaneous broker failures in controller logs.
2. Under-replicated partitions spike (kafka.server:type=ReplicaManager,name=UnderReplicatedPartitions).
3. Offline partitions (kafka.controller:type=KafkaController,name=OfflinePartitionsCount > 0).
4. Producer errors: NOT_ENOUGH_REPLICAS when min.insync.replicas can no longer be satisfied.
5. Consumer lag increases as partition leaders become unavailable.

Prometheus alert rule:

groups:
  - name: kafka-zone-failure
    rules:
      - alert: KafkaUnderReplicatedSpike
        expr: |
          sum by (cluster) (kafka_server_replicamanager_underreplicatedpartitions) > 10
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "Under-replicated partitions spike in {{ $labels.cluster }}"
          description: "Possible zone failure. {{ $value }} partitions are under-replicated."

Immediate Response Actions

1. Do not restart brokers in the failed zone. Restarting during active infrastructure issues can prolong recovery or trigger avoidable leader churn. Wait for the cloud provider to resolve the zone issue.
2. Verify controller health. Confirm ActiveControllerCount sums to 1 and the active controller is in a healthy zone. If it was in the failed zone, confirm a new controller was elected.
3. Check for offline and under-replicated partitions:

kafka-topics.sh --bootstrap-server broker1:9092 \
  --describe --under-replicated-partitions

4. Assess data-loss risk. If any partition has lost all ISRs in the surviving zones, decide whether to wait for recovery or accept data loss via unclean leader election.
5. Communicate with stakeholders. Notify application teams of the incident, expected timeline, and recommended actions.

Managing Unclean Leader Election

Unclean leader election promotes an out-of-sync replica to leader when no ISR member is available. This causes data loss: records acknowledged to producers but not yet replicated to the promoted replica are lost.

By default, unclean.leader.election.enable is false. Partitions with no surviving ISR member stay offline until a failed broker recovers.

Enable unclean leader election for a specific topic during a zone failure:

kafka-configs.sh --bootstrap-server broker1:9092 \
  --entity-type topics \
  --entity-name critical-events \
  --alter \
  --add-config unclean.leader.election.enable=true

This is a deliberate, data-loss-accepting decision. After the zone recovers and replicas are back in sync, revert it:

kafka-configs.sh --bootstrap-server broker1:9092 \
  --entity-type topics \
  --entity-name critical-events \
  --alter \
  --delete-config unclean.leader.election.enable

Zone Recovery and Reintegration

1. Verify broker health. Check disk, memory, and network. Look for filesystem damage from an ungraceful shutdown.
2. Monitor under-replicated partitions. The count should steadily decrease. If it plateaus, investigate replication throttling or network bottlenecks.
3. Rebalance leaders after all replicas are back in the ISR:

kafka-leader-election.sh --bootstrap-server broker1:9092 \
  --election-type preferred \
  --all-topic-partitions

4. Validate end-to-end health. Run producer and consumer test clients from all zones. Confirm throughput and lag have recovered.

Chaos Engineering

Test multi-AZ resilience regularly:

• Zone isolation: Use network policies to simulate a complete zone partition. Verify cluster availability and continued produce/consume.
• Zone blackout: Terminate all broker instances in a zone. Measure time to full recovery.
• Controller zone failure: Target the zone with the active controller. Measure failover time and producer latency impact.
• Asymmetric network partition: Zone A reaches B but not C; B and C reach each other. Tests partial-connectivity behavior.

Tools such as Gremlin or custom fault-injection scripts work well; integrate the tests into a continuous verification pipeline.

Capacity Planning and Cost Optimization

Sizing for Zone Failure Headroom

The cluster must handle peak load with one zone unavailable. If normal utilization is 70% across three zones, losing one zone pushes the remaining two to 105%—a cascading failure.

Safe approach: Size each zone to handle at least 50% of peak load. That caps steady-state utilization at roughly 67% per zone (two zones absorb 100% of peak). Many teams target ~60% per-zone utilization, accepting some degradation during a zone failure.

This directly affects Broker Sizing and Hardware Selection for Kafka Clusters.

Inter-Zone Data Transfer Cost Modeling

Cloud providers typically charge $0.01–$0.02/GB for inter-AZ transfer (often billed on both send and receive). For a cluster ingesting 100 MB/s with RF=3 and one replica per zone, follower replication moves ~200 MB/s across zones:

Monthly cross-zone bytes ≈ 0.2 GB/s × 2,628,000 s/month ≈ 525,600 GB
Monthly cost ≈ 525,600 GB × $0.01/GB ≈ $5,256/month (replication only)

Consumer fetches that cross zones add to this. Cost-reduction strategies:

• Compression: Set compression.type=zstd on producers. Log/event data commonly compresses 3–5×, reducing both replication bytes and storage.
• Consumer colocation (KIP-392): Set replica.selector.class=org.apache.kafka.common.replica.RackAwareReplicaSelector on the brokers and client.rack on consumers so they fetch from a same-zone in-sync follower instead of the (possibly remote) leader. Requires Kafka 2.4+.
• Selective replication factor: Not every topic needs RF=3. Lower-criticality topics can use RF=2, accepting reduced zone-failure tolerance.

Storage Tiering

RF=3 means roughly 3× storage consumption. Tiered storage (KIP-405, generally available in Kafka 3.9+) offloads older log segments to object storage such as Amazon S3, which is cheaper and inherently multi-AZ durable. Ensure the remote backend is itself zone-resilient.

Monitoring and Observability

Key Metrics

Partition health (broker, ReplicaManager):

• kafka.server:type=ReplicaManager,name=UnderReplicatedPartitions — should be 0.
• kafka.server:type=ReplicaManager,name=UnderMinIsrPartitionCount — partitions below min.insync.replicas; should be 0. These are the partitions that reject acks=all writes.

Offline partitions (controller):

• kafka.controller:type=KafkaController,name=OfflinePartitionsCount — partitions with no leader; should be 0.

Replication health:

• kafka.server:type=ReplicaFetcherManager,name=MaxLag,clientId=Replica — maximum follower lag in records; sustained high values mean followers are falling behind.
• kafka.network:type=RequestMetrics,name=TotalTimeMs,request=FetchFollower — follower fetch latency; spikes indicate inter-AZ network issues.

Controller health:

• kafka.controller:type=ControllerStats,name=UncleanLeaderElectionsPerSec — must be 0.
• kafka.controller:type=KafkaController,name=ActiveControllerCount — must sum to 1.

Network:

• Inter-AZ throughput and latency via cloud provider metrics (e.g., AWS CloudWatch NetworkIn/NetworkOut).
• TCP retransmission rates.

Dashboard Structure

1. Cluster-level health: Active controller, online brokers by zone, under-replicated partitions, offline partitions.
2. Throughput by zone: Produce and consume rates per zone.
3. Latency: Produce and follower-fetch request latency, broken out by zone where possible.
4. Replication health: Max lag by broker, under-replicated trend, throttle activity.
5. Resource utilization: CPU, memory, disk, network per broker, grouped by zone.

Example PromQL queries:

# Under-replicated partitions by topic
sum by (topic) (kafka_server_replicamanager_underreplicatedpartitions) > 0

# Leader count per rack (uneven values indicate leader skew across zones)
sum by (rack) (kafka_server_replicamanager_leadercount)

These assume a JMX exporter that lowercases MBean names into kafka_server_replicamanager_* and attaches a rack label.

Alerting Strategy

Each alert should link to a runbook with step-by-step response procedures.

Automation and Infrastructure as Code

Terraform for Multi-AZ Broker Provisioning

variable "availability_zones" {
  type    = list(string)
  default = ["us-east-1a", "us-east-1b", "us-east-1c"]
}

variable "brokers_per_zone" {
  type    = number
  default = 3
}

resource "aws_instance" "kafka_broker" {
  count = var.brokers_per_zone * length(var.availability_zones)

  ami               = data.aws_ami.kafka.id
  instance_type     = "i3en.3xlarge"
  availability_zone = var.availability_zones[floor(count.index / var.brokers_per_zone)]
  subnet_id         = aws_subnet.kafka[var.availability_zones[floor(count.index / var.brokers_per_zone)]].id

  vpc_security_group_ids = [aws_security_group.kafka_broker.id]

  root_block_device {
    volume_type = "gp3"
    volume_size = 100
  }

  user_data = templatefile("${path.module}/templates/broker_init.sh.tpl", {
    rack_id           = var.availability_zones[floor(count.index / var.brokers_per_zone)]
    broker_id         = count.index + 1
    zookeeper_connect = var.zookeeper_connect_string
  })

  tags = {
    Name = "kafka-broker-${count.index + 1}"
    Rack = var.availability_zones[floor(count.index / var.brokers_per_zone)]
  }
}

Ansible for Configuration Management

- name: Configure Kafka broker properties
  template:
    src: server.properties.j2
    dest: /opt/kafka/config/server.properties
    owner: kafka
    group: kafka
    mode: '0640'
  notify: restart kafka

Jinja2 template (server.properties.j2):

broker.id={{ broker_id }}
broker.rack={{ rack_id }}
listeners=INTERNAL://{{ ansible_default_ipv4.address }}:9092
advertised.listeners=INTERNAL://{{ inventory_hostname }}:9092
inter.broker.listener.name=INTERNAL
listener.security.protocol.map=INTERNAL:SSL

default.replication.factor=3
min.insync.replicas=2
unclean.leader.election.enable=false

log.dirs=/data/kafka
num.partitions=12
log.retention.hours=168

{% if kafka_mode == 'zookeeper' %}
zookeeper.connect={{ zookeeper_connect_string }}
{% else %}
controller.quorum.voters={{ controller_quorum_voters }}
controller.listener.names=CONTROLLER
{% endif %}

Automated Topic Provisioning

#!/bin/bash
# create-topic.sh - Automated topic creation with durable defaults
set -euo pipefail

TOPIC_NAME=$1
PARTITIONS=${2:-12}
REPLICATION_FACTOR=${3:-3}

kafka-topics.sh --bootstrap-server "${BOOTSTRAP_SERVERS}" \
  --create \
  --topic "${TOPIC_NAME}" \
  --partitions "${PARTITIONS}" \
  --replication-factor "${REPLICATION_FACTOR}" \
  --config min.insync.replicas=2 \
  --config retention.ms=604800000

# Show resulting replica placement
kafka-topics.sh --bootstrap-server "${BOOTSTRAP_SERVERS}" \
  --describe --topic "${TOPIC_NAME}"

With all brokers carrying a broker.rack, Kafka places the new topic’s replicas across racks automatically; the --describe output lets you confirm it.

Continuous Compliance Checks

Run periodically (e.g., hourly via cron or a Kubernetes CronJob). This example uses the kafka-python admin client:

from kafka.admin import KafkaAdminClient, ConfigResource, ConfigResourceType

admin = KafkaAdminClient(bootstrap_servers="broker1:9092")

topics = [t for t in admin.list_topics() if not t.startswith("_")]
resources = [ConfigResource(ConfigResourceType.TOPIC, t) for t in topics]
results = admin.describe_configs(resources)

for resource, response in zip(resources, results.resources if hasattr(results, "resources") else []):
    pass  # iterate per client version

Validate that each topic has min.insync.replicas >= 2 and unclean.leader.election.enable=false, and alert on violations. The exact response-parsing shape depends on your client library version, so test against your installed kafka-python release.

Conclusion

Multi-AZ Kafka deployment is a continuous practice spanning architecture, operations, and automation. The key principles:

1. Design for failure. Every decision should answer: “What happens when a zone fails?”
2. Place metadata for quorum survival. No single zone may hold a majority of KRaft controllers (or ZooKeeper ensemble members).
3. Test continuously. Regular zone-failure simulations are the only reliable way to validate your configuration.
4. Automate provisioning and compliance. Reduce human error from broker setup to topic creation.
5. Monitor at zone granularity. Invest in under-replicated/offline-partition visibility and tiered alerts.
6. Optimize cost without compromising resilience. Use compression, KIP-392 follower fetching, and tiered storage to manage inter-AZ transfer cost.

For further reading: Configuring Multi-AZ Kafka on AWS with EBS and Placement Groups, Cluster Architecture & Provisioning, the official Apache Kafka documentation, and the Confluent Multi-Datacenter Deployment Guide.