Implementing Rack Awareness in Apache Kafka for Fault Tolerance

In the domain of Cluster Architecture & Provisioning, few operational patterns matter more to production resilience than rack awareness. When you run Apache Kafka at scale, the physical reality of your infrastructure—racks of servers, top-of-rack switches, power distribution units, cloud availability zones—becomes a first-class input to your system’s ability to survive failure. Rack awareness is the mechanism by which Kafka acknowledges this topology and uses it to spread partition replicas across distinct failure domains, so that losing a single rack does not make data unavailable. This article is an operational guide for platform, SRE, and backend engineers who need to implement, validate, and maintain rack awareness in production clusters.

The concept is simple: assign each broker a broker.rack identifier, and Kafka spreads replicas of each partition across different racks whenever it can. The operational reality involves coordinating with your infrastructure metadata, understanding how Kafka’s placement algorithm behaves when racks and replicas don’t divide evenly, and validating that the topology you intended is the topology you actually have. We cover the placement mechanics, broker configuration, validation, multi-AZ trade-offs, Kubernetes, and failure-testing procedures.

A note on versions: ZooKeeper mode was removed in Apache Kafka 4.0 (released March 2025); 4.0 and later run KRaft only. The rack-awareness behavior described here is the same under both, but where the underlying metadata propagation differs, we call it out.

The Mechanics of Rack-Aware Replica Placement

To implement rack awareness effectively you first need to understand how Kafka uses rack information during replica assignment. This is what lets you reason about placements that don’t match your expectations and design topic configurations that match your durability goals.

Rack-aware placement runs when a topic is created and when partitions are reassigned—either manually via kafka-reassign-partitions.sh or when you generate a new assignment after a topology change. (Note that an ordinary broker failure does not automatically relocate replicas: Kafka elects a new leader from the surviving in-sync replicas, but the replica set is only changed by an explicit reassignment.) In all of these cases the goal is the same: distribute the replicas of a partition across as many distinct broker.rack values as possible.

The algorithm works per partition. For replication factor N, Kafka tries to place each of the N replicas on a broker in a rack not yet used for that partition. If the cluster has fewer distinct racks than the replication factor, some racks will necessarily host more than one replica, but Kafka still maximizes diversity. With RF=3 and only two racks, you get two replicas on one rack and one on the other—never three on a single rack.

Mechanically, Kafka uses a rack-alternating round-robin: it starts from an offset that shifts the assignment around the broker list (so leadership and replicas are spread, not all stacked on the lowest-numbered brokers), then walks the brokers placing each replica on a broker whose rack is not already in use for that partition, falling back to a less-balanced choice only when no rack-distinct broker is available. The practical takeaway is unchanged: to achieve true rack-level fault tolerance you need at least as many distinct racks as your replication factor.

The metadata layer differs by deployment but the placement logic does not. In a legacy ZooKeeper-managed cluster (Kafka 3.x and earlier), broker rack IDs live in ZooKeeper and are read by the controller. In a KRaft cluster, the rack is part of the broker registration record in the metadata log. For teams weighing the two, see KRaft vs ZooKeeper: Choosing the Right Metadata Management; rack awareness itself behaves identically either way.

Configuring Rack Awareness on Kafka Brokers

Implementation begins with assigning a broker.rack value to every broker. The value is a free-form string identifying the failure domain—physical rack, availability zone, or whatever boundary you protect against. It is set in the broker’s server.properties (or, in containers, via an environment variable consumed at startup).

The most important decision is the value itself. On premises this might be rack-12 or dc1-row4-rack7. In the cloud it is typically the availability zone, such as us-east-1a or eu-west-2c. The key requirement is consistency: every broker in the same failure domain must use the exact same string. A frequent and silent failure is using rack-12 and Rack-12 for the same rack—Kafka treats them as two racks and happily places “diverse” replicas that are actually co-located.

For bare-metal or VM deployments, add the line to each broker’s server.properties:

# Assign the broker to a failure domain (here, an AWS AZ)
broker.rack=us-east-1a

broker.rack is a read-only configuration: it is read once at broker startup and cannot be changed dynamically with kafka-configs.sh. Changing it requires editing server.properties and performing a rolling restart, one broker at a time, confirming each broker has rejoined the ISR before moving on. (If you have more than one broker per rack, avoid creating or altering topics while a rename is in flight.)

For fleets managed by configuration management, derive the rack from infrastructure metadata so it is never hand-edited. On AWS you can read the AZ from the Instance Metadata Service at boot. The example below uses IMDSv2 (token-based), which is the default on current instances:

#!/bin/bash
# Fetch a session token (IMDSv2), then read the availability zone
TOKEN=$(curl -s -X PUT "http://169.254.169.254/latest/api/token" \
  -H "X-aws-ec2-metadata-token-ttl-seconds: 60")
AZ=$(curl -s -H "X-aws-ec2-metadata-token: $TOKEN" \
  http://169.254.169.254/latest/meta-data/placement/availability-zone)
# Append the rack ID to the broker config before startup
echo "broker.rack=$AZ" >> /etc/kafka/server.properties

The same pattern applies on other clouds: read the zone from GCP’s metadata server or Azure’s Instance Metadata Service and write it into the config.

To confirm the cluster-wide broker-to-rack mapping, query the live cluster with the Admin client rather than parsing logs. kafka-broker-api-versions.sh --bootstrap-server localhost:9092 lists the registered brokers, and any AdminClient describeCluster call returns each Node with its rack(). On a KRaft cluster you can also inspect the metadata log offline with kafka-metadata-shell.sh, an interactive shell over a metadata snapshot or log directory:

# Open the metadata shell against the cluster-metadata log directory
kafka-metadata-shell.sh --directory /var/lib/kafka/__cluster_metadata-0
# or against a specific snapshot/checkpoint file:
# kafka-metadata-shell.sh --snapshot /var/lib/kafka/__cluster_metadata-0/00000000000000000123-0000000001.checkpoint

Inside the shell, navigate the metadata tree with ls, cat, and tree (for example cat /image/brokers) to see each broker registration, including its rack. There is no --command flag and no --snapshot ... --command "broker" invocation; that form does not exist. For non-interactive dumping of metadata records, use kafka-dump-log.sh --cluster-metadata-decoder --files <metadata-log-file>.

Validating Rack-Aware Replica Distribution

After configuring rack IDs, validate that Kafka actually places replicas the way you expect. Treat this as ongoing monitoring, not a one-time check—re-validate after topic creation, reassignment, and cluster expansion.

The starting point is kafka-topics.sh --describe, which shows the leader, replica list, and in-sync replicas (ISR) per partition:

kafka-topics.sh --bootstrap-server localhost:9092 \
  --describe --topic critical-events

This output identifies replicas by broker ID only—it does not print rack IDs, and there is no --rack option for kafka-topics.sh. To produce a rack-aware view you must join the broker IDs against their rack assignments, which you get from the cluster description. The cleanest way to do this is with the Admin API: describeCluster() returns each broker’s rack, and describeTopics() returns the replica assignment, so a short script can flag any partition whose replicas don’t span the required number of racks.

If you prefer the CLI, capture both views and join them. Note the broker-to-rack list comes from the running cluster, not from a metadata-shell --command invocation (which does not exist):

# 1. Describe the topic to get per-partition replica lists
kafka-topics.sh --bootstrap-server localhost:9092 \
  --describe --topic critical-events > topic-desc.txt

# 2. For a rack-aware check, look up each broker's rack via the Admin API
#    (kafka-broker-api-versions.sh / AdminClient.describeCluster), build a
#    brokerId -> rack map, and verify each partition's replicas span >= the
#    number of racks required by your RF. A few lines of Python using the
#    kafka-python or confluent-kafka AdminClient is the practical approach.

For continuous monitoring, integrate a rack check into your stack. Kafka exposes replica counts via JMX—kafka.server:type=ReplicaManager,name=PartitionCount and kafka.server:type=ReplicaManager,name=LeaderCount—but no built-in metric reports rack diversity. Build a small exporter that calls the Admin API on a schedule, computes per-topic rack diversity, and emits a custom gauge (e.g. kafka_partition_rack_diversity = 1 when every partition’s replicas are rack-distinct, 0 otherwise). Alert when it drops to 0 to catch misconfigurations and bad reassignments immediately.

The most convincing validation is a controlled rack failure: stop every broker in one rack and confirm all partitions still have a leader and meet min.insync.replicas. For RF=3 with min.insync.replicas=2, losing one rack must not under-replicate below the ISR floor or block producers—provided replicas are spread across at least three racks. With only two racks, a rack loss leaves some partitions with a single surviving replica, which violates min.insync.replicas=2 and blocks acks=all producers. This is the concrete reason your rack count, replication factor, and durability settings must be designed together.

Advanced Multi-Rack and Multi-AZ Deployment Strategies

In the cloud, rack awareness maps naturally onto availability zones. Each AZ has independent power, cooling, and networking, making it the right unit for broker.rack. Multi-AZ deployments add three concerns: inter-AZ latency, cross-AZ transfer cost, and the placement of controller/quorum nodes.

A common layout across three AZs places an equal number of brokers per zone: 9 brokers as 3+3+3 in us-east-1a/b/c, with broker.rack set to the AZ. With RF=3 and min.insync.replicas=2, this survives the complete loss of one AZ without data loss or producer blocking. Crucially, the metadata layer must survive the same loss: spread the controllers (or, in 3.x, the ZooKeeper ensemble) across AZs too.

Controller placement deserves attention. In a KRaft cluster, run an odd number of controllers—three, one per AZ—so the metadata quorum keeps a majority (2 of 3) through a single-AZ outage. (Tolerating f failures requires 2f+1 controllers.) If the active controller is in the failed AZ, KRaft elects a new one from the surviving voters with only a brief interruption. In a legacy ZooKeeper cluster the equivalent rule applies to the ZooKeeper ensemble, and the single elected controller broker re-elects from a surviving node.

Inter-AZ latency matters because followers fetch from leaders across zones. If the round-trip time between AZs is high, it raises end-to-end produce latency for acks=all. Kafka’s rack-aware placement does not optimize replication for locality—it only guarantees rack diversity, which by design forces cross-AZ replication. What you can optimize is consumer locality: configure the broker’s replica.selector.class to org.apache.kafka.common.replica.RackAwareReplicaSelector and set each consumer’s client.rack. Introduced in KIP-392 (Kafka 2.4), this lets a consumer fetch from a follower in its own AZ instead of the cross-AZ leader. To be precise about scope: it matches the consumer’s client.rack to a replica’s broker.rack and affects consumer fetches only—it does not change inter-broker replication, and it does not move leadership.

# Broker-side: enable rack-aware read-replica selection
replica.selector.class=org.apache.kafka.common.replica.RackAwareReplicaSelector

# Consumer-side: advertise the consumer's zone so it can read a local follower
client.rack=us-east-1a

Cross-AZ transfer cost is a real line item. In AWS, intra-region cross-AZ traffic is charged in both directions (commonly around $0.01–$0.02/GB per direction, varying by region). Because rack-aware placement deliberately puts replicas in different AZs, every acks=all produce crosses AZ boundaries to the followers, and the cost scales with throughput. Managing it is a durability-versus-budget trade-off: enable fetch-from-follower (above) to cut consumer cross-AZ traffic; use lower durability (RF=2 / min.insync.replicas=1) only for genuinely non-critical topics; or, for DR rather than HA, run separate per-region clusters linked by MirrorMaker 2 instead of stretching one cluster’s replication across zones.

Hardware choices feed directly into this. As covered in Broker Sizing and Hardware Selection for Kafka Clusters, under-provisioned network bandwidth turns the extra cross-AZ replication traffic into replication lag and ISR shrinks. Size NICs for peak replication plus client traffic, and use enhanced networking (AWS ENA, or higher-bandwidth GCP/Azure instance families) where throughput is high.

Rack Awareness in Containerized and Kubernetes Environments

Kubernetes complicates rack awareness because “rack” must be mapped onto a topology domain—usually the node’s zone. The Rack Awareness Setup in Kubernetes with Strimzi guide covers Strimzi in depth; the principles below apply to any Kubernetes deployment.

The natural mapping is the standard node label topology.kubernetes.io/zone, which cloud providers populate automatically. With Strimzi you configure this declaratively and the operator does the rest:

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: my-cluster
spec:
  kafka:
    replicas: 3
    rack:
      topologyKey: topology.kubernetes.io/zone
    # ... listeners, storage, config ...

Given this, Strimzi sets each broker’s broker.rack from the value of the node’s topology.kubernetes.io/zone label, and it automatically adds a node affinity rule so brokers only schedule onto nodes that carry the topology label. Strimzi does not, by default, force one broker per zone; if you want pods actively spread across zones, add a podAntiAffinity (or topologySpreadConstraints) rule under spec.kafka.template.pod:

spec:
  kafka:
    template:
      pod:
        affinity:
          podAntiAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              - labelSelector:
                  matchExpressions:
                    - key: strimzi.io/name
                      operator: In
                      values: ["my-cluster-kafka"]
                topologyKey: topology.kubernetes.io/zone

Remember that anti-affinity is a scheduling constraint, not a Kafka guarantee. With requiredDuringScheduling, a pod that cannot satisfy the rule stays Pending rather than landing in the wrong zone—make sure each zone actually has capacity, or use preferred and accept that diversity can degrade under resource pressure.

For non-Strimzi deployments (Confluent for Kubernetes, Bitnami/custom Helm charts), you wire the same thing by hand. The catch is that the Downward API can expose a Pod’s node name but not the node’s zone label—fieldRef only reads Pod fields, and spec.nodeName is the host name, not the AZ. So inject the node name and resolve it to a zone at startup:

env:
  - name: KAFKA_NODE_NAME
    valueFrom:
      fieldRef:
        fieldPath: spec.nodeName

In the broker entrypoint, use the node name to look up the node’s topology.kubernetes.io/zone label via the Kubernetes API and export it as the broker rack (the exact env var depends on the image; Confluent images map KAFKA_BROKER_RACK to broker.rack):

ZONE=$(kubectl get node "$KAFKA_NODE_NAME" \
  -o jsonpath='{.metadata.labels.topology\.kubernetes\.io/zone}')
export KAFKA_BROKER_RACK="$ZONE"

This requires the Pod’s ServiceAccount to have RBAC permission to get nodes (a ClusterRole + ClusterRoleBinding, since Node is cluster-scoped).

Finally, mind persistent volumes. Most cloud block volumes are zonal: if a broker Pod is rescheduled into a different zone, its existing PV cannot follow, so the StatefulSet’s volume affinity will keep the Pod pinned to its original zone. That preserves data but means a zone outage leaves that broker down until the zone returns—so your replication factor must already tolerate losing every broker in a zone. If you need a Pod to restart in another zone with its data, you need cross-zone-replicated storage; otherwise rely on Kafka’s own replication to keep the partition available from surviving zones while the failed broker waits for its zone (and volume) to recover.

Operational Procedures and Failure Testing

Rack awareness is not set-and-forget. The topology drifts as you add and replace hardware, and you need procedures that keep it accurate and proof that the system behaves under failure.

Reassignment during cluster expansion. Adding brokers does not rebalance existing partitions onto the new racks—Kafka only applies rack-aware placement to new assignments. To use new racks you generate and execute a reassignment with kafka-reassign-partitions.sh, throttled so it does not starve production traffic:

# 1. List the topics to move
echo '{"topics": [{"topic": "orders"}], "version": 1}' > topics-to-move.json

# 2. Generate a proposed plan for the target broker set.
#    --generate prints BOTH the current assignment (save it for rollback)
#    and a proposed assignment. Copy the "Proposed partition reassignment"
#    JSON into reassignment-plan.json before executing.
kafka-reassign-partitions.sh --bootstrap-server localhost:9092 \
  --topics-to-move-json-file topics-to-move.json \
  --broker-list "0,1,2,3,4,5" \
  --generate

# 3. Execute the reviewed plan with a throttle (bytes/sec; 104857600 = 100 MB/s)
kafka-reassign-partitions.sh --bootstrap-server localhost:9092 \
  --reassignment-json-file reassignment-plan.json \
  --throttle 104857600 \
  --execute

# 4. Verify progress; --verify also removes the throttle once complete
kafka-reassign-partitions.sh --bootstrap-server localhost:9092 \
  --reassignment-json-file reassignment-plan.json \
  --verify

The --generate step is a proposal printed to stdout (current plus proposed assignment); it is not a ready-to-run file, so review and extract the proposed JSON before --execute. The throttle is enforced via the leader.replication.throttled.rate / follower.replication.throttled.rate broker configs and the throttled-replica lists, which --verify cleans up on completion. After it finishes, re-run your rack-diversity validation.

Handling rack failures. When a rack drops, the controller detects the dead brokers and elects new leaders from the surviving ISR, so partitions stay available as long as a rack-distinct in-sync replica exists. Track controller health with the JMX MBean kafka.controller:type=KafkaController,name=ActiveControllerCount—it must sum to exactly 1 across the cluster; 0 means no controller, >1 means split brain. As the rack recovers, brokers rejoin and catch up, and some partitions will be under-replicated until they do. List them with:

kafka-topics.sh --bootstrap-server localhost:9092 \
  --describe --under-replicated-partitions

If brokers fell out of the ISR during a long outage, they must replicate the backlog from the leader, which can be heavy. You can temporarily raise the replication throttle to speed recovery (or set it generously, then clear it afterward):

# Raise the replication throttle on a recovering broker (200 MB/s here)
kafka-configs.sh --bootstrap-server localhost:9092 \
  --entity-type brokers --entity-name 3 \
  --alter --add-config 'leader.replication.throttled.rate=209715200,follower.replication.throttled.rate=209715200'

# Remove the throttle once the broker is caught up
kafka-configs.sh --bootstrap-server localhost:9092 \
  --entity-type brokers --entity-name 3 \
  --alter --delete-config 'leader.replication.throttled.rate,follower.replication.throttled.rate'

Chaos testing. Build confidence with regular, controlled rack failures. The lightest test isolates a rack at the network layer (firewall rules or Kubernetes NetworkPolicies) and observes failover: leaders move to surviving racks, acks=all producers with min.insync.replicas=2 keep succeeding, and consumers continue from the new leaders. Restore connectivity and confirm every partition returns to fully replicated. For a harder test, stop all broker processes in a rack simultaneously—this exercises both Kafka’s failover and your alerting and runbooks. Make sure on-call actually pages on under-replicated and offline partitions.

Monitoring and alerting. At minimum, alert on:

Under-replicated partitions — kafka.server:type=ReplicaManager,name=UnderReplicatedPartitions. Alert if non-zero for more than a few minutes.
Offline partitions — kafka.controller:type=KafkaController,name=OfflinePartitionsCount. Any non-zero value means a partition has no leader; page immediately.
Under-min-ISR partitions — kafka.server:type=ReplicaManager,name=UnderMinIsrPartitionCount. Non-zero means producers with acks=all are (or will be) blocked.
Active controller — ActiveControllerCount summed across brokers must equal 1.
Rack-diversity violations — your custom exporter from the validation section, alerting when any topic’s replicas are not rack-distinct.

Wire these into your incident system with runbook links that walk responders through assessing impact and driving recovery.

Conclusion

Rack awareness is foundational to Kafka fault tolerance, but it is more than a single config line. It requires understanding the per-partition placement algorithm, deriving broker.rack from authoritative infrastructure metadata, validating that the intended topology actually holds, and rehearsing failure. The procedures for expansion, rack failure, and chaos testing are core operations work, not optional extras.

These concerns intersect the rest of cluster management—from Cluster Architecture & Provisioning to Broker Sizing and Hardware Selection for Kafka Clusters and your choice of metadata layer. In Kubernetes, mapping racks onto topology domains adds work, but Strimzi automates the broker-side wiring. The principle stays constant across versions and deployments: know your physical topology, configure brokers to reflect it, and continuously verify that your replicas—and therefore your data—are where you think they are.

In this section

Rack Awareness Setup in Kubernetes with Strimzi