Rack Awareness Setup in Kubernetes with Strimzi

Introduction

Rack awareness distributes Kafka brokers and their partition replicas across distinct failure domains — racks, availability zones, or physical hosts — so that the loss of one domain cannot take all replicas of a partition offline. On Kubernetes, where pods are scheduled dynamically and can move between nodes, you cannot rely on a static broker.rack baked into server.properties. The rack ID must be derived from wherever the pod actually lands.

Strimzi, the Kubernetes operator for Apache Kafka, solves this by reading a topology label from the node a broker pod is scheduled on and using it to set broker.rack for that broker. This guide walks through configuring rack awareness in a Strimzi-managed cluster: the Kubernetes topology labeling it depends on, the Kafka custom resource configuration, the RBAC it requires, and how to verify that replicas actually land in different zones. It builds on Cluster Architecture & Provisioning principles and the platform-agnostic concepts in Implementing Rack Awareness in Kafka for Fault Tolerance.

How Strimzi Implements Rack Awareness

In a bare-metal or VM deployment, you set broker.rack per broker in server.properties. That works because the topology is stable and known up front. Kubernetes breaks that assumption: pods are rescheduled on node failure, resource pressure, or autoscaling, so a broker’s failure domain is only known at the moment the scheduler places its pod.

Strimzi uses topology-label-based rack awareness (type: topology-label). When this is enabled, each Kafka broker pod runs an init container that queries the Kubernetes API for the node it is scheduled on, reads the configured topology label from that node, and writes the value out for the broker container. Strimzi then sets broker.rack to that value before the broker starts. The broker registers its rack ID in the cluster metadata (KRaft or, on older clusters, ZooKeeper), and Kafka’s replica placement logic spreads replicas across as many distinct rack IDs as possible.

Two consequences of this design matter operationally:

  • Because the init container reads a Node object — a cluster-scoped resource — Strimzi needs cluster-level RBAC. The Cluster Operator binds the strimzi-kafka-broker ClusterRole (which grants get on nodes) to each broker’s service account via a ClusterRoleBinding. Rack awareness will not work if that binding is missing, which commonly happens when the operator is deployed with namespace-scoped permissions only.
  • The rack ID is computed at pod startup from the current node. If a broker is rescheduled into a different zone, its rack ID changes on the next start — which is correct for replica placement, but creates a storage-locality problem covered in the troubleshooting section.

The init container reads the node label through the Kubernetes API. It is not delivered through a Downward API volume — the Downward API can only expose pod fields and the pod’s own nodeName, not arbitrary node labels — which is exactly why the cluster-scoped RBAC above is required.

The configuration lives in the Kafka custom resource under spec.kafka.rack, taking a type and a topologyKey that names the node label to use.

Prerequisites and Kubernetes Topology Setup

The underlying cluster must expose accurate topology information through node labels. Rack awareness derives broker.rack from a node label such as the well-known topology.kubernetes.io/zone, or a custom label defined by your provisioning tooling.

Verifying node topology labels

Audit the labels on your worker nodes. Cloud providers populate the standard topology.kubernetes.io/* labels automatically:

kubectl get nodes -L topology.kubernetes.io/zone,topology.kubernetes.io/region

Example output from a three-zone cluster:

NAME                                       STATUS   ROLES    AGE   VERSION   ZONE         REGION
ip-10-0-1-45.us-east-2.compute.internal    Ready    <none>   9d    v1.29.4   us-east-2a   us-east-2
ip-10-0-2-178.us-east-2.compute.internal   Ready    <none>   9d    v1.29.4   us-east-2b   us-east-2
ip-10-0-3-92.us-east-2.compute.internal    Ready    <none>   9d    v1.29.4   us-east-2c   us-east-2

(-L <label> adds the label as an output column; it is more robust than custom-columns with escaped dots in the label key.)

Managed services (EKS, AKS, GKE) and most provisioning tools (kOps, Cluster API) set these labels by default. On-premises or custom clusters may need manual labeling to represent physical racks, chassis, or power domains.

Labeling nodes for custom failure domains

Where cloud zones do not apply, define a custom topology label. For physical rack positions:

kubectl label node worker-1 topology.example.com/rack=rack-12A
kubectl label node worker-2 topology.example.com/rack=rack-12B
kubectl label node worker-3 topology.example.com/rack=rack-14C

Use a domain-qualified key (topology.example.com/rack) to avoid collisions and document intent. Every node eligible to run a broker must carry the label: a broker scheduled on an unlabeled node starts with no rack ID and is treated as a single implicit rack, skewing replica placement.

Spreading broker pods with anti-affinity

Rack awareness sets the rack ID correctly, but it does not by itself force the scheduler to place broker pods in different zones. Configure pod anti-affinity in the Kafka resource template so two brokers do not land in the same zone:

spec:
  kafka:
    template:
      pod:
        affinity:
          podAntiAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              - labelSelector:
                  matchExpressions:
                    - key: strimzi.io/name
                      operator: In
                      values:
                        - my-cluster-kafka
                topologyKey: topology.kubernetes.io/zone

This tells the scheduler not to co-locate two broker pods on nodes sharing a topology.kubernetes.io/zone value. Combined with rack awareness, you get both physical spread and correct rack IDs.

Configuring Rack Awareness in the Kafka Custom Resource

Rack awareness is set under spec.kafka.rack. The type selects the mechanism (topology-label is the common one) and topologyKey names the node label.

Basic rack configuration

A complete, apply-ready manifest referencing the standard zone label:

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: my-cluster
spec:
  kafka:
    version: 3.7.0
    replicas: 3
    rack:
      type: topology-label
      topologyKey: topology.kubernetes.io/zone
    listeners:
      - name: plain
        port: 9092
        type: internal
        tls: false
      - name: tls
        port: 9093
        type: internal
        tls: true
    config:
      offsets.topic.replication.factor: 3
      transaction.state.log.replication.factor: 3
      transaction.state.log.min.isr: 2
      default.replication.factor: 3
      min.insync.replicas: 2
    storage:
      type: jbod
      volumes:
        - id: 0
          type: persistent-claim
          size: 100Gi
          deleteClaim: false
  zookeeper:
    replicas: 3
    storage:
      type: persistent-claim
      size: 100Gi
      deleteClaim: false

On apply, the Cluster Operator:

  1. Ensures the ClusterRoleBinding that lets each broker’s service account read its Node exists.
  2. Adds the rack init container to each broker pod. At startup it reads the value of the topologyKey label from the node and makes it available to the broker container.
  3. Sets broker.rack to that value for each broker, which the broker then registers with the cluster metadata.

No manual action on the broker pods is required.

If you manage brokers with KafkaNodePool resources, set rack on the Kafka resource as above; the rack configuration applies cluster-wide and is not duplicated per node pool.

Using a custom topology label

To use a custom label, change topologyKey to match:

spec:
  kafka:
    rack:
      type: topology-label
      topologyKey: topology.example.com/rack

This adapts rack awareness to any failure-domain taxonomy — physical racks, rooms, power grids, or logical groupings.

Verifying the rack value on running brokers

Changing the rack configuration triggers a rolling restart of the broker pods. Watch it:

kubectl get pods -l strimzi.io/name=my-cluster-kafka -w

Once the brokers are up, confirm each picked up its rack ID. The most direct check is to read the rendered broker config inside the pod. Strimzi writes the generated configuration to a file in the broker container; locate the active config and grep for the property:

kubectl exec -it my-cluster-kafka-0 -- \
  bash -c 'grep -R "^broker.rack" /tmp/strimzi.properties /opt/kafka/custom-config 2>/dev/null'

The exact path varies by Strimzi version, so the grep -R over both common locations is deliberate. You should see a line like broker.rack=us-east-2a.

Do not rely on kafka-configs.sh --describe --entity-type brokers to read broker.rack. That command returns only dynamic broker configs; broker.rack is a read-only, static, per-broker property and does not appear in its output. Likewise, kafka-broker-api-versions.sh reports supported API versions, not broker configuration — it never shows rack information.

For a KRaft-based cluster, the authoritative source is the cluster metadata. Use kafka-metadata-shell.sh (note: the tool is kafka-metadata-shell.sh, not kafka-metadata.sh) against the metadata log directory, then read a broker’s registration node:

kubectl exec -it my-cluster-kafka-0 -- bin/kafka-metadata-shell.sh \
  --directory /var/lib/kafka/data/kafka-log0/__cluster_metadata-0 \
  ls /brokers
# then, inside or via a one-shot command:
kubectl exec -it my-cluster-kafka-0 -- bin/kafka-metadata-shell.sh \
  --directory /var/lib/kafka/data/kafka-log0/__cluster_metadata-0 \
  cat /brokers/0/registration

kafka-metadata-shell.sh accepts --directory (a metadata log directory) or --snapshot (a single snapshot/checkpoint file) and exposes the metadata as a filesystem-like tree navigated with ls, cat, cd, and find. It does not take a --cluster-id or --command flag. The broker’s registration record includes its rack.

The broker IDs and racks reported here are what you cross-reference against replica assignments next.

Verifying Rack-Aware Replica Placement

Configuring rack awareness is only half the job; confirm Kafka actually spreads replicas across racks.

Create a test topic

Use a replication factor that matches the number of failure domains. For three zones, RF 3:

kubectl exec -it my-cluster-kafka-0 -- bin/kafka-topics.sh \
  --bootstrap-server localhost:9092 \
  --create \
  --topic rack-aware-test \
  --partitions 6 \
  --replication-factor 3

When broker.rack is set on the brokers, Kafka’s replica placement spreads each partition’s replicas across as many distinct racks as possible, only doubling up within a rack when there are fewer racks than the replication factor.

Inspect replica distribution

kubectl exec -it my-cluster-kafka-0 -- bin/kafka-topics.sh \
  --bootstrap-server localhost:9092 \
  --describe \
  --topic rack-aware-test

Example:

Topic: rack-aware-test  Partition: 0  Leader: 2  Replicas: 2,0,1  Isr: 2,0,1
Topic: rack-aware-test  Partition: 1  Leader: 0  Replicas: 0,1,2  Isr: 0,1,2

Cross-reference each partition’s replica list against the broker-to-rack mapping you obtained from the metadata shell. In a correctly configured three-zone cluster, every partition’s three replicas should sit on brokers in three different zones, and no partition should have two replicas in one zone.

Test fault tolerance with a simulated zone failure

You can simulate the loss of a zone by cordoning and draining its nodes. Do this in a non-production cluster, or with great care:

# Identify nodes in a specific zone
kubectl get nodes -l topology.kubernetes.io/zone=us-east-2a -o name

# Prevent new scheduling onto them
kubectl cordon <node-name>

# Evict existing pods
kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data

After draining, confirm topics stay available and that the in-sync replica count for each partition does not fall below min.insync.replicas (2 in the manifest above). With RF 3 across three zones, losing one zone leaves two in-sync replicas, so producers using acks=all continue to commit.

Troubleshooting Common Misconfigurations

Missing RBAC for node access

If rack awareness silently fails to set broker.rack — replicas land without rack spreading and the broker config shows no rack line — the most common cause on locked-down clusters is missing the strimzi-kafka-broker ClusterRoleBinding. The broker init container cannot read its Node without it.

Diagnosis: Inspect the init container logs for a forbidden/permissions error:

kubectl logs my-cluster-kafka-0 -c kafka-init

A nodes is forbidden message confirms the issue.

Remediation: Deploy the operator’s full RBAC (the strimzi-kafka-broker ClusterRole and its ClusterRoleBinding). Namespace-scoped (RoleBinding-only) operator installs cannot grant access to cluster-scoped Node objects.

Missing or inconsistent node labels

A broker on a node lacking the topologyKey label starts with no rack ID and is treated as one implicit rack, skewing placement.

Diagnosis: Find nodes missing the label:

kubectl get nodes -o json | jq -r '.items[] | select(.metadata.labels["topology.kubernetes.io/zone"] == null) | .metadata.name'

Remediation: Label all eligible worker nodes. If a node legitimately has no failure domain, exclude it from the broker pool with node selectors or taints rather than leaving it unlabeled.

Pod anti-affinity conflicts

If anti-affinity is too strict for the available topology, pods stay Pending.

Diagnosis:

kubectl describe pod my-cluster-kafka-2

Look for events such as 0/9 nodes are available: 3 node(s) didn't match pod anti-affinity rules.

Remediation: Add nodes in distinct failure domains, relax requiredDuringSchedulingIgnoredDuringExecution to preferredDuringSchedulingIgnoredDuringExecution, or reduce broker replicas. Scaling the node pool is preferable to weakening the rule in production.

Rack/storage zone mismatch after rescheduling

When a broker is rescheduled into a different zone, its rack ID updates on the next start — but a zone-pinned PersistentVolume stays in its original zone. The result is a broker in one zone reading data in another, adding cross-zone latency and transfer cost (and, with single-zone storage classes, the pod may not schedule at all).

Diagnosis: Compare the broker pod’s node zone with its PV zone:

# Node zone of the running pod
kubectl get pod my-cluster-kafka-0 -o jsonpath='{.spec.nodeName}' \
  | xargs -I{} kubectl get node {} -o jsonpath='{.metadata.labels.topology\.kubernetes\.io/zone}{"\n"}'

# Zone the PV is pinned to (via its nodeAffinity)
kubectl get pv "$(kubectl get pvc data-0-my-cluster-kafka-0 -o jsonpath='{.spec.volumeName}')" \
  -o jsonpath='{.spec.nodeAffinity.required.nodeSelectorTerms[*].matchExpressions[*].values[*]}{"\n"}'

(The PVC name follows Strimzi’s data-<volumeId>-<cluster>-kafka-<brokerId> convention; adjust for your storage configuration. Zone-aware PVs record their zone in spec.nodeAffinity, not in a label.)

Remediation: Use a storage class with volumeBindingMode: WaitForFirstConsumer so the volume is provisioned in the same zone as the pod that first consumes it, and set allowedTopologies to constrain provisioning. This keeps each broker’s storage co-located with the only zone in which its pod can run.

Stale rack in KRaft metadata

In KRaft clusters, broker rack lives in the metadata log as part of each broker’s registration. A rack change takes effect when the broker re-registers (on restart), so transient mismatches resolve themselves.

Diagnosis: Read the registrations and compare against current node zones:

kubectl exec -it my-cluster-kafka-0 -- bin/kafka-metadata-shell.sh \
  --directory /var/lib/kafka/data/kafka-log0/__cluster_metadata-0 \
  cat /brokers/0/registration

Remediation: A controlled rolling restart of the affected broker forces re-registration with the correct rack. See the Apache Kafka KRaft documentation for metadata management details.

Operational Considerations

Monitor rack distribution over time

Replica distribution drifts as topics are created and brokers are added or replaced. The JMX metric kafka.server:type=ReplicaManager,name=UnderReplicatedPartitions (and the cluster-wide UnderMinIsrPartitionCount) catches replication problems generally, but neither proves rack diversity. For that, run a periodic job that pulls replica assignments via the Admin API (or kafka-topics.sh --describe), maps broker IDs to racks, and alerts on any partition whose replicas share a rack. Strimzi can expose Kafka’s JMX metrics to Prometheus via a metricsConfig (JMX Exporter) configuration on the Kafka resource.

Cluster expansion and broker replacement

When adding brokers, label the new nodes first and confirm anti-affinity still admits the larger replica set. Strimzi performs the rolling update, but verify the new brokers register with the expected rack IDs. New partitions placed after expansion are rack-aware; existing partitions are not moved automatically.

To rebalance existing partitions across the larger broker set, use Cruise Control. Strimzi integrates it through the cruiseControl section on the Kafka resource and the KafkaRebalance custom resource, which generates and applies an optimization proposal. Cruise Control’s default goals include RackAwareGoal, so rebalances preserve rack diversity. (The lower-level kafka-reassign-partitions.sh tool can also do this manually, but does not optimize for rack on its own.)

GitOps and infrastructure as code

Keep the Kafka resource and node labels in version control and reconcile with Argo CD or Flux for reproducible, auditable rack configuration. Apply topology labels as part of node provisioning so they are never forgotten. A Terraform EKS node group can set zone labels at creation:

resource "aws_eks_node_group" "kafka" {
  # ... other configuration ...
  labels = {
    "topology.kubernetes.io/zone" = "us-east-2a"
  }
}

(EKS managed node groups span the subnets you give them; pin a node group to a single zone — one subnet — when you need a deterministic zone label per group rather than interpolating one value across a multi-AZ group.)

Periodic disaster-recovery testing

Validate rack awareness regularly rather than assuming it. Schedule controlled zone-failure drills — cordon/drain as shown above, or use a chaos tool such as LitmusChaos or Gremlin — and check the expected outcome: topics stay available, no partition drops below min.insync.replicas, and leadership re-elects automatically. These drills surface hidden single-zone dependencies (hardcoded broker endpoints, single-AZ monitoring collectors) that undermine resilience.

Align with broader architecture

Rack awareness is one layer of a fault-tolerance strategy that also spans broker sizing, storage tiering, and network topology — see Cluster Architecture & Provisioning. For the replica-placement algorithm and the trade-offs behind it, see Implementing Rack Awareness in Kafka for Fault Tolerance.

Conclusion

On Kubernetes, a broker’s failure domain is decided by the scheduler, so rack awareness has to be derived at pod startup rather than hardcoded. Strimzi does this with an init container that reads a topology label from the node and sets broker.rack for each broker — provided the cluster-scoped RBAC that lets brokers read their Node is in place.

A working setup is the sum of several parts: accurate node topology labels, the rack block on the Kafka resource, the strimzi-kafka-broker ClusterRoleBinding, anti-affinity to spread pods, and zone-aware storage so a rescheduled broker does not get stranded from its data. Verify the result the right way — read broker.rack from the rendered broker config or the KRaft metadata shell, not from kafka-configs.sh — then prove it end to end by checking replica assignments and rehearsing a zone failure. Treat those checks as recurring operational work, not a one-time task, and rack awareness keeps delivering the availability guarantees it promises.

Back to Implementing Rack Awareness In Kafka For Fault Tolerance