KRaft vs ZooKeeper: Choosing the Right Metadata Management

Few decisions in a Kafka deployment carry as much long-term operational weight as how cluster metadata is managed. For over a decade, Apache ZooKeeper was the coordination backbone: it managed controller elections, topic configurations, access control lists, and cluster membership. KRaft—Kafka’s built-in consensus layer based on the Raft protocol—removes that external dependency entirely.

As of Apache Kafka 4.0 (released March 2025), this is no longer an open choice for new clusters: ZooKeeper mode has been removed, and KRaft is the only supported metadata mode. ZooKeeper was deprecated in 3.5, and 3.9.x is the final release line that supports it. The practical question for platform and SRE teams in 2026 is therefore not “which should I pick for a greenfield cluster” but “how do I operate KRaft well, and how do I get my remaining ZooKeeper clusters onto a 4.x-compatible footing before the bridge release falls out of support.” This article compares the two systems to make those trade-offs concrete, and ties into broader decisions around Cluster Architecture & Provisioning.

The discussion assumes familiarity with Kafka fundamentals and focuses on engineering trade-offs. Commands and config keys here are grounded in the Apache Kafka KRaft operations documentation.

Architectural Foundations: How Metadata Consensus Works

Metadata is the authoritative state of the cluster: which brokers are alive, which broker is the controller, partition-to-replica mappings, leadership assignments, topic config overrides, and security state. Inconsistency here causes split-brain scenarios, unavailability, or data loss, so the consensus mechanism behind it is central to cluster reliability.

ZooKeeper-Based Metadata Management

In a ZooKeeper-managed cluster, metadata lives in a separate ensemble of ZooKeeper nodes—typically three or five servers running an independent process. Kafka brokers connect as ZooKeeper clients and use ZooKeeper’s atomic broadcast protocol (Zab) to read and write state. A single elected broker, the controller, holds an in-memory copy of the full metadata, synchronized from ZooKeeper through watches and ephemeral nodes. When a broker fails, the controller detects the ZooKeeper session expiration, recomputes partition leadership, and writes the new state back to ZooKeeper; other brokers then read it.

This creates a dependency chain: brokers depend on the controller, the controller depends on ZooKeeper, and ZooKeeper depends on its own quorum. An ensemble needs a majority available for writes. In a three-node ensemble, losing two nodes makes the Kafka cluster unable to accept metadata changes—no new topics, stalled reassignments, no controller failover. Operating that separate distributed system (its JVM heap pressure, transaction-log disk I/O, and sensitivity to inter-node latency) is real, recurring work.

The controller is also a single point of metadata propagation. A controlled shutdown performs a graceful handoff, but on an uncontrolled failure the new controller must rebuild its full view by reading every relevant znode from ZooKeeper. In clusters with tens of thousands of partitions this replay can take tens of seconds to minutes, during which leadership changes are blocked. This controller-failover delay is one of the most significant pain points in ZooKeeper-based deployments.

KRaft-Based Metadata Management

KRaft embeds the Raft consensus protocol directly in Kafka. A set of nodes designated as controllers forms a Raft quorum that owns the cluster metadata, stored as a single-partition, replicated, append-only log (the internal __cluster_metadata topic). The Raft leader among the controllers is the active controller; it processes all metadata writes and replicates them to follower controllers. When the leader fails, Raft elects a new leader from the remaining controllers, typically within a few election timeouts.

flowchart TD AC["Active controller (Raft leader)"] -->|"writes __cluster_metadata"| L["Metadata log"] AC -->|replicates| F1["Follower controller"] AC -->|replicates| F2["Follower controller"] F1 -->|"on leader failure, elect new leader"| AC B1["Broker"] -->|fetch metadata deltas| AC B2["Broker"] -->|fetch metadata deltas| AC

The simplification is real: instead of a separate ZooKeeper ensemble with its own configuration, monitoring, and failure modes, you operate one Kafka process type. The metadata log reuses Kafka’s existing storage and replication machinery, which removes an entire class of consistency issues that arose from mapping ZooKeeper’s hierarchical znode model onto Kafka’s log-structured state.

Failover is faster because standby controllers already hold the metadata log locally. A new leader reconstructs in-memory state from its local log and the latest snapshot rather than re-reading an external ensemble, so recovery is bounded by local I/O. In practice this is the difference between sub-second-to-seconds (KRaft) and tens of seconds (ZooKeeper) at large partition counts.

A minimal KRaft controller controller.properties:

# KRaft dedicated controller
process.roles=controller
node.id=1
controller.quorum.bootstrap.servers=controller-1:9093,controller-2:9093,controller-3:9093
controller.listener.names=CONTROLLER
listeners=CONTROLLER://0.0.0.0:9093
log.dirs=/var/lib/kafka/metadata

Kafka 3.9 (KRaft version 1, KIP-853) deprecated the static controller.quorum.voters property in favor of controller.quorum.bootstrap.servers, which enables dynamic quorums (adding/removing controllers without a full restart). On 3.8 and earlier you must still use the static form, e.g. controller.quorum.voters=1@controller-1:9093,2@controller-2:9093,3@controller-3:9093. The metadata log directory defaults to the first entry in log.dirs; set the optional metadata.log.dir only if you want it on a separate volume.

A combined broker/controller node (acceptable for small or non-critical clusters; the docs advise dedicated controllers for critical deployments so controllers and brokers scale independently):

# Combined broker and controller node (smaller / non-critical clusters)
process.roles=broker,controller
node.id=4
controller.quorum.bootstrap.servers=controller-1:9093,controller-2:9093,controller-3:9093
controller.listener.names=CONTROLLER
listeners=CONTROLLER://0.0.0.0:9093,PLAINTEXT://0.0.0.0:9092
advertised.listeners=PLAINTEXT://broker-4:9092
log.dirs=/var/lib/kafka/data

Operational Complexity: Provisioning and Day-Two Maintenance

Most of an SRE team’s metadata-related effort goes to day-two work—provisioning, upgrades, scaling, backup/restore, and incident response—rather than the initial install.

Provisioning and Automation

A ZooKeeper-based cluster means orchestrating two distributed systems. Automation must form the ZooKeeper ensemble (generate each node’s myid, distribute zoo.cfg, start nodes, and poll a health check such as the srvr/stat four-letter word until a leader appears) before any broker can register. The ordering is strict, which widens the surface for race conditions and transient failures in CI/CD.

KRaft’s quorum is self-forming: given a consistent controller.quorum.bootstrap.servers value, controllers discover each other and elect a leader automatically, so controllers and brokers can be started concurrently. The one mandatory step is formatting storage with a shared cluster ID. A minimal Terraform sketch:

# Terraform: KRaft dedicated controllers
resource "aws_instance" "kafka_controller" {
  count         = 3
  ami           = var.kafka_ami
  instance_type = "c6i.xlarge"

  user_data = templatefile("${path.module}/init-controller.sh.tpl", {
    node_id      = count.index + 1
    cluster_id   = var.kafka_cluster_id   # generated once: kafka-storage.sh random-uuid
    metadata_dir = "/var/lib/kafka/metadata"
  })

  tags = { Name = "kafka-controller-${count.index + 1}" }
}

Generate the cluster ID once (not per node—every node in a cluster must format with the same ID) and pass it in:

bin/kafka-storage.sh random-uuid
# e.g. MkU3OEVBNTcwNTJENDM2Qk

The bootstrap script formats storage idempotently and starts Kafka:

#!/usr/bin/env bash
# init-controller.sh - KRaft controller bootstrap
set -euo pipefail
KAFKA_HOME=/opt/kafka
CLUSTER_ID="${cluster_id}"

# --ignore-formatted makes this a no-op if storage is already formatted.
"$KAFKA_HOME/bin/kafka-storage.sh" format \
    --config "$KAFKA_HOME/config/controller.properties" \
    --cluster-id "$CLUSTER_ID" \
    --ignore-formatted

exec "$KAFKA_HOME/bin/kafka-server-start.sh" "$KAFKA_HOME/config/controller.properties"

This self-contained bootstrap aligns naturally with immutable-infrastructure patterns where nodes are replaced rather than repaired. (On Kafka 3.9+ with dynamic quorums you can instead format the first controller with --standalone and join the rest with --no-initial-controllers plus controller.quorum.auto.join.enable=true; the shared-cluster-ID format step shown above remains the portable baseline.)

Upgrades and Rolling Restarts

Upgrading a ZooKeeper-based cluster means sequencing two systems: roll the ZooKeeper ensemble one node at a time with quorum checks between restarts, then roll the Kafka brokers. The cross-system version matrix widens the maintenance window and the risk of compatibility surprises.

KRaft has one system to roll. The standard order is to upgrade the controller quorum first, then the brokers, one node at a time. Kafka coordinates feature levels through the metadata.version feature flag (set with kafka-features.sh) rather than the old inter.broker.protocol.version; you bump metadata.version only after every node is on the new binary. A typical sequence:

# 1. Upgrade controllers one at a time
for controller in controller-1 controller-2 controller-3; do
    ssh "$controller" "systemctl stop kafka"
    ssh "$controller" "yum update -y kafka"   # or apt / tarball swap
    ssh "$controller" "systemctl start kafka"
    # Confirm the node rejoined and the quorum has a leader before moving on
    ssh "$controller" "/opt/kafka/bin/kafka-metadata-quorum.sh \
        --bootstrap-controller $controller:9093 describe --status"
done

# 2. Upgrade brokers one at a time
for broker in broker-1 broker-2 broker-3 broker-4 broker-5 broker-6; do
    ssh "$broker" "systemctl stop kafka"
    ssh "$broker" "yum update -y kafka"
    ssh "$broker" "systemctl start kafka"
    sleep 60   # allow under-replicated partitions to recover before the next node
done

# 3. After all nodes are on the new version, raise the metadata.version feature
kafka-features.sh --bootstrap-server broker-1:9092 upgrade --metadata 4.0

kafka-metadata-quorum.sh exposes quorum health directly—describe --status for leader/epoch/high-watermark and voter state, describe --replication for per-replica lag—replacing the need to watch a separate ZooKeeper ensemble during the roll. (--bootstrap-controller targets the controller listener directly; --bootstrap-server against a broker also works.)

Monitoring and Observability

ZooKeeper monitoring needs its own metrics, dashboards, and alerts—outstanding requests, follower sync lag, ephemeral-node count, session expirations—with semantics distinct from broker metrics. A ZooKeeper ensemble degrading on latency may not trip any broker alert, so the root cause of cluster degradation can be invisible to the primary stack.

KRaft surfaces metadata health through Kafka’s existing JMX metrics. The well-documented signals:

  • kafka.controller:type=KafkaController,name=ActiveControllerCount — 1 on the active controller, 0 elsewhere; the cluster-wide sum should be exactly 1.
  • kafka.controller:type=ControllerEventManager,name=EventQueueTimeMs and ...,name=EventQueueProcessingTimeMs — controller queue wait and processing latency.
  • kafka.server:type=raft-metrics — Raft-layer gauges including commit-latency-avg/commit-latency-max, current-state (leader/follower/candidate/…), and current-leader (-1 = unknown).
  • kafka.server:type=MetadataLoader,name=CurrentControllerId — the controller ID each node currently sees (-1 if none).

All of these come through the same JMX endpoint and Prometheus JMX exporter already used for broker metrics. A KRaft alerting rule set:

groups:
  - name: kafka_kraft
    rules:
      - alert: KRaftNoActiveController
        # JMX-exporter name; the exact string depends on your exporter config
        expr: sum(kafka_controller_kafkacontroller_activecontrollercount) != 1
        for: 1m
        labels: { severity: critical }
        annotations:
          summary: "KRaft has no single active controller"
          description: "Cluster-wide ActiveControllerCount != 1; metadata writes may be blocked."

      - alert: KRaftHighCommitLatency
        # raft-metrics commit-latency-max is a gauge in ms (not a histogram)
        expr: max(kafka_server_raftmetrics_commit_latency_max) > 100
        for: 5m
        labels: { severity: warning }
        annotations:
          summary: "High KRaft commit latency"
          description: "Max Raft commit latency > 100ms, indicating quorum performance degradation."

Prometheus metric names depend entirely on your JMX-exporter mapping (lowercaseOutputName, regex rules, etc.). The Raft commit-latency metrics are exposed as avg/max gauges, so query them directly—there is no _bucket series and histogram_quantile() does not apply. Confirm the exact strings against your own /metrics output before deploying these rules.

This unified surface keeps metadata health in the same dashboards as data-plane health.

Scalability and Performance Characteristics

The metadata layer’s scaling limits cap the maximum practical size of a cluster.

Partition Count Limits

ZooKeeper-based clusters have a well-known practical ceiling—commonly cited around 200,000 partitions per cluster (with the controller, not ZooKeeper alone, as the usual limiter), varying with heap, network, and metadata churn. Several factors drive it: per-partition state is tracked individually and watched by the controller, so memory and watch overhead grow with partition count; an uncontrolled controller failover must re-read that state from ZooKeeper, which scales with total partitions; and ZooKeeper’s write path is effectively single-leader, capping metadata write throughput.

KRaft stores metadata as a compact binary log rather than individual znodes, and compacts it through periodic snapshots so that recovery only replays entries since the last snapshot rather than the entire history. This decouples recovery time from raw partition count and ties it to the rate of metadata change. KRaft has been demonstrated at substantially higher partition counts (Confluent and the Apache project have publicly tested into the millions of partitions per cluster) with recovery that stays bounded.

Inspect quorum and snapshot/replication state:

# Quorum summary: leader, epoch, high-watermark, voters/observers
kafka-metadata-quorum.sh --bootstrap-controller controller-1:9093 describe --status

# Per-replica replication detail (LogEndOffset, Lag, last-caught-up time)
kafka-metadata-quorum.sh --bootstrap-controller controller-1:9093 describe --replication

Write Throughput and Latency

Metadata writes include topic creation, partition reassignment, config changes, and broker registration. In ZooKeeper, creating a 100-partition topic means many separate quorum-acknowledged znode writes. KRaft batches metadata changes into Raft log records and appends them as sequential, page-cache-friendly disk I/O—the same write path Kafka uses for data—yielding materially higher metadata write throughput for equivalent operations.

KRaft also helps read-heavy metadata propagation: brokers fetch metadata deltas (KIP-631 records) and any follower controller holds the full committed log, so the leader is not the sole read source. In ZooKeeper, reads default to the leader (followers can serve slightly stale reads only if explicitly configured), which becomes a bottleneck at scale.

Network and Resource Efficiency

A production ZooKeeper node typically wants several GB of heap, fast dedicated disks for the transaction log (often separate from snapshots), and low inter-node latency. For a five-node ensemble that is meaningful infrastructure that does not directly serve Kafka throughput.

Dedicated KRaft controllers can run leaner because they handle only metadata, not data traffic—a small, fast SSD for the metadata log and modest CPU/RAM go a long way. On small clusters, colocating the controller role with brokers removes the dedicated tier entirely. The savings are most visible in edge or multi-cluster fleets where each cluster previously needed its own ensemble.

When planning controller hardware, the principles in Broker Sizing and Hardware Selection for Kafka Clusters apply, with one distinction: controllers do mostly sequential writes to the metadata log plus random reads during snapshot recovery, favoring low-latency SSDs for the metadata directory.

High Availability and Disaster Recovery

If metadata cannot be read or written, the cluster is unmanageable even when the data brokers are healthy. That makes the metadata layer the linchpin of availability across node failures, network partitions, AZ outages, and region loss.

Controller Failover Behavior

In ZooKeeper, failover is multi-phase: ZooKeeper detects the controller’s session expiration (after the configured session timeout, commonly 6–18 seconds), removes the controller ephemeral node, watching brokers are notified, an election runs, and the new controller reads full metadata from ZooKeeper. For ~100,000 partitions that read phase can take tens of seconds, during which no leadership changes can occur—so a broker failing concurrently leaves its partitions leaderless until recovery finishes.

KRaft failover is faster because followers already hold the metadata log. They detect missing heartbeats within an election timeout (on the order of hundreds of milliseconds with default Raft timeouts), elect a new leader, and the leader resumes processing from local state plus the latest snapshot. End-to-end failover is typically seconds, versus tens of seconds for ZooKeeper at comparable scale. (Treat specific millisecond figures as default-dependent rather than guaranteed; they move with controller.quorum.election.timeout.ms and related settings.)

This lets operators run tighter client timeouts and faster failure detection with KRaft, improving responsiveness during incidents.

Multi-AZ Deployment Considerations

Spanning AZs is standard for resilience, and the metadata quorum must span them too. For ZooKeeper, that means distributing ensemble members across three AZs (e.g., 1-1-1 for three nodes, or 2-2-1 for five). Every write needs a cross-AZ quorum ack, so inter-AZ latency directly affects write performance; where it climbs into the tens of milliseconds, throughput degrades noticeably.

KRaft controller quorums face the same physics but soften it through Raft pipelining—multiple records can be in flight, overlapping round-trips. Place one controller per AZ across three AZs so losing any single AZ still leaves a two-node majority. With dynamic quorums, list reachable controllers in controller.quorum.bootstrap.servers; the AZ topology is implicit in the hostnames. This pattern aligns with Multi-AZ Kafka Deployment Strategies for High Availability.

# Identical on all three controllers; node.id differs per node.
# controller-az1 / -az2 / -az3 resolve to one controller per AZ.
node.id=1
controller.quorum.bootstrap.servers=controller-az1:9093,controller-az2:9093,controller-az3:9093

Backup and Restore

Backing up ZooKeeper-based metadata usually means snapshotting the ZooKeeper data directory or exporting the tree, and restoring means stopping the ensemble, replacing the data directory, and restarting—with downtime. Because ZooKeeper and Kafka data live in separate systems, getting a consistent recovery point across both is awkward.

KRaft keeps metadata in a self-contained, append-only log on the controllers, so a file-system-level copy of the controllers’ metadata log.dirs (ideally via a filesystem snapshot for a consistent point) captures cluster state. There is no Kafka CLI that exports metadata to a portable backup file; kafka-metadata-quorum.sh describe is for inspection, not export. Treat the controller log directories—plus the cluster ID—as your backup artifact.

# Inspect (not export) current quorum state
kafka-metadata-quorum.sh --bootstrap-controller controller-1:9093 describe --status

# Re-provision a replacement controller against an existing cluster.
# Use the cluster's existing cluster-id; restore the metadata log from a
# filesystem snapshot, or let the node re-replicate from the quorum.
kafka-storage.sh format \
    --config /opt/kafka/config/controller.properties \
    --cluster-id <existing-cluster-id> \
    --ignore-formatted

Because metadata and data are versioned through the same Kafka mechanisms, a KRaft cluster can be rebuilt from a metadata backup plus the brokers’ existing data directories without coordinating recovery of two separate systems.

Migration: From ZooKeeper to KRaft

For teams still on ZooKeeper, migration is now a prerequisite for staying current: you cannot upgrade a ZooKeeper-based cluster directly to 4.0. The supported path is to migrate to KRaft on a 3.x release (3.9.x is the recommended bridge) and only then upgrade to 4.x. The migration framework (KIP-866) supports an online, rollback-capable transition.

Migration Architecture

Migration runs in a hybrid “dual-write” mode. You stand up KRaft controllers configured for migration; they take over as the active controller and copy existing metadata out of ZooKeeper into the Raft log, then write every subsequent change to both the KRaft quorum and ZooKeeper. That dual write is what makes rollback to ZooKeeper possible right up until the final step. Once brokers have been restarted into KRaft mode and the cluster is verified, you disable migration mode and decommission ZooKeeper.

The detailed, command-by-command procedure—including the broker/controller config flags and the order of restarts—is covered in the Step-by-Step Guide to Migrating from ZooKeeper to KRaft.

Version Prerequisites

ZooKeeper-to-KRaft migration shipped as early access in 3.4 (test only; no ACL migration, no rollback) and became production-ready in 3.6. For any real migration in 2026, run a recent 3.x bridge release—3.9.x is recommended, since it is the last line that supports ZooKeeper and the migration tooling and brings dynamic KRaft quorums. Confirm every broker is on the same migration-capable version before you start.

A pre-migration sanity check:

#!/usr/bin/env bash
# pre-migration-check.sh
set -euo pipefail
BOOTSTRAP="broker-1:9092"

echo "== Broker API versions (confirm uniform, migration-capable build) =="
kafka-broker-api-versions.sh --bootstrap-server "$BOOTSTRAP" | head -5

echo "== Partition count (sanity-check metadata size) =="
kafka-topics.sh --bootstrap-server "$BOOTSTRAP" --describe 2>/dev/null \
  | grep -c "Partition:"

echo "== ZooKeeper reachability and mode =="
ZK=$(grep '^zookeeper.connect' /opt/kafka/config/server.properties | cut -d= -f2 | cut -d, -f1)
ZK_HOST=${ZK%%:*}; ZK_PORT=${ZK##*:}
echo "stat" | nc "$ZK_HOST" "$ZK_PORT" | grep -i Mode || echo "ZooKeeper unreachable or 4lw disabled"

echo "== Disk headroom for the new metadata log =="
df -h /var/lib/kafka/metadata

Post-Migration Validation

After cutover and ZooKeeper decommission, exercise the metadata path end to end—quorum health, topic lifecycle, and produce/consume:

#!/usr/bin/env bash
set -euo pipefail
BROKER="broker-1:9092"
CONTROLLER="controller-1:9093"

# 1. Quorum health
kafka-metadata-quorum.sh --bootstrap-controller "$CONTROLLER" describe --status

# 2. Create a test topic
kafka-topics.sh --create --topic migration-test --partitions 10 \
    --replication-factor 3 --bootstrap-server "$BROKER"

# 3. Produce and consume one message
echo "test $(date)" | kafka-console-producer.sh --topic migration-test \
    --bootstrap-server "$BROKER"
kafka-console-consumer.sh --topic migration-test --from-beginning \
    --max-messages 1 --bootstrap-server "$BROKER"

# 4. Clean up
kafka-topics.sh --delete --topic migration-test --bootstrap-server "$BROKER"

Security and Access Control

The metadata layer holds ACLs and cluster-level security state, so the choice of backend affects how that state is stored, propagated, and recovered.

ACL Storage and Propagation

ZooKeeper-based clusters use AclAuthorizer (kafka.security.authorizer.AclAuthorizer), which stores ACLs as znodes under /kafka-acl and propagates changes through ZooKeeper watches—so ACL updates inherit ZooKeeper’s consistency and throughput characteristics, and a busy leader can bottleneck authorization changes.

KRaft uses StandardAuthorizer (org.apache.kafka.metadata.authorizer.StandardAuthorizer, introduced by KIP-801), which stores ACLs as records in the __cluster_metadata log. Each node serves authorization decisions from its local metadata cache, updated through Raft replication, so ACL changes propagate consistently with all other metadata and land in the same append-only audit trail.

Secure Deployment Patterns

Both backends support TLS and mTLS for inter-node traffic, but the configuration surface differs. ZooKeeper needs its own TLS setup (keystores, truststores, possibly a separate CA). KRaft reuses Kafka’s existing security configuration, so the controller listener can share the certificate and authentication machinery already used for broker-to-broker traffic.

A controller secured with mTLS:

process.roles=controller
node.id=1
controller.quorum.bootstrap.servers=controller-1:9093,controller-2:9093,controller-3:9093

# TLS on the controller listener
controller.listener.names=CONTROLLER
listeners=CONTROLLER://0.0.0.0:9093
listener.security.protocol.map=CONTROLLER:SSL
ssl.keystore.location=/etc/kafka/certs/controller.keystore.jks
ssl.keystore.password=${KEYSTORE_PASSWORD}
ssl.key.password=${KEY_PASSWORD}
ssl.truststore.location=/etc/kafka/certs/truststore.jks
ssl.truststore.password=${TRUSTSTORE_PASSWORD}
ssl.client.auth=required

# KRaft ACL authorizer
authorizer.class.name=org.apache.kafka.metadata.authorizer.StandardAuthorizer

This reuses the certificate infrastructure already managed for broker and client communication, reducing the number of distinct security artifacts to rotate.

Making the Decision in a Post-4.0 World

The forward-looking decision is effectively made: KRaft is the only metadata mode in Kafka 4.x, so every new cluster is KRaft, and every cluster you intend to keep current must reach KRaft.

New Clusters

Deploy KRaft. Beyond removing a separate distributed system, you get faster controller failover, higher partition ceilings, and a unified monitoring and security surface. KRaft is especially compelling for edge and resource-constrained deployments, where a dedicated ensemble was disproportionate, and for IaC/immutable-infrastructure shops, where the self-forming quorum collapses multi-phase provisioning into a single concurrent start.

Existing ZooKeeper Clusters

A stable, well-understood ZooKeeper cluster does not need to migrate today—but it does need a plan, because 3.9.x is the last line that runs it, and 4.0 has already removed ZooKeeper. The practical sequencing is to migrate to KRaft on a 3.9.x bridge release first, validate, then upgrade to 4.x. Combine the migration with an already-scheduled hardware refresh or major upgrade where possible to amortize the maintenance window. The authoritative references are the Apache Kafka KRaft documentation and KIP-866; the Kafka 4.0 upgrade guide documents the mandatory migrate-then-upgrade path.

Conclusion

The KRaft-versus-ZooKeeper question has resolved into a migration question. ZooKeeper served Kafka well for over a decade, but it carried the cost of a second distributed system—its own failure modes, monitoring, and scaling limits. KRaft folds metadata management into Kafka itself, reusing the same log-structured storage, replication, and consensus that make the data plane robust, and it is now the only supported mode from Kafka 4.0 onward.

For new clusters, deploy KRaft. For existing ZooKeeper clusters, the path is well defined: migrate to KRaft on a 3.9.x bridge using the KIP-866 dual-write framework, validate thoroughly, then upgrade to 4.x. As you plan, remember the metadata layer does not stand alone—it shapes topology, hardware, and automation decisions covered in Cluster Architecture & Provisioning. The operational practices that matter here are the unglamorous ones: thorough validation, staged rollouts, quorum-aware monitoring, and automated provisioning.

In this section

1 guide in this area.

Back to Cluster Architecture Provisioning