Performing a Rolling Upgrade from Kafka 2.8 to 3.5

This guide is a step-by-step procedure for upgrading a ZooKeeper-managed Apache Kafka cluster from 2.8 to 3.5 with no downtime for producers or consumers. The mechanism is Kafka’s standard rolling restart combined with a pinned inter.broker.protocol.version (IBP), which keeps newly upgraded 3.5 brokers speaking the 2.8 inter-broker protocol until every node has been upgraded. This work sits within Operations, Security & Observability and builds on the principles in Rolling Upgrades and Maintenance Without Downtime.

The procedure follows the official Apache Kafka 3.5 upgrade guide. Use the latest 3.5 patch release, 3.5.2, rather than 3.5.0; the steps are identical.

Pre-Upgrade Compatibility and Dependency Audit

Before touching a broker, confirm that your clients, connectors, and stream-processing apps support Kafka 3.5. The upgrade documentation is the authoritative compatibility reference.

Two version-specific gotchas matter here:

  • Scala 2.12 and Java 8 are deprecated (since 3.0) and are removed in 4.0. They still work in 3.5, but if you build against the Scala 2.12 artifact, plan to move to 2.13 now. This guide uses the kafka_2.13 build.
  • log.message.format.version is deprecated and ignored once inter.broker.protocol.version is 3.0 or higher (KIP-724). The on-disk message format is assumed to be the latest, so a 2.8→3.5 upgrade should not touch this property unless you have explicitly overridden it to an old value — see the note below.

Confirm the cluster is healthy and fully replicated before starting. Do not begin if any partition is under-replicated.

# Confirm broker versions / reachability
kafka-broker-api-versions.sh --bootstrap-server broker1:9092 | head -5

# There must be zero under-replicated partitions before you start
kafka-topics.sh --bootstrap-server broker1:9092 --describe --under-replicated-partitions

Audit your security configuration, since most failed rolling upgrades surface as a broker that cannot rejoin:

  • TLS: if you use mTLS, verify the truststore contains the required CA chain and that ssl.endpoint.identification.algorithm (hostname verification) is consistent across brokers.
  • SASL/SCRAM: confirm the enabled mechanisms (SCRAM-SHA-256 / SCRAM-SHA-512) and the sasl.enabled.mechanisms / sasl.mechanism.inter.broker.protocol settings match on every broker. A mismatch causes the broker to fail the inter-broker handshake on restart.

Staging the New Binary and Configuration

Install the 3.5.2 binary on every broker alongside the existing 2.8 installation. Do not overwrite 2.8 — keeping it in place is what makes a per-broker rollback a one-line symlink swap.

# On each broker node
cd /opt
wget https://archive.apache.org/dist/kafka/3.5.2/kafka_2.13-3.5.2.tgz
tar -xzf kafka_2.13-3.5.2.tgz
# Do NOT repoint the /opt/kafka symlink yet — that happens per-broker during the restart loop

Copy your existing 2.8 server.properties into the 3.5.2 config directory and add a single property to pin the inter-broker protocol to 2.8:

# server.properties — pin IBP for the duration of the binary upgrade
inter.broker.protocol.version=2.8

This is the only override required for a standard 2.8→3.5 upgrade. With it set to 2.8, an upgraded 3.5 broker advertises and speaks the 2.8 inter-broker protocol, so it interoperates with brokers still running 2.8.

Do not set log.message.format.version. Because the upgrade pins IBP and you almost certainly use the default (latest) message format on 2.8, this property is unnecessary. The official guide is explicit: “If you are upgrading from 0.11.0.x or above, and you have not overridden the message format, then you only need to override the inter-broker protocol version.” The property is deprecated and removed in 4.0; the value is assumed to be the latest format whenever IBP is 3.0+. Only carry it forward if you had deliberately set an old format on 2.8, and clear it as part of this upgrade.

Review the 3.5.0 notable changes for any deprecated or removed properties that affect your config before you proceed.

Executing the Rolling Restart Sequence

Restart brokers one at a time, waiting for each to rejoin and return the cluster to zero under-replicated partitions before moving on. The controller tolerates any restart order; restarting non-controller brokers first simply avoids extra controller failovers, so do the active controller last.

For each broker:

  1. Stop the broker. kafka-server-stop.sh sends SIGTERM. With controlled.shutdown.enable=true (the default), this triggers a controlled shutdown: the broker migrates its partition leadership away before exiting. Wait for the process to exit and confirm leadership has moved in the logs — there is no separate “drain” command.
  2. Switch the binary. Point /opt/kafka at the 3.5.2 install.
  3. Start the broker with the pinned server.properties.
  4. Verify the broker rejoins and all its replicas catch up before continuing.
# On the target broker

# 1. Controlled shutdown via SIGTERM
kafka-server-stop.sh
# Wait for the process to exit and for "Controlled shutdown succeeded" in server.log

# 2. Repoint the symlink to 3.5.2
ln -sfn /opt/kafka_2.13-3.5.2 /opt/kafka

# 3. Start the upgraded broker
kafka-server-start.sh -daemon /opt/kafka/config/server.properties

# 4. Confirm it is serving
kafka-broker-api-versions.sh --bootstrap-server localhost:9092 | head -5

Watch server.log during startup, especially the SASL/TLS handshakes and ZooKeeper session establishment. If the broker does not join, the cause is almost always a security-config mismatch or a TLS trust problem; the security documentation lists the required settings per mechanism.

After each restart, confirm full replication before touching the next broker:

kafka-topics.sh --bootstrap-server broker1:9092 --describe --under-replicated-partitions

Per-broker cycle time is dominated by replica catch-up; size your wait on the broker’s data volume and num.recovery.threads.per.data.dir, not a fixed clock.

Advancing the Inter-Broker Protocol Version

Once every broker runs 3.5.2 binaries with inter.broker.protocol.version=2.8, and after you have verified cluster behavior and performance, bump the IBP. This is a deliberate second rolling restart because the inter-broker protocol is negotiated per connection.

Set inter.broker.protocol.version=3.5 on each broker and restart it, one at a time, in the same controlled fashion as before.

# On each broker, then restart it (controlled shutdown + start)
sed -i 's/^inter.broker.protocol.version=2.8/inter.broker.protocol.version=3.5/' /opt/kafka/config/server.properties

kafka-server-stop.sh           # controlled shutdown
kafka-server-start.sh -daemon /opt/kafka/config/server.properties

The accepted value is 3.5 (equivalently 3.5-IV2); 3.5.0 is not the canonical form. After the final broker restarts, the IBP advance is complete — there is no separate message-format step to perform.

You can confirm the negotiated API versions afterward:

kafka-broker-api-versions.sh --bootstrap-server broker1:9092 | head -5

Post-Upgrade Validation and Monitoring

With the IBP advanced, validate that the cluster is healthy on 3.5.

Confirm consumer groups survived the upgrade and are not stuck:

# Spot-check group state and lag (run --describe on the groups that matter)
kafka-consumer-groups.sh --bootstrap-server broker1:9092 --list
kafka-consumer-groups.sh --bootstrap-server broker1:9092 --describe --group my-critical-group

Watch the standard JMX metrics during and after the upgrade. The ones that catch upgrade problems first:

  • kafka.server:type=ReplicaManager,name=UnderReplicatedPartitions — must return to 0.
  • kafka.controller:type=KafkaController,name=ActiveControllerCount — must be exactly 1 across the cluster.
  • kafka.controller:type=KafkaController,name=OfflinePartitionsCount — must be 0.
  • kafka.server:type=ReplicaManager,name=UnderMinIsrPartitionCount — must be 0.

If you scrape these with the Prometheus JMX exporter into Grafana, confirm every broker is still being scraped after its restart.

Finally, run end-to-end traffic from a representative client: produce to a non-production topic and consume it back. Pay attention to any custom partitioners or serializers that depend on version-specific client APIs.

Rollback Procedures and Contingency Planning

Have a tested rollback before you start. Because the 2.8 binaries are still in place, rolling back a broker is a symlink swap — but only while its IBP is still 2.8.

# On the broker to roll back (IBP still 2.8)
kafka-server-stop.sh
ln -sfn /opt/kafka_2.13-2.8.0 /opt/kafka
kafka-server-start.sh -daemon /opt/kafka/config/server.properties

The hard constraint: once you advance the IBP to 3.5, you cannot downgrade a broker to 2.8, because a 2.8 broker cannot speak the 3.5 inter-broker protocol and will fail to join. The IBP bump is therefore the point of no return for binary downgrade. Before that point, rollback is trivial; treat the verification window between the two rolling restarts as your real soak test, and do not bump IBP until you trust the new binaries.

Common failure modes during the upgrade:

  • ZooKeeper session timeouts under metadata load — recoverable by rolling back the affected broker and revisiting zookeeper.session.timeout.ms and zookeeper.connection.timeout.ms. (Check the 3.5.0 notable changes for any deprecated ZooKeeper-related properties before assuming a value carries forward unchanged.)
  • SASL/SCRAM failures from config drift between brokers.
  • TLS handshake errors from inconsistent cipher-suite or hostname-verification settings.

Each is recoverable: roll the affected broker back (while IBP is still 2.8), fix the configuration, and re-attempt.

A 2.8→3.5 rolling upgrade is a structured, repeatable process: pin the IBP, roll the binaries one broker at a time, verify, then advance the IBP in a second rolling restart. The discipline that preserves availability is incremental change and continuous validation against under-replicated and offline-partition metrics — exactly the practices in Rolling Upgrades and Maintenance Without Downtime.

Back to Rolling Upgrades And Maintenance Without Downtime