Rolling Upgrades and Maintenance Without Downtime

Part of Operations, Security & Observability, this guide covers how to upgrade and maintain an Apache Kafka cluster without interrupting client traffic. Kafka tolerates planned maintenance by design, but a clean rolling restart still demands a methodical sequence, an understanding of the replication protocol, and continuous monitoring. The procedures here target platform, SRE, and backend engineers running Kafka in KRaft mode (the only mode supported from Kafka 4.0 onward; ZooKeeper mode was removed).

The core principle is that Kafka keeps serving producers and consumers as long as an in-sync replica (ISR) remains available for every partition. By restarting brokers one at a time and waiting for the cluster to fully recover between restarts, you can upgrade an entire cluster while clients see nothing worse than a transient metadata refresh. The word “between” is where most failed upgrades go wrong: restart the next broker before the previous one has rebuilt its replicas, and you stack two replicas’ worth of unavailability onto the same partitions. This article covers the full lifecycle — pre-upgrade preparation, the rolling restart, version/feature compatibility, monitoring, and post-upgrade validation — and the failure modes that actually bite: leadership churn, under-replicated partitions that won’t clear, and security config that diverged between old and new binaries.

How Kafka Stays Available During Maintenance

Kafka survives broker restarts without data loss or unavailability because of its replication model. Each topic partition has a leader replica and a configurable number of followers spread across brokers. Producers write to the leader; followers replicate the leader’s log. The replicas that are fully caught up form the in-sync replica set (ISR). As long as at least one ISR member stays available, the partition keeps accepting writes (subject to min.insync.replicas) and serving reads.

When a broker goes offline, the controller — in KRaft mode, the active controller elected from the controller quorum — detects the departure and triggers a leader election for any partitions whose leader was on that broker. For partitions with a healthy ISR this is fast, typically completing in well under a second. Clients fetch updated metadata and reconnect to the new leaders. A seamless rolling restart depends on keeping those leadership transitions clean and ensuring the cluster has enough headroom to absorb the temporary loss of one broker.

The numbers that govern safety

Three settings decide whether maintenance is safe, and they have to be reasoned about together, not in isolation:

min.insync.replicas is a topic-level setting (also settable as a broker default that topics inherit). With replication factor 3 and min.insync.replicas=2, a partition tolerates exactly one broker down: writes still commit because 2 replicas remain in-sync. The combination RF=3, min.insync.replicas=2, acks=all is the standard safe baseline for rolling maintenance because it leaves one unit of headroom — the broker you are restarting. Drop to RF=2 or raise min.insync.replicas to 3 and you have zero headroom: the moment you stop a broker, every partition it led falls below the threshold and acks=all producers fail with NotEnoughReplicas/NotEnoughReplicasAfterAppend. Note that min.insync.replicas only constrains acks=all producers; acks=1 or acks=0 producers keep writing (and silently risk data loss), so the setting protects durability, not availability, for those clients.

# Topic-level configuration for safe rolling maintenance (assumes replication.factor=3)
kafka-configs --bootstrap-server localhost:9092 \
  --entity-type topics --entity-name critical-events \
  --alter --add-config min.insync.replicas=2

The catch is what happens to ISR during the restart, not after. When you stop a broker, its replicas don’t leave the ISR instantly — the leader removes a follower only after it stops fetching for replica.lag.time.max.ms (default 30 seconds). So for up to ~30 seconds after a clean shutdown, the partition’s recorded ISR still lists the departed broker, and a second failure in that window can briefly push the partition below min.insync.replicas. This is the concrete reason “wait for full recovery between brokers” matters: you are not just waiting for catch-up, you are waiting for the ISR to settle back to its full membership before you remove the next replica. The Apache Kafka replication documentation is the authoritative reference for these guarantees.

Pre-Upgrade Preparation and Compatibility Checks

A rolling upgrade begins well before the first restart. Skipping preparation is the most common cause of extended downtime during what should be routine work.

Version and Feature Compatibility

In a KRaft cluster there is no inter-broker protocol version (IBP) to manage; inter.broker.protocol.version and log.message.format.version were ZooKeeper-era controls. (log.message.format.version has been a no-op since 3.0 — assumed to be 3.0 whenever the protocol is 3.0 or higher — and was removed in 4.0 per KIP-724.)

KRaft instead uses cluster-wide feature flags, the most important being metadata.version. The software upgrade and the metadata.version upgrade are decoupled: you upgrade the broker binaries first, leaving metadata.version where it is, and only finalize it once every node runs the new release. Finalizing is a dynamic operation — no second rolling restart is required. Check the current finalized version before you start:

# Show finalized feature versions, including metadata.version
kafka-features --bootstrap-server localhost:9092 describe

The decoupling is what makes the upgrade reversible up to a point. While the cluster runs mixed binaries but the old metadata.version, you can roll back by reinstalling the old binaries — nothing new has been written to the metadata log that the old code can’t read. Once you finalize, that escape hatch closes: a metadata.version downgrade is generally not supported once it introduces new metadata records. Treat finalization as the commit point of the whole upgrade and run a soak period of normal traffic before you reach it. To upgrade brokers to 4.x, the cluster must already be on KRaft with a metadata version of at least 3.3 (the first GA KRaft release). Always read the official upgrade guide for your exact source and target versions.

Client Compatibility Validation

Kafka’s broker–client protocol is backward compatible across a wide range of versions, but some features (transactions/exactly-once, newer consumer-group/rebalance protocols, queues) require minimum client versions. Audit your client fleet before upgrading. You can inspect what each broker supports with:

kafka-broker-api-versions --bootstrap-server localhost:9092 | head -20

Cluster Health Baseline

Before touching any broker, establish a baseline. Check for under-replicated and unavailable partitions and confirm the controller quorum is stable. A cluster with pre-existing issues will not tolerate a rolling restart gracefully — if you start a restart with 5 partitions already under-replicated, you cannot distinguish “recovered” from “still broken” when deciding whether to move on.

# Under-replicated and leaderless partitions (both should return nothing)
kafka-topics --bootstrap-server localhost:9092 --describe --under-replicated-partitions
kafka-topics --bootstrap-server localhost:9092 --describe --unavailable-partitions

# Controller quorum status (KRaft)
kafka-metadata-quorum --bootstrap-server localhost:9092 describe --status

Resolve any under-replicated partitions before proceeding, and confirm the quorum has a single stable leader with all voters present. In the describe --status output, check that the lag of every follower voter is small and non-growing; a voter with persistent log-end-offset lag is a quorum member that will not be ready to take over if you restart the current leader.

Security Configuration Audit

If the cluster uses authentication and authorization, verify that every broker shares consistent security settings. A mismatched truststore, an expired certificate, or a missing SASL mechanism can prevent a restarted broker from rejoining. This matters most in environments running mTLS and SASL/SCRAM. Confirm certificate expiry extends well past your maintenance window:

# Verify broker certificate validity dates
openssl x509 -in /etc/kafka/secrets/broker.crt -noout -dates

The Rolling Restart Procedure

The principle is simple: restart one broker at a time, wait for full recovery, then move on. In practice it requires careful sequencing and continuous monitoring.

Step-by-Step Sequence

1. Restart broker nodes before controller nodes. Restarting an active controller forces a quorum re-election, which is safe but adds churn. In a deployment with dedicated controllers, do the broker nodes first and the controllers last; in combined-mode nodes, leave the current quorum leader for last.

2. Let controlled shutdown drain leadership. You do not manually “drain” a broker before restart. With controlled.shutdown.enable=true (the default), the broker migrates leadership off itself before exiting, minimizing leader elections and client impact. Identify the quorum leader so you can sequence around it:

# Current controller quorum leader (KRaft); the field is LeaderId
kafka-metadata-quorum --bootstrap-server localhost:9092 describe --status | grep "LeaderId"

3. Stop the broker cleanly. Send SIGTERM (what systemctl stop does) and wait for a clean exit. Kafka’s shutdown hook flushes logs and runs controlled shutdown so the controller learns of the departure in an orderly way.

# Graceful shutdown
sudo systemctl stop kafka
# Manual process management equivalent:
kill -TERM "$(cat /var/run/kafka/kafka.pid)"

4. Perform the maintenance task — replace binaries for an upgrade, change configuration, or service the hardware.

5. Restart and wait for full recovery. Start the broker and wait until it has caught up on every replica it hosts. The under-replicated count should briefly rise as the broker rejoins, then return to zero.

sudo systemctl start kafka
# Wait until the under-replicated partition count returns to zero
watch -n 5 'kafka-topics --bootstrap-server localhost:9092 \
  --describe --under-replicated-partitions | grep -c "Topic:"'

6. Validate broker health — confirm it has rejoined, is reporting metrics, and is serving requests.

7. Repeat for each broker, allowing enough recovery time between nodes. The right interval is “until URP is back to zero and ISRs are stable,” which scales with partition count and data volume. Use this to set expectations rather than a fixed timer:

The failure mode of a fixed timer (e.g. “sleep 120 then next”) is that on a large broker the replicas are still catching up when you stop the next one — gate on the metric, not the clock.

Controlled Shutdown vs. Hard Kill

Controlled shutdown, triggered by SIGTERM, migrates leadership off the departing broker before the process exits. This is far preferable to a hard kill, which forces the controller to detect the failure and elect leaders reactively. Controlled shutdown is not unconditional: if it can’t move leadership cleanly it retries up to controlled.shutdown.max.retries (default 3) with a backoff between attempts, and if all retries fail the broker logs the failure and shuts down anyway. The practical implication is that a broker shutdown taking unusually long usually means controlled shutdown is fighting an unhealthy cluster — check for existing URP before you assume the stop command hung. Always use SIGTERM and let shutdown finish. If a broker genuinely hangs, SIGKILL may be unavoidable, but expect brief unavailability for any partition where that broker was the only in-sync replica.

Handling Controller Nodes

When you restart the active controller (or a combined-mode node that is the quorum leader), expect a quorum re-election. The new leader serves metadata from the replicated __cluster_metadata log, so there is no rebuild from an external store. Confirm a single stable leader before continuing:

kafka-metadata-quorum --bootstrap-server localhost:9092 describe --status

Finalizing the Version After the Software Upgrade

During the rolling phase the cluster temporarily runs mixed software versions, which Kafka supports. Once every node is on the new release, finalize the metadata feature version. This is a single dynamic command — there is no second rolling restart.

# After all nodes are upgraded, finalize the cluster's feature versions
kafka-features --bootstrap-server localhost:9092 upgrade --release-version 4.1.0

Verify the result:

kafka-features --bootstrap-server localhost:9092 describe

This replaces the old ZooKeeper-era “bump IBP, restart again, then bump message format” workflow entirely. The Performing a Rolling Upgrade from Kafka 2.8 to 3.5 guide walks through a specific multi-step jump, including the ZooKeeper-to-KRaft considerations that apply when your starting point predates KRaft.

Monitoring and Observability During Maintenance

Effective monitoring is the difference between a confident upgrade and a nerve-wracking one. During a rolling restart, track both cluster-level health and per-broker recovery, and know in advance which signals are expected to move so you don’t chase normal transients.

Key Metrics to Watch

The single most useful pairing is IsrShrinksPerSec against IsrExpandsPerSec: a shrink with no matching expand is the signature of a replica that fell out of the ISR and never came back — exactly the URP-won’t-clear case below — and it’s visible before the topic-level URP count tells you why.

Prometheus and Grafana

Exact Prometheus metric names depend on your JMX exporter’s mapping rules, so confirm them against your exporter config rather than assuming. With the common JMX-Prometheus exporter defaults, the gauge MBeans above surface as lowercased names such as:

# Under-replicated partitions (should be 0 in steady state)
kafka_server_replicamanager_underreplicatedpartitions

# Active controller count across the cluster (must equal 1)
sum(kafka_controller_kafkacontroller_activecontrollercount)

# Offline partitions (must be 0)
kafka_controller_kafkacontroller_offlinepartitionscount

# Partition count per broker (verify even distribution after restart)
kafka_server_replicamanager_partitioncount

The IsrShrinksPerSec/IsrExpandsPerSec MBeans are rate meters; depending on your exporter rules they appear with a _count or _oneminuterate suffix (e.g. kafka_server_replicamanager_isrshrinkspersec_count) rather than a _total counter — check your exporter output before writing alerts. For a full setup, see Monitoring Kafka with JMX, Prometheus, and Grafana.

Log Inspection During Restart

Broker logs show the restart sequence in detail:

# Watch startup and replica role transitions
tail -f /var/log/kafka/server.log | grep -E "Loading|Loaded|became leader|became follower|Recovered"

A healthy restart shows log loading completing, a series of follower transitions as the broker catches up, and eventually leader transitions for partitions where it is the preferred leader. If you instead see a long stretch of “Loading” without “Loaded” on a large broker, the broker is replaying logs after an unclean shutdown — let it finish rather than restarting it again, which only restarts the recovery.

Common Pitfalls and Recovery Strategies

Leadership Churn

When a broker rejoins, the automatic leader-rebalancer (auto.leader.rebalance.enable, on by default) moves leadership for partitions back to their preferred (first-listed) replica. A background thread checks the imbalance every leader.imbalance.check.interval.seconds (default 300) and acts once a broker’s imbalance exceeds leader.imbalance.per.broker.percentage (default 10). On a large cluster this can move many leaderships at once, briefly disrupting clients each time. You can disable it during maintenance and re-enable it afterward; it is a cluster-wide dynamic config, so no restart is needed:

# Disable automatic leader rebalancing for the maintenance window
kafka-configs --bootstrap-server localhost:9092 \
  --entity-type brokers --entity-default \
  --alter --add-config auto.leader.rebalance.enable=false

Disabling it trades smooth restarts for a temporarily lopsided leadership distribution — the just-restarted broker carries few or no leaders until you rebalance — so re-enable it (or run a one-shot election) before you call the upgrade done. When you are ready, restore balanced leadership with a one-shot preferred election. Note that --all-topic-partitions, --topic, and --path-to-json-file are mutually exclusive — pick one. There is no flag to exclude a broker; to target specific partitions, use a JSON file:

# Rebalance preferred leaders across all partitions
kafka-leader-election --bootstrap-server localhost:9092 \
  --election-type preferred --all-topic-partitions

Under-Replicated Partitions That Won’t Clear

If URP persists after a restart (you see IsrShrinksPerSec without a matching expand), work the causes in order of likelihood:

1. Disk exhaustion — a full log directory blocks replica catch-up. This is the most common cause and the fastest to confirm.
2. Network reachability — the broker cannot reach a peer’s listener (often a firewall or DNS change that landed with the maintenance).
3. Corrupt segments — a damaged segment can stall replication for the affected partitions; look for CorruptRecordException or recovery loops in the log.

# Disk usage on the log directory
df -h /var/lib/kafka/data

# Connectivity to a peer broker's listener
nc -zv broker-2.internal 9092

# Replication-related errors in the log
grep -iE "error|corrupt|replica" /var/log/kafka/server.log | tail -50

If the partition is stuck because its only surviving in-sync replica is the broker you’re working on, you are in the unclean-leader-election tradeoff: waiting preserves data but keeps the partition offline; forcing election restores availability at the risk of losing un-replicated writes. Decide that policy deliberately per topic rather than discovering it mid-incident.

Client Authentication Failures After Restart

If a restarted broker fails to authenticate with clients or peers, the cause is usually a security misconfiguration — a wrong keystore path or password, or an expired certificate — that diverged between the old and new config. The tell is that the broker starts but other brokers/clients log handshake or SASL-auth errors against it specifically. Validate the listener with an explicit handshake (9093 here being an SSL/mTLS listener):

openssl s_client -connect broker-0.internal:9093 \
  -cert /etc/kafka/secrets/broker.crt \
  -key /etc/kafka/secrets/broker.key \
  -CAfile /etc/kafka/secrets/ca.crt

For the authorization model and validating ACLs after configuration changes, see Kafka ACLs and Authorization Best Practices.

Post-Upgrade Validation and Cleanup

Once every node runs the new version and the cluster is stable, validate before declaring the upgrade complete.

Functional Validation

Confirm producers can write and consumers can read on critical topics:

echo "post-upgrade-test-$(date +%s)" | kafka-console-producer \
  --bootstrap-server localhost:9092 --topic health-check
kafka-console-consumer --bootstrap-server localhost:9092 \
  --topic health-check --from-beginning --timeout-ms 5000

Performance Validation

Compare pre- and post-upgrade produce/consume latency, throughput, and ISR stability against the baseline you captured before starting. A brief regression during the rolling phase is normal; sustained degradation after the cluster is stable points to a configuration or version-specific issue worth resolving before you finalize the metadata version, since finalizing removes your rollback path.

Finalize Features and Clean Up

Finalize the metadata version if you have not already (see above), then remove temporary overrides. Re-enable automatic leader rebalancing if you disabled it:

# Re-enable automatic leader rebalancing
kafka-configs --bootstrap-server localhost:9092 \
  --entity-type brokers --entity-default \
  --alter --add-config auto.leader.rebalance.enable=true

Record the new software version, finalized metadata.version, and any client-compatibility notes in your runbooks.

The procedure reduces to five gates: a verified health baseline, controlled-shutdown restarts one node at a time, a per-broker wait that ends on a metric rather than a timer, a soak before the one dynamic feature finalization that commits the upgrade, and a leadership rebalance to finish flat. Hold those gates and the same playbook covers patch upgrades, major migrations, and routine configuration or hardware maintenance — with clients seeing nothing worse than a metadata refresh.