How to Configure Cooperative Rebalancing for Zero-Downtime Deployments

When a Kafka consumer joins or leaves a group, the group coordinator triggers a rebalance. Under the legacy eager protocol, that rebalance is a stop-the-world event: every consumer revokes all of its partitions and waits for a fresh assignment before resuming. During a rolling restart of a large fleet, this means a brief but total processing pause on every rebalance — multiplied by however many instances cycle. Cooperative (incremental) rebalancing changes the math: consumers keep processing the partitions they will retain and revoke only the ones that actually move. This guide covers how to configure, validate, and troubleshoot cooperative rebalancing so a rolling deployment stops being a visible event in your latency graphs.

Eager vs. Cooperative: what actually changes

In the eager protocol, every membership change causes all members to receive an onPartitionsRevoked callback for their entire assignment, stop consuming, and rejoin. No partition in the group makes progress until the new assignment is distributed. For high-throughput or low-latency workloads, a few seconds of stall per rebalance accumulates quickly across a sequential rolling restart of dozens of instances.

The incremental cooperative rebalance protocol — KIP-429, shipped in Kafka 2.4.0 — keeps partitions in place where possible. When a rebalance starts, every member sends its current ownership to the coordinator, which computes a sticky target assignment that minimizes movement. The mechanism is two-phase:

1. First rebalance. The new assignment is computed. A member’s onPartitionsRevoked is invoked only for partitions that must move to a different consumer; if it has nothing to give up, the callback is not invoked at all. Members keep consuming everything they are retaining throughout.
2. Follow-up rebalance. Because some partitions were revoked but not yet handed out (a consumer cannot take a partition until its previous owner has released it), the protocol triggers a second rebalance that assigns those now-free partitions to their final owners via onPartitionsAssigned.

The net effect during a rolling restart: only the partitions belonging to the instance being cycled experience a brief gap, while every other instance keeps processing. The broader mechanics are covered in Consumer Group Rebalancing: Protocols and Tuning; here we stay focused on the deployment workflow.

The three options you actually choose between for a rolling deploy:

Cooperative is the protocol-level win; static membership (below) is what eliminates the rebalance entirely for a clean restart. They compose, and using both is the target end state.

Prerequisites and compatibility

The CooperativeStickyAssignor and the incremental protocol require client version 2.4.0 or later. The broker imposes no special requirement beyond what the consumer’s classic group protocol already needs — the cooperative behavior is negotiated entirely between clients through the assignor, so a 2.4+ consumer can use it against older brokers. Even so, run on a recent release: several stickiness and follow-up-rebalance bugs were fixed across the 2.5/2.6 lines, so 2.6.0+ is a sensible floor.

A few application-level considerations:

• Kafka Streams uses its own StreamsPartitionAssignor and has run cooperative rebalancing by default since 2.4. You do not set partition.assignment.strategy for a Streams app — see the Streams section below.
• Mixed-version groups. If you list cooperative-sticky as the only strategy, members must all support it. The supported upgrade path is to run both assignors during the transition (covered below) so the group degrades safely to eager rather than failing to form.
• Rebalance-listener logic. Because partitions are no longer revoked en masse, any state cleanup or offset flush you do in onPartitionsRevoked must act only on the partitions actually passed to the callback. Cleaning up state for partitions you still own leads to duplicate processing or dropped state.

Per the Kafka consumer configuration reference, CooperativeStickyAssignor is the recommended strategy for plain consumers that want incremental rebalancing.

Configuring a plain Kafka consumer

Set partition.assignment.strategy to CooperativeStickyAssignor:

Properties props = new Properties();
props.setProperty("bootstrap.servers", "broker1:9092,broker2:9092");
props.setProperty("group.id", "my-consumer-group");
props.setProperty("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
props.setProperty("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
props.setProperty("partition.assignment.strategy",
    "org.apache.kafka.clients.consumer.CooperativeStickyAssignor");
props.setProperty("enable.auto.commit", "false");

KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);

Migrating an existing group safely

You cannot flip a running group from RangeAssignor straight to CooperativeStickyAssignor in a single bounce — the two use different rebalance protocols, and the group must never simultaneously contain members on the eager and cooperative protocols. The documented two-step rolling upgrade is:

// Step 1: deploy ALL members with both assignors. The eager (range)
// protocol stays in effect because range is still in the list and the
// group continues to negotiate the highest strategy common to all members.
props.setProperty("partition.assignment.strategy",
    "org.apache.kafka.clients.consumer.CooperativeStickyAssignor," +
    "org.apache.kafka.clients.consumer.RangeAssignor");

// Step 2: after every member is running step 1, do a second rolling bounce
// that removes RangeAssignor. Now the whole group is cooperative-sticky.
props.setProperty("partition.assignment.strategy",
    "org.apache.kafka.clients.consumer.CooperativeStickyAssignor");

The consumer always selects the most-preferred strategy supported by all current members. During step 1 the common strategy is still range (eager); only after step 2 removes range from every member does the group switch to the cooperative protocol. The danger is stopping halfway: leaving both assignors in place permanently keeps you silently on the eager protocol — the migration looks done but the deploy still stalls the whole group. Confirm the switch with the --state check below before you call it finished.

Kubernetes: keep the config identical

A frequent failure mode is canary and stable tracks running different assignor lists, which can stall protocol negotiation. Pin the consumer config in a single ConfigMap shared by every pod in the group:

apiVersion: v1
kind: ConfigMap
metadata:
  name: kafka-consumer-config
data:
  consumer.properties: |
    bootstrap.servers=broker1:9092,broker2:9092
    group.id=my-consumer-group
    partition.assignment.strategy=org.apache.kafka.clients.consumer.CooperativeStickyAssignor
    enable.auto.commit=false
    max.poll.interval.ms=300000
    session.timeout.ms=45000
    heartbeat.interval.ms=3000

Kafka Streams: cooperative is already on

For a Kafka Streams application you do not set partition.assignment.strategy — Streams installs its own StreamsPartitionAssignor, which has used the cooperative protocol by default since 2.4. Manually overriding the assignor will break Streams. The only assignment-related knob you touch is upgrade.from, and only when performing a rolling upgrade across a version that changed the protocol/metadata format. For example, when upgrading a live app from 2.3 (which predates cooperative Streams) you bounce once with upgrade.from set to the old version so the app speaks the old protocol during the mixed phase, then bounce again with it removed:

Properties streamsProps = new Properties();
streamsProps.setProperty("bootstrap.servers", "broker1:9092,broker2:9092");
streamsProps.setProperty("application.id", "my-streams-app");
// Set ONLY during an upgrade FROM 2.3 or earlier; remove on the second bounce.
streamsProps.setProperty("upgrade.from", "2.3");

On a steady-state 2.4+ Streams app, neither property is needed.

Tuning session, heartbeat, and poll timeouts

These three settings govern failure detection and therefore how often rebalances fire. Because cooperative rebalancing makes each rebalance cheap (only moving partitions pause), you can detect dead members reasonably quickly without paying the eager-protocol penalty — but the trade-off between fast detection and rebalance churn from transient blips still applies.

• session.timeout.ms — how long the coordinator waits without a heartbeat before declaring a member dead. It must fall within the broker’s group.min.session.timeout.ms/group.max.session.timeout.ms bounds (defaults 6 s and 30 min). The client default is 45 s.
• heartbeat.interval.ms — how often the consumer heartbeats. Keep it around one third of the session timeout so several heartbeats can be missed before expiry. Client default is 3 s.
• max.poll.interval.ms — the maximum gap between poll() calls before the consumer is considered stuck and proactively leaves the group (triggering a rebalance). This is independent of heartbeats, which flow from a background thread. Client default is 5 min (300000 ms).

A reasonable production starting point:

session.timeout.ms=45000
heartbeat.interval.ms=15000
max.poll.interval.ms=300000

Note that max.poll.interval.ms is bounded by your slowest poll-loop iteration, not by the rebalance protocol. If a single poll()-to-poll() cycle (record processing included) can exceed this value, raise it or reduce max.poll.records; otherwise the consumer self-evicts mid-processing. Observe real behavior through the JMX bean kafka.consumer:type=consumer-coordinator-metrics,client-id=* (attributes such as heartbeat-rate, last-heartbeat-seconds-ago, and the rebalance metrics below) and tune from measurements rather than guesses.

Rebalance listeners for partial revocation

The main code change when adopting cooperative rebalancing is making the listener handle a subset of partitions. Under the cooperative protocol onPartitionsRevoked receives only the partitions leaving this consumer; the rest keep flowing. onPartitionsLost (added by KIP-429) fires when ownership is lost uncleanly — e.g. the member fenced itself by blowing max.poll.interval.ms or hit an invalid generation — in which case you must not commit, because another consumer may already own those partitions.

import org.apache.kafka.clients.consumer.ConsumerRebalanceListener;
import org.apache.kafka.common.TopicPartition;
import java.util.Collection;
import java.util.Map;
import java.util.concurrent.ConcurrentHashMap;

public class CooperativeRebalanceListener implements ConsumerRebalanceListener {
    private final Map<TopicPartition, Long> partitionOffsets = new ConcurrentHashMap<>();

    @Override
    public void onPartitionsRevoked(Collection<TopicPartition> partitions) {
        // Cooperative: ONLY the partitions moving away are passed here.
        // Clean up / commit state for exactly these; retained partitions
        // (not in this collection) keep processing uninterrupted.
        for (TopicPartition partition : partitions) {
            Long offset = partitionOffsets.remove(partition);
            if (offset != null) {
                flushPartitionState(partition, offset);  // safe: we still own it here
            }
        }
    }

    @Override
    public void onPartitionsAssigned(Collection<TopicPartition> partitions) {
        // Always invoked at the end of a rebalance, possibly with an empty set.
        // Initialize state for newly acquired partitions only.
        for (TopicPartition partition : partitions) {
            // e.g. restore per-partition state, seek if needed
        }
    }

    @Override
    public void onPartitionsLost(Collection<TopicPartition> partitions) {
        // Unclean loss (fenced / invalid generation): another consumer may
        // already own these. Do NOT commit; discard in-memory state.
        for (TopicPartition partition : partitions) {
            partitionOffsets.remove(partition);
            discardPartitionState(partition);
        }
    }

    private void flushPartitionState(TopicPartition partition, Long offset) { /* ... */ }
    private void discardPartitionState(TopicPartition partition) { /* ... */ }
}

Two operational notes. First, because a cooperative rebalance is two-phase, onPartitionsRevoked and onPartitionsAssigned can each be invoked across the consecutive rebalances of a single membership change — keep the handlers idempotent per partition. Second, the protocol does not coordinate a direct state handoff between the old and new owner; if a partition carries large local state, the new owner must rebuild it from the changelog/source rather than relying on the listener.

Validating in staging

Stand up a group of at least three instances on cooperative-sticky and exercise three scenarios: a rolling restart, a scale-out, and an unclean kill. For each, confirm that retained partitions keep processing and that no records are lost or double-processed.

First confirm the negotiated strategy. The assignment strategy is a group-level property, so use --state, not --members:

kafka-consumer-groups.sh \
  --bootstrap-server broker1:9092 \
  --group my-consumer-group \
  --describe --state

The ASSIGNMENT-STRATEGY column should read cooperative-sticky. If it shows range (or another eager strategy), negotiation fell back and at least one member is misconfigured.

To see who owns what, use --members --verbose, which adds an ASSIGNMENT column listing each member’s partitions:

kafka-consumer-groups.sh \
  --bootstrap-server broker1:9092 \
  --group my-consumer-group \
  --describe --members --verbose

Capture this before, during, and after a rolling restart. Under healthy cooperative behavior the surviving members’ assignments are unchanged except for absorbing the cycled instance’s partitions — no churn among partitions that did not need to move. Any movement among survivors points to a stickiness problem or a badly skewed starting distribution.

For finer evidence, log every assignment/revocation event with timestamps. A healthy cooperative rebalance on a surviving consumer looks like a small revoke set, uninterrupted processing of the rest, and a follow-up assignment — not a full revoke-and-reacquire of the whole assignment. Concretely, on the unclean kill test the survivors should see one moving partition each (the dead member’s), onPartitionsRevoked called with empty or small sets, and rebalance-latency-max measured in the low hundreds of milliseconds — not seconds.

Troubleshooting

Protocol negotiation fell back to eager. If the coordinator cannot find a strategy supported by all members, the group uses the highest common one — typically range with the eager protocol — silently. Check the ASSIGNMENT-STRATEGY column via --describe --state. In Kubernetes this usually means a stale image or a divergent ConfigMap on some pods; standardize and redeploy.

Too many rebalances. Cooperative rebalancing lowers the cost of a rebalance but not its triggers: slow poll loops exceeding max.poll.interval.ms, network blips tripping session.timeout.ms, and membership churn all still cause them — and the lower cost can mask an instability you’d otherwise notice. Watch rebalance-rate-per-hour and failed-rebalance-rate-per-hour on kafka.consumer:type=consumer-coordinator-metrics,client-id=*. A steady group should rebalance only on deliberate scaling or deploys, so last-rebalance-seconds-ago should keep climbing between deploys; if it repeatedly resets to near zero with no deploy in flight, you have churn. To distinguish the cause: a max.poll.interval.ms violation shows up as a member that self-evicts and rejoins (an onPartitionsLost on that member), whereas a session.timeout.ms blip shows up as a member dropping and returning without ever losing the poll loop. Trim per-iteration work or lower max.poll.records for the former; raise session.timeout.ms or fix the network for the latter.

Assignment imbalance after scale-out. CooperativeStickyAssignor prefers minimal movement, so a newly added consumer can start under-loaded because existing owners keep their partitions. The assignor rebalances toward even distribution over subsequent generations, so transient skew right after adding a member is expected. Resist “fixing” it by bouncing the whole group at once — that reintroduces the stop-the-world pause you adopted cooperative rebalancing to avoid. If skew persists across several generations, the cause is structural (uneven keying or too few partitions for the member count), not the assignor; a 6-partition topic can never balance evenly across 4 members.

To see the protocol in detail, raise the coordinator logger to DEBUG:

log4j.logger.org.apache.kafka.clients.consumer.internals.ConsumerCoordinator=DEBUG

This logs JoinGroup/SyncGroup exchanges, including the selected assignor and the per-member assignment, so you can confirm cooperative-sticky is in use and that only the expected partitions move.

Operational best practices

Pair with static membership. Configure a stable, unique group.instance.id per instance (KIP-345, Kafka 2.3+). A static member does not send a LeaveGroup request when it shuts down, so a restart that completes within session.timeout.ms causes no rebalance at all — the coordinator returns the member’s cached assignment when it rejoins. This is the biggest single win for rolling restarts; cooperative rebalancing then handles the cases where a rebalance is genuinely required. Because the group now relies on session.timeout.ms to detect a truly-gone member, size it to cover your worst-case restart (pod reschedule, image pull):

props.setProperty("group.instance.id", "consumer-instance-1"); // e.g. StatefulSet ordinal
props.setProperty("session.timeout.ms", "45000");

Note the trade-off: with static membership a crashed (not restarted) instance leaves its partitions unprocessed until session.timeout.ms elapses, so don’t set it arbitrarily high. The practical floor is your p99 restart time plus margin; the ceiling is the longest processing gap you can tolerate on a hard crash.

Shut down cleanly — and note the static-member caveat. Calling consumer.close() lets a dynamic member send LeaveGroup so the group rebalances immediately instead of waiting out the session timeout. A static member deliberately does not leave on close, which is the point. Use wakeup() to break the poll loop, then close() with a timeout:

Runtime.getRuntime().addShutdownHook(new Thread(() -> {
    consumer.wakeup(); // unblocks poll() with WakeupException
}));
// In the poll loop's catch/finally:
//   } catch (WakeupException e) { /* expected on shutdown */ }
//   } finally { consumer.close(Duration.ofSeconds(30)); }

close() commits offsets (if auto-commit is on), runs onPartitionsRevoked, and leaves the group for dynamic members. Call it from the application thread that owns the consumer — KafkaConsumer is not safe for concurrent use, which is why the hook only calls wakeup().

Gate deploys on a rebalance test. In CI/CD, deploy the new consumer to staging, trigger a rolling restart, and assert that the ASSIGNMENT-STRATEGY stays cooperative-sticky and that no per-partition processing gap exceeds your threshold. Fail the release if it regresses to eager.

Monitor the right metrics. Export the consumer-coordinator JMX metrics to Prometheus and alert on: a group whose ASSIGNMENT-STRATEGY is no longer cooperative-sticky (scrape it from kafka-consumer-groups.sh --describe --state, since there is no JMX attribute for it); a rising rebalance-rate-per-hour or failed-rebalance-rate-per-hour; and growing rebalance-latency-avg/rebalance-latency-max. The Kafka monitoring documentation is the authoritative list of available metrics.

Cooperative rebalancing, static membership, clean shutdown, and a deploy gate are not alternatives — they stack. Static membership removes the rebalance for a clean restart, the cooperative protocol makes the unavoidable ones cheap, clean shutdown keeps dynamic members from waiting out the timeout, and the deploy gate stops a silent regression to eager from reaching production. Together they turn a rolling restart into a non-event for message processing.

Related Pages

• Producers, Consumers & Delivery Semantics