Automated Cluster Rebalancing with Cruise Control

Managing partition distribution in a growing Apache Kafka cluster is a relentless operational challenge. Without automation, engineers resort to manual partition reassignment, a process that is slow, error-prone, and disruptive to production traffic. Cruise Control, an open-source project originally developed at LinkedIn, addresses this by continuously monitoring the cluster and executing rebalancing operations against a configurable, prioritized list of goals. This page is an operational guide for platform, SRE, and backend engineers who need to deploy, configure, and integrate Cruise Control to achieve self-healing cluster management.

Understanding the Rebalancing Imperative

Kafka’s fundamental unit of parallelism and scalability is the partition. When a new broker joins a cluster, it arrives empty: it contributes storage and compute capacity but handles no traffic until partitions are explicitly moved onto it. Similarly, when a broker is decommissioned, its partitions must be migrated away before it can be shut down. Even in a stable cluster, organic growth in topic throughput or skewed producer key distribution creates hot spots—brokers with disproportionately high CPU, network, or disk utilization.

Manual intervention via the kafka-reassign-partitions.sh tool requires an operator to generate a JSON reassignment plan, verify its correctness, and execute it in throttled batches. This workflow breaks down at scale. A cluster with thousands of partitions needs an algorithmic approach that balances load across multiple resource dimensions at once. Cruise Control models the cluster state, frames optimization as a set of hard and soft goals, and generates an execution proposal that can be applied automatically or after human approval. This is a cornerstone of a mature Operations, Security & Observability practice, where automation replaces toil with predictable, repeatable processes.

The operational burden extends beyond moving partitions. For clusters secured with authentication and authorization, the Cruise Control service itself must be integrated into your security fabric—a topic explored alongside Securing Kafka with mTLS and SASL/SCRAM. Without automation, every broker addition or removal becomes a project requiring change windows, manual verification, and rollback planning.

Cruise Control Architecture and Deployment

Cruise Control is a standalone Java application that runs alongside your Kafka brokers, typically on dedicated hardware or co-located on broker nodes with sufficient headroom. It comprises four core components: the Load Monitor, the Analyzer, the Anomaly Detector, and the Executor.

The Load Monitor samples metrics from all brokers at a configurable interval, building a time-series model of resource utilization per broker and per partition. These metrics include CPU usage, network bytes in and out, disk utilization, and leader/replica counts. The Analyzer consumes this model and, when triggered, generates an optimization proposal that moves partitions and leadership to satisfy a prioritized list of goals. The Anomaly Detector continuously evaluates the cluster against predefined anomaly conditions—broker failures, metric anomalies, disk failures, and goal violations—and can trigger automatic remediation. The Executor carries out approved proposals, throttling reassignments to respect configured bandwidth and concurrency limits.

flowchart TD Brokers["Kafka brokers"] --> LM["Load Monitor (samples metrics)"] LM --> Analyzer["Analyzer (generates proposal vs goals)"] LM --> AD["Anomaly Detector (broker/disk/metric/goal anomalies)"] AD -- "self-healing triggers" --> Analyzer Analyzer --> Executor["Executor (throttled reassignment)"] Executor --> Brokers

Deployment begins with downloading a release from the Cruise Control GitHub repository. The application reads a properties file specifying the Kafka bootstrap servers, the ZooKeeper or KRaft connection details required for the running Kafka version, and metric sampling parameters. A minimal configuration looks like this:

# cruisecontrol.properties
bootstrap.servers=kafka-broker-1:9092,kafka-broker-2:9092,kafka-broker-3:9092
zookeeper.connect=zk-1:2181,zk-2:2181,zk-3:2181

# Metric sampling
metric.sampler.class=com.linkedin.kafka.cruisecontrol.monitor.sampling.CruiseControlMetricsReporterSampler
metric.sampling.interval.ms=60000
partition.metrics.window.ms=300000
num.partition.metrics.windows=5

# Anomaly detection
anomaly.detection.interval.ms=120000

# Executor concurrency and throttling
num.concurrent.partition.movements.per.broker=5
execution.progress.check.interval.ms=10000
default.replication.throttle=52428800

The Load Monitor’s data retention is governed by partition.metrics.window.ms (the aggregation window size) and num.partition.metrics.windows (how many windows to keep), not by a single retention property. default.replication.throttle caps reassignment bandwidth in bytes per second per broker—52428800 is 50 MB/s.

For production, you must also enable the Cruise Control metrics reporter on every Kafka broker so the CruiseControlMetricsReporterSampler has data to consume. Add the following to each broker’s server.properties:

# Broker-side metrics reporter
metric.reporters=com.linkedin.kafka.cruisecontrol.metricsreporter.CruiseControlMetricsReporter
cruise.control.metrics.reporter.bootstrap.servers=kafka-broker-1:9092,kafka-broker-2:9092,kafka-broker-3:9092

The reporter jar (cruise-control-metrics-reporter) must be on each broker’s classpath. After starting Cruise Control, verify it has discovered the cluster by querying its REST API:

curl -X GET "http://cruise-control:9090/kafkacruisecontrol/state"

The response reports the MonitorState, ExecutorState, AnalyzerState, and AnomalyDetectorState substates, including how many valid metric windows the Load Monitor has accumulated. The Analyzer cannot generate trustworthy proposals until enough windows are populated, so expect a warm-up period after first start. This API is the primary interface for triggering rebalances, checking status, and managing the system.

Configuring Goals for Optimal Balance

Cruise Control’s rebalancing logic is driven by an ordered list of goals. Goals are either hard or soft: hard goals must be satisfied for a proposal to be valid; soft goals are optimized in priority order, with higher-priority goals taking precedence. The default goal list works for many deployments, but production tuning requires understanding each goal’s effect.

Common hard goals include RackAwareGoal, which keeps replicas of a partition on brokers in different racks (failure domains), and capacity goals such as ReplicaCapacityGoal, DiskCapacityGoal, NetworkInboundCapacityGoal, NetworkOutboundCapacityGoal, and CpuCapacityGoal, which prevent brokers from exceeding configured limits. Soft distribution goals such as DiskUsageDistributionGoal, NetworkInboundUsageDistributionGoal, and CpuUsageDistributionGoal even out resource utilization. Leadership balance is handled by LeaderReplicaDistributionGoal and LeaderBytesInDistributionGoal.

Override the active goals via the goals property in cruisecontrol.properties. Each goal is a fully qualified class name under com.linkedin.kafka.cruisecontrol.analyzer.goals:

goals=com.linkedin.kafka.cruisecontrol.analyzer.goals.RackAwareGoal,com.linkedin.kafka.cruisecontrol.analyzer.goals.ReplicaCapacityGoal,com.linkedin.kafka.cruisecontrol.analyzer.goals.DiskCapacityGoal,com.linkedin.kafka.cruisecontrol.analyzer.goals.NetworkInboundCapacityGoal,com.linkedin.kafka.cruisecontrol.analyzer.goals.DiskUsageDistributionGoal,com.linkedin.kafka.cruisecontrol.analyzer.goals.LeaderBytesInDistributionGoal

To preview a configuration without touching the cluster, call the rebalance endpoint with dryrun=true (this is the default, so omitting it also runs a dry run). The goals parameter accepts the short class names:

curl -X POST "http://cruise-control:9090/kafkacruisecontrol/rebalance?dryrun=true&goals=RackAwareGoal,ReplicaCapacityGoal,NetworkInboundCapacityGoal,DiskUsageDistributionGoal"

The response summarizes inter-broker partition movements, data to move, and the number of replicas affected. Review it carefully before applying. Goal tuning is iterative: start with a conservative set, observe proposals in dry-run mode, and introduce additional soft goals as you gain confidence. Note that any goals you pass at request time must be a subset of the goals configured in goals (or default.goals).

Operational Workflows: Adding, Removing, and Fixing Brokers

The three most common triggers for rebalancing are broker addition, broker decommissioning, and remediation of failures or hot spots. Cruise Control provides dedicated REST endpoints for each. All modification endpoints default to dryrun=true, so you must pass dryrun=false explicitly to execute.

Adding a Broker

When a new broker joins the cluster it registers with the controller but owns no partitions. To move replicas onto it, call add_broker with the new broker id (the parameter accepts a comma-separated list):

curl -X POST "http://cruise-control:9090/kafkacruisecontrol/add_broker?brokerid=4&dryrun=false"

add_broker only moves replicas from existing brokers onto the new broker; it does not shuffle replicas among the existing brokers. The operation respects the configured concurrency (num.concurrent.partition.movements.per.broker) and the default.replication.throttle, so production traffic is protected. Monitor progress via the user_tasks endpoint, using the task id returned in the response header:

curl -X GET "http://cruise-control:9090/kafkacruisecontrol/user_tasks?user_task_ids=<task-uuid>"

Removing a Broker

Before decommissioning a broker you must migrate all partitions off it. The remove_broker endpoint does this while keeping the cluster balanced, moving replicas only onto the remaining brokers:

curl -X POST "http://cruise-control:9090/kafkacruisecontrol/remove_broker?brokerid=3&dryrun=false"

The operation can take hours on large clusters. Cruise Control will not violate replication or rack-awareness constraints, so ensure your topics have a replication factor that the remaining brokers can still satisfy. Once the task completes, confirm the broker holds no partitions with kafka-log-dirs.sh before shutting it down.

Fixing Offline Replicas and Hot Spots

Broker or disk failures can leave partitions under-replicated. The fix_offline_replicas endpoint moves offline replicas (those on dead brokers or dead disks) onto healthy brokers:

curl -X POST "http://cruise-control:9090/kafkacruisecontrol/fix_offline_replicas?dryrun=false"

For uneven load without a topology change, the generic rebalance endpoint redistributes partitions across the existing brokers:

curl -X POST "http://cruise-control:9090/kafkacruisecontrol/rebalance?dryrun=false&goals=RackAwareGoal,ReplicaCapacityGoal,NetworkInboundCapacityGoal,DiskUsageDistributionGoal"

If your cluster uses ACLs, the Cruise Control principal needs the rights to read metadata and drive reassignments—broadly, DESCRIBE and ALTER on the cluster and topics, plus access to its sample-store topics. For guidance on constructing these policies, see Kafka ACLs and Authorization Best Practices.

Anomaly Detection and Self-Healing

The Anomaly Detector elevates Cruise Control from a planning tool to an autonomous operator. It evaluates four anomaly types: broker failures, goal violations, metric anomalies, and disk failures. When an anomaly is detected and self-healing is enabled for that type, Cruise Control can trigger a remediation rebalance without human intervention.

Self-healing is off by default. Enable it in cruisecontrol.properties with the master switch plus the per-type toggles:

# Self-healing configuration
self.healing.enabled=true
broker.failure.self.healing.enabled=true
goal.violation.self.healing.enabled=true
metric.anomaly.self.healing.enabled=true
disk.failure.self.healing.enabled=true

# Allow anomaly-detection threads to use estimated capacity
# when exact broker capacity is unavailable
anomaly.detection.allow.capacity.estimation=true

broker.failure.self.healing.enabled is the most consequential. When a broker fails and does not recover within broker.failure.self.healing.threshold.ms, Cruise Control reassigns its replicas to healthy brokers, restoring full replication automatically. In large clusters this cuts mean time to recovery from hours to minutes.

Self-healing carries risk: a network partition that isolates Cruise Control from the cluster could trigger unnecessary movements. Mitigate this by raising broker.failure.detection.backoff.ms so a transient outage doesn’t immediately register as a permanent failure, and by deploying Cruise Control across multiple failure domains. The Apache Kafka documentation describes how the controller itself handles broker failures, which Cruise Control complements rather than replaces.

Integrating Cruise Control into Infrastructure as Code

Manual REST calls are fine for ad-hoc operations, but a mature platform wants programmatic integration. The REST API can be driven by configuration management tools, CI/CD pipelines, or Kubernetes operators (Strimzi, for example, manages Cruise Control declaratively).

For teams using Ansible, deploying Cruise Control, templating its configuration, and triggering rebalances can be codified into playbooks so every environment uses identical, version-controlled configuration. For a complete walkthrough with sample playbooks and Jinja2 templates, see Automating Cruise Control Rebalancing with Ansible.

You can also wire Cruise Control into your monitoring stack: a Prometheus alert on sustained broker imbalance can call the rebalance endpoint via a webhook. The following Python snippet shows a minimal integration with requests:

import json
import requests

CRUISE_CONTROL_URL = "http://cruise-control:9090/kafkacruisecontrol"

def trigger_rebalance(goals: list[str], dryrun: bool = True) -> dict:
    params = {
        "dryrun": str(dryrun).lower(),
        "goals": ",".join(goals),
        "json": "true",
    }
    response = requests.post(f"{CRUISE_CONTROL_URL}/rebalance", params=params)
    response.raise_for_status()
    return response.json()

# Example: a dry-run rebalance with a specific goal list
result = trigger_rebalance(
    goals=["RackAwareGoal", "ReplicaCapacityGoal", "DiskUsageDistributionGoal"],
    dryrun=True,
)
print(json.dumps(result, indent=2))

Passing json=true makes Cruise Control return a structured JSON body instead of the default plaintext, which is what response.json() expects. Long-running operations may return HTTP 202 with the result available later via user_tasks; production code should handle that and poll. Extend the script to inspect the proposal, gate on the volume of data movement, and approve the rebalance (dryrun=false) only when it falls within an acceptable threshold. That closes the loop between observability and remediation.

Monitoring and Tuning Cruise Control in Production

Cruise Control exposes its own sensors over JMX, which you should scrape into your observability stack. The MBeans use the form kafka.cruisecontrol:name=<Component>.<sensor>. Useful examples:

  • kafka.cruisecontrol:name=Executor.replica-action-in-progress — whether a reassignment is currently executing.
  • kafka.cruisecontrol:name=LoadMonitor.total-monitored-windows — number of valid metric windows the Load Monitor has accumulated (a low value means proposals are running on thin data).
  • kafka.cruisecontrol:name=AnomalyDetector.number-of-self-healing-started — how many self-healing actions have fired.
  • kafka.cruisecontrol:name=AnomalyDetector.balancedness-score — the Anomaly Detector’s overall cluster balance score.

A matching jmx_exporter rule for these MBeans:

# prometheus jmx_exporter config for Cruise Control
lowercaseOutputName: true
rules:
  - pattern: 'kafka.cruisecontrol<name=(.+)><>(\w+)'
    name: 'kafka_cruisecontrol_$1_$2'
    type: GAUGE

Tuning Cruise Control means adjusting sampling intervals, executor concurrency, and goal priorities based on observed behavior. partition.metrics.window.ms and num.partition.metrics.windows together control how much history the Load Monitor retains: more or larger windows smooth out transient spikes but slow the response to genuine load changes. Reassignment impact is bounded by two independent knobs—num.concurrent.partition.movements.per.broker (how many movements run at once) and default.replication.throttle (bytes/second per broker). Start with a conservative throttle such as 50 MB/s per broker and raise it gradually while watching client latency percentiles.

The Cruise Control wiki is the authoritative reference for configuration properties, REST endpoints, and sensors. Review the project’s release notes regularly, as goals and the anomaly-detection logic continue to evolve.

In this section

1 guide in this area.

Back to Operations Security Observability