Automating Cruise Control Rebalancing with Ansible

Partition balance in a large Apache Kafka cluster drifts constantly: topics are created, partitions grow, brokers are added or decommissioned, and traffic patterns shift away from the original distribution. Left unchecked, that drift produces hot brokers, uneven network utilization, and unpredictable tail latency. LinkedIn’s Cruise Control counters it by continuously sampling cluster metrics and proposing—or, with self-healing enabled, automatically executing—rebalancing operations. But Cruise Control is itself a service that must be built, deployed, configured, and operated. For teams already running infrastructure through Ansible, wiring its lifecycle and rebalancing workflows into existing playbooks removes manual toil, enforces consistency across environments, and version-controls the operational knowledge that keeps a cluster healthy.

This guide assumes a foundational understanding of Cruise Control’s architecture and goals. If you are new to the tool, start with the broader overview in Automated Cluster Rebalancing with Cruise Control to understand the metrics, goals, and anomaly detectors that drive the rebalancing logic. Here the focus is the practical automation of Cruise Control’s lifecycle and rebalancing workflows, for platform and SRE engineers who need repeatable, auditable operations. These patterns sit within the wider domain of Operations, Security & Observability, where automation is a force multiplier for reliability.

Everything below is grounded in the Cruise Control REST API and configuration wikis. Two details bite people constantly, so settle them up front: every state-changing endpoint defaults to dryrun=true, and request parameters are query parameters in snake_case (exclude_recently_demoted_brokers, allow_capacity_estimation)—not a JSON body.

Choosing an Operating Mode

Before writing any role, decide how rebalancing should be triggered. The three modes below are not mutually exclusive—most mature setups run self-healing for broker failures while gating goal-violation fixes behind human approval—but each has a distinct blast radius and audit story.

The rest of this guide builds all three, plus the provisioning, validation, and security scaffolding they share.

Provisioning Cruise Control with Ansible

Cruise Control is distributed as source and built with Gradle; there is no ready-to-run binary tarball to unpack and start. A GitHub release tag like 2.5.146 gives you a source tree that you compile with ./gradlew jar copyDependantLibs, producing the runnable jars plus their dependencies under cruise-control/build/. Recent 2.5.x releases require Java 17. An Ansible role therefore has to either build on the target (or a build host) or, more commonly, build once in CI and ship the resulting artifact. Place Cruise Control on dedicated utility hosts rather than co-locating it with brokers: metric sampling and proposal computation are CPU- and heap-hungry, and you do not want them contending with broker request handling.

Start with a role structure that separates defaults, templates, and tasks. The defaults file captures the version pin, paths, and JVM heap—consistent across environments, overridable per group:

# roles/cruise_control/defaults/main.yml
cruise_control_version: "2.5.146"        # a real release tag; confirm against the releases page
cruise_control_user: "cruise-control"
cruise_control_group: "cruise-control"
cruise_control_install_dir: "/opt/cruise-control"
cruise_control_data_dir: "/var/lib/cruise-control"
cruise_control_port: 9090
cruise_control_heap_opts: "-Xms4G -Xmx4G"
cruise_control_bootstrap_servers: >-
  {{ groups['kafka_brokers']
     | map('extract', hostvars, ['ansible_host'])
     | map('regex_replace', '$', ':9092')
     | join(',') }}

Because the source tree must be compiled, the cleanest pattern is to build a self-contained artifact in CI (the source plus cruise-control/build/) and publish it to an internal artifact store. The role then fetches that artifact, unpacks it, and lays down a systemd unit. If you prefer to build on a host, the role must also install a JDK 17 and run ./gradlew jar copyDependantLibs before the service can start—at which point the role owns a compiler toolchain on production hosts, which most teams would rather avoid.

# roles/cruise_control/tasks/main.yml
- name: Fetch pre-built Cruise Control artifact from internal store
  ansible.builtin.unarchive:
    src: "{{ cruise_control_artifact_url }}/cruise-control-{{ cruise_control_version }}.tar.gz"
    dest: "{{ cruise_control_install_dir }}"
    remote_src: true
    creates: "{{ cruise_control_install_dir }}/kafka-cruise-control-start.sh"
    owner: "{{ cruise_control_user }}"
    group: "{{ cruise_control_group }}"

- name: Deploy cruisecontrol.properties from template
  ansible.builtin.template:
    src: cruisecontrol.properties.j2
    dest: "{{ cruise_control_install_dir }}/config/cruisecontrol.properties"
    owner: "{{ cruise_control_user }}"
    group: "{{ cruise_control_group }}"
    mode: '0640'
  notify: restart cruise control

The service launches with the bundled start script, kafka-cruise-control-start.sh config/cruisecontrol.properties [port]; your systemd unit calls it with the templated config path. The handler restarts the service only when configuration changes, so a no-op playbook run leaves a busy rebalance executor untouched. By templating cruisecontrol.properties you inject environment-specific values—chiefly bootstrap.servers and the capacity.config.file path (default config/capacityJBOD.json) that describes broker disk, CPU, and network capacity. Template the capacity file from inventory too, so that adding or removing brokers keeps the capacity model in sync without manual edits. A stale capacity file is a silent failure mode: Cruise Control will happily plan against capacities that no longer match the hardware.

Configuring Anomaly Detection and Self-Healing Notifications

Cruise Control’s anomaly detectors—goal violations, broker failures, disk failures, metric anomalies, and topic anomalies—are the triggers that initiate self-healing. In an automated setup you want Ansible to configure these detectors and route their notifications to your existing alerting channels. The notification path is driven by the anomaly.notifier.class property. The default is SelfHealingNotifier, which only logs; the project also ships notifiers that post to Slack, Microsoft Teams, and Alerta, each subclassing SelfHealingNotifier.

Self-healing is opt-in. The master switch is self.healing.enabled, and you can additionally gate it per anomaly type (for example self.healing.goal.violation.enabled). A common production stance is to enable self-healing for broker.failure and disk.failure—where waiting for a human costs availability—while leaving goal.violation notify-only, so a person reviews load-driven moves. The snippet below enables self-healing, selects the Slack notifier, and points it at a webhook whose URL comes from an Ansible Vault–encrypted variable so secrets stay out of plaintext:

# templates/cruisecontrol.properties.j2 (excerpt)
self.healing.enabled=true
anomaly.detection.goals=com.linkedin.kafka.cruisecontrol.analyzer.goals.RackAwareGoal,com.linkedin.kafka.cruisecontrol.analyzer.goals.ReplicaCapacityGoal,com.linkedin.kafka.cruisecontrol.analyzer.goals.DiskCapacityGoal

anomaly.notifier.class=com.linkedin.kafka.cruisecontrol.detector.notifier.SlackSelfHealingNotifier
slack.self.healing.notifier.webhook={{ cruise_control_slack_webhook }}
slack.self.healing.notifier.channel="#kafka-alerts"

Note that anomaly.detection.goals must be a subset of default.goals, which in turn must be a superset of hard.goals; otherwise Cruise Control refuses to start. This ordering constraint is the most common reason a freshly templated properties file fails to boot, so validate it in CI rather than discovering it on deploy. If you need to post to a system the built-in notifiers don’t cover, implement a custom class against the SelfHealingNotifier/AnomalyNotifier API, drop the JAR into the Cruise Control libs directory with Ansible, and reference it in anomaly.notifier.class. There is no generic anomaly.notifier.webhook.url property—each notifier defines its own keys.

After deploying the configuration, run a post-deployment validation task that queries the /kafkacruisecontrol/state endpoint with substates=anomaly_detector and confirms self-healing is enabled for the anomaly types you expect. A silent misconfiguration that disables anomaly detection leaves the cluster unmonitored for imbalance—and because the failure is silent, you only learn about it when a broker is already hot.

Triggering On-Demand Rebalances via Ansible

While self-healing handles steady state, operators frequently need an explicit rebalance: after a broker replacement, before a planned maintenance window, or in response to a sudden traffic spike. Ansible’s uri module fits Cruise Control’s REST API well. Two facts shape the playbook:

• /proposals is a GET that computes a proposal without moving data; /rebalance is a POST that defaults to dryrun=true and only moves data when you pass dryrun=false.
• All options are query parameters. goals takes simple class names (RackAwareGoal), not fully-qualified names. Booleans use snake_case: allow_capacity_estimation, exclude_recently_demoted_brokers, exclude_recently_removed_brokers.

The playbook below fetches a proposal first (a pure preview—no body, no data movement), inspects the summary, and only then executes:

# playbooks/rebalance.yml
- name: Cruise Control rebalance
  hosts: cruise_control[0]
  vars:
    cc_api_base: "http://{{ ansible_host }}:9090/kafkacruisecontrol"
  tasks:
    - name: Fetch rebalance proposal (preview, no data movement)
      ansible.builtin.uri:
        url: >-
          {{ cc_api_base }}/proposals?goals=RackAwareGoal,ReplicaCapacityGoal&allow_capacity_estimation=false&exclude_recently_demoted_brokers=true&exclude_recently_removed_brokers=true&json=true
        method: GET
        timeout: 120
      register: proposal

    - name: Display proposal summary
      ansible.builtin.debug:
        msg: >-
          Replica movements: {{ proposal.json.summary.numReplicaMovements }},
          leader movements: {{ proposal.json.summary.numLeaderMovements }},
          inter-broker data to move: {{ proposal.json.summary.dataToMoveMB }} MB

    - name: Execute rebalance if approved
      ansible.builtin.uri:
        url: >-
          {{ cc_api_base }}/rebalance?dryrun=false&goals=RackAwareGoal,ReplicaCapacityGoal&exclude_recently_demoted_brokers=true&exclude_recently_removed_brokers=true&json=true
        method: POST
        timeout: 300
        return_content: true
      register: rebalance
      when: execute_rebalance | default(false) | bool

The summary object is real and stable: numReplicaMovements, numLeaderMovements, dataToMoveMB, numIntraBrokerReplicaMovements, intraBrokerDataToMoveMB, monitoredPartitionsPercentage, and excludedTopics are all returned by /proposals and /rebalance when json=true. Read dataToMoveMB before approving: a multi-terabyte move on a production cluster can saturate replication bandwidth for hours. The execute_rebalance variable is the manual gate—in a pipeline, run the proposal step on a schedule, post the summary to Slack, and require human approval before the POST.

Throttling: the lever most automation forgets

A rebalance that moves data as fast as the cluster allows will compete with live producer and consumer traffic. Cruise Control caps this with num.concurrent.partition.movements.per.broker (default 5), and every /rebalance call can override it per-request without restarting the service. The relevant query parameters are concurrent_partition_movements_per_broker, concurrent_leader_movements, concurrent_intra_broker_partition_movements, and max_partition_movements_in_cluster. For a maintenance-window rebalance you might raise concurrency to drain faster; for a mid-day fix on a latency-sensitive cluster, lower it so the move stays in the background:

# conservative, daytime: minimize impact on live traffic
.../rebalance?dryrun=false&goals=RackAwareGoal&concurrent_partition_movements_per_broker=2&json=true

Throttling is the difference between a rebalance your applications never notice and one that pages your on-call. Make the concurrency a per-cluster Ansible variable rather than a hardcoded default.

Critically, /rebalance is asynchronous. If the operation runs longer than webserver.request.maxBlockTimeMs (default 10000 ms), Cruise Control returns a User-Task-Id header (a UUID) instead of the final result, and the move continues in the background. For any non-trivial rebalance you will cross this threshold, so treat the async path as the normal case—don’t treat the immediate response as “done.” Poll the /kafkacruisecontrol/user_tasks endpoint—optionally filtered by user_task_ids—until the task reaches a terminal state, then log the final summary:

    - name: Wait for rebalance task to complete
      ansible.builtin.uri:
        url: "{{ cc_api_base }}/user_tasks?user_task_ids={{ rebalance.user_task_id }}&json=true"
        method: GET
      register: task_state
      until: task_state.json.userTasks[0].Status in ['Completed', 'CompletedWithError']
      retries: 60
      delay: 30
      when: execute_rebalance | default(false) | bool

One operational footnote: Cruise Control limits concurrent async user tasks via max.active.user.tasks (default 5). If a scheduled rebalance, a self-healing action, and an operator’s manual call collide, later requests can be rejected—another reason to gate on-demand runs rather than fire them blindly.

For fully autonomous environments—staging clusters where temporary imbalance is acceptable—you can drop the gate and let the playbook execute directly, but keep the /user_tasks poll so the run only reports success once the executor has actually finished.

Scheduling Periodic Rebalancing with Ansible and Cron

Not every team wants fully automated, anomaly-triggered rebalancing. A middle ground is to schedule periodic rebalances with Ansible-managed cron jobs, giving you predictable, time-bounded windows that align with change-management processes.

Ansible’s cron module deploys a wrapper script that calls the API, kicks off a non-dry-run rebalance, and then tracks the resulting task to completion. Because /rebalance returns asynchronously, the script captures the User-Task-Id header and polls /user_tasks rather than grepping the initial response for a status that isn’t there:

#!/usr/bin/env bash
# /usr/local/bin/cruise-control-rebalance.sh
# Managed by Ansible — do not edit manually.
set -euo pipefail

API_BASE="http://localhost:9090/kafkacruisecontrol"
LOG_FILE="/var/log/cruise-control/rebalance-$(date +%Y%m%d-%H%M).log"
GOALS="RackAwareGoal,ReplicaCapacityGoal"

echo "Starting scheduled rebalance at $(date)" >> "$LOG_FILE"

# Kick off the rebalance and capture the async User-Task-Id header.
TASK_ID=$(curl -s -D - -o /dev/null -X POST \
  "${API_BASE}/rebalance?dryrun=false&goals=${GOALS}&exclude_recently_demoted_brokers=true&exclude_recently_removed_brokers=true&json=true" \
  | awk -F': ' 'tolower($1)=="user-task-id"{print $2}' | tr -d '\r')

if [[ -z "$TASK_ID" ]]; then
  echo "No User-Task-Id returned; rebalance completed synchronously or failed. See logs." >> "$LOG_FILE"
  exit 1
fi

echo "Tracking task ${TASK_ID}" >> "$LOG_FILE"
for _ in $(seq 1 60); do
  STATUS=$(curl -s "${API_BASE}/user_tasks?user_task_ids=${TASK_ID}&json=true" \
    | python3 -c 'import sys,json; print(json.load(sys.stdin)["userTasks"][0]["Status"])')
  echo "Status: ${STATUS}" >> "$LOG_FILE"
  case "$STATUS" in
    Completed) echo "Rebalance completed successfully." >> "$LOG_FILE"; exit 0 ;;
    CompletedWithError) echo "Rebalance finished with errors." >> "$LOG_FILE"; exit 1 ;;
  esac
  sleep 30
done

echo "Rebalance did not reach a terminal state in the allotted time." >> "$LOG_FILE"
exit 1

The Ansible task to schedule it:

- name: Schedule weekly rebalance via cron
  ansible.builtin.cron:
    name: "Cruise Control weekly rebalance"
    job: "/usr/local/bin/cruise-control-rebalance.sh"
    weekday: "0"
    hour: "3"
    minute: "0"
    user: "{{ cruise_control_user }}"

This runs every Sunday at 03:00, a typical low-traffic window. The 30-second poll times out after 60 iterations (~30 minutes); size that ceiling against your largest expected dataToMoveMB, since a large move that is still healthy will otherwise exit non-zero and page someone. Manage log rotation for the output directory with Ansible too, via logrotate config or a cleanup cron job. Keeping the schedule in version control lets you audit when rebalances occur and coordinate them with other maintenance such as broker rolling restarts.

Validating Rebalance Outcomes and Cluster Health

Automation without verification is dangerous. After any rebalance, validate that the cluster reached a healthier state. The /state endpoint exposes several substates, selected with substates=monitor,executor,analyzer,anomaly_detector. Mind the JSON field names—they are not what an “imbalance dashboard” might suggest:

• The monitor substate (MonitorState in the JSON) reports data-coverage fields such as numValidWindows, numValidPartitions, and state. It does not report average CPU or disk utilization—those live in the load model returned by load/proposal endpoints, not in /state.
• The anomaly-detector substate (AnomalyDetectorState) reports selfHealingEnabled, selfHealingDisabled, and recentGoalViolations (a list of recent violations, not a single count).

Coverage matters more than it looks. Cruise Control will not produce a valid load model—and therefore will not rebalance meaningfully—until it has monitored at least min.valid.partition.ratio of partitions (default 0.995). A post-rebalance validation that ignores coverage can “pass” against a stale or partial model. So the check is two-pronged: confirm the monitor has full data coverage, then assert there are no recent goal violations.

- name: Validate cluster state after rebalance
  hosts: cruise_control[0]
  vars:
    cc_api_base: "http://{{ ansible_host }}:9090/kafkacruisecontrol"
  tasks:
    - name: Get monitor and anomaly-detector state
      ansible.builtin.uri:
        url: "{{ cc_api_base }}/state?substates=monitor,anomaly_detector&json=true"
        method: GET
      register: cluster_state

    - name: Assert no recent goal violations
      ansible.builtin.assert:
        that:
          - (cluster_state.json.AnomalyDetectorState.recentGoalViolations | length) == 0
        fail_msg: >-
          Cluster still reports {{ cluster_state.json.AnomalyDetectorState.recentGoalViolations | length }}
          recent goal violation(s) after rebalance.

    - name: Report monitor coverage
      ansible.builtin.debug:
        msg:
          - "Monitor state: {{ cluster_state.json.MonitorState.state }}"
          - "Valid windows: {{ cluster_state.json.MonitorState.numValidWindows }}"
          - "Valid partitions: {{ cluster_state.json.MonitorState.numValidPartitions }}"

For deeper integration, push these signals into your observability stack. Cruise Control exposes its own JMX metrics; the Prometheus JMX Exporter can scrape them, and Ansible can deploy that exporter config alongside the service. This closes the loop: Ansible provisions Cruise Control, drives rebalances, and surfaces the results in your dashboards.

Managing Multi-Cluster and Multi-Environment Deployments

Production Kafka deployments often span multiple clusters across regions or environments, and Ansible’s inventory system models that naturally. Define group variables per cluster—bootstrap servers, capacity configuration, rebalancing goals, and the throttling concurrency discussed earlier—and apply the same role and playbooks uniformly, with per-cluster overrides handled through the inventory hierarchy.

# inventories/production/hosts.ini
[us_east_kafka_brokers]
broker-east-1 ansible_host=10.0.1.11
broker-east-2 ansible_host=10.0.1.12

[us_east_cruise_control]
cc-east-1 ansible_host=10.0.1.100

[eu_west_kafka_brokers]
broker-west-1 ansible_host=10.0.2.11
broker-west-2 ansible_host=10.0.2.12

[eu_west_cruise_control]
cc-west-1 ansible_host=10.0.2.100

Group variables in group_vars/us_east_cruise_control.yml and group_vars/eu_west_cruise_control.yml set region-specific goals—perhaps the EU cluster enforces strict rack awareness for data-residency reasons while the US cluster prioritizes CPU balance. Target a specific cluster by limiting the run to its group:

ansible-playbook -i inventories/production playbooks/rebalance.yml --limit us_east_cruise_control

This scales to dozens of clusters without duplicating logic, and enables canary rollouts: ship a new Cruise Control version or changed goal configuration to a staging cluster first, validate, then promote to production. The Apache Kafka documentation offers complementary operational guidance.

Securing the Cruise Control API

The REST API can trigger data movement across the entire cluster, so in production it must be secured. If your Kafka cluster uses mTLS or SASL/SCRAM, configure Cruise Control to authenticate to the brokers accordingly via its cruisecontrol.properties security settings. The API itself should be protected too—Cruise Control supports HTTPS and pluggable security providers (e.g. Basic, SPNEGO/Kerberos, JWT), and you can additionally front it with an authenticating reverse proxy.

A common pattern is an Nginx proxy enforcing client-certificate auth, deployed by Ansible:

# templates/nginx-cruise-control.conf.j2
server {
    listen 443 ssl;
    server_name cruise-control.{{ inventory_hostname }}.internal;

    ssl_certificate     /etc/ssl/certs/cruise-control.crt;
    ssl_certificate_key /etc/ssl/private/cruise-control.key;
    ssl_client_certificate /etc/ssl/certs/internal-ca.crt;
    ssl_verify_client   on;

    location / {
        proxy_pass http://127.0.0.1:9090;
        proxy_set_header Host      $host;
        proxy_set_header X-Real-IP $remote_addr;
    }
}

Issue and rotate the certificates from a separate Ansible role integrated with your PKI, such as HashiCorp Vault or cert-manager. Layering HTTPS, client-certificate or SSO authentication, and least-privilege network access ensures only authorized systems and operators can initiate rebalancing—a gap that is easy to overlook when automating cluster operations, and one where a single unauthenticated POST can move terabytes.

Automating Cruise Control with Ansible turns rebalancing from a manual, error-prone chore into a codified, auditable process. The load-bearing patterns are: pick an operating mode per anomaly type, deploy a CI-built artifact, preview with GET /proposals before POSTing /rebalance, throttle deliberately, track the async /user_tasks to a terminal state, and validate against monitor coverage and goal violations before declaring success. As your Kafka footprint grows, this automation becomes less a convenience than a necessity for maintaining the balance your applications depend on.