Automating Topic Repartitioning with Cruise Control and Prometheus

Partition layouts that once served a workload well degrade as traffic shifts, services come online, and retention policies change. The symptoms are familiar: consumer lag concentrated on a few partitions, producer timeouts, and brokers whose CPU or disk usage drift apart. Manual reassignment is slow and reactive. This guide builds a closed loop that detects imbalance with Prometheus, validates a proposed fix, and lets Cruise Control execute it under throttle—without inventing metrics or endpoints that do not exist.

Everything below is grounded in the real Cruise Control REST API, the real metric names produced by the Prometheus JMX Exporter, and the real kafka_exporter consumer-lag metrics. Where a name depends on your exporter configuration, that dependency is called out explicitly.

The techniques here build on Topics, Partitions & Data Modeling, which covers how partition counts, keying, and replication factor determine throughput and fault tolerance. For Cruise Control’s architecture, goals, and execution model, see Automated Repartitioning with Kafka Cruise Control.

Understanding the Repartitioning Trigger Pipeline

Before writing automation, define precisely what should trigger it. Every reassignment consumes network bandwidth, raises disk I/O, and adds controller metadata churn. Repartition only when the operational benefit clearly outweighs that cost.

The pipeline has three stages: metric collection, threshold evaluation against your Service Level Objectives (SLOs), and a gating layer that prevents repartitioning storms. Each stage must be independently observable—when automation misbehaves in a production messaging system, the blast radius is large.

A critical distinction up front: Cruise Control moves replicas between brokers (reassignment) and changes leadership; it does not increase a topic’s partition count. Increasing partitions (kafka-topics.sh --alter --partitions) is a separate, irreversible operation that breaks key-based ordering and is not something to automate blindly. This pipeline therefore automates rebalancing of existing partitions and treats “insufficient partitions” as an alert for human review, not for automated execution.

That distinction drives the entire eligibility model. Each condition is classified by whether a rebalance can actually fix it:

Only the first two are things a replica/leader move can correct. The rest indicate infrastructure problems or require a partition-count change; routing them to automation would, at best, do nothing and, at worst, pile reassignment I/O onto an already-degraded cluster.

Metric Selection and Rationale

We draw signals from three real sources, each scraped by Prometheus:

1. Kafka broker JMX, via the Prometheus JMX Exporter run as a Java agent on every broker. Metric names depend on the exporter’s rule file; the names below match the widely used Strimzi kafka-metrics.yaml (which sets lowercaseOutputName: true).
2. Cruise Control’s own JMX metrics, exposed by running the JMX Exporter as an agent on the Cruise Control process. The Strimzi kafka-cruise-control-metrics.yaml rule kafka.cruisecontrol<name=(.+)><>(\w+) → kafka_cruisecontrol_$1_$2 turns the MBean kafka.cruisecontrol:name=AnomalyDetector.balancedness-score into kafka_cruisecontrol_anomalydetector_balancedness_score.
3. Consumer lag, via kafka_exporter, which exposes kafka_consumergroup_lag with labels consumergroup, topic, and partition.

The balancedness-score metric (0 = worst, 100 = best) is the cleanest cluster-wide imbalance signal Cruise Control publishes: it is the cumulative score of how well the cluster satisfies the goals in anomaly.detection.goals, recomputed every anomaly.detection.interval.ms (default 1 hour). A falling balancedness score is exactly the structural-imbalance condition we want to gate automation on.

# prometheus-rules/kafka-repartitioning-alerts.yml
groups:
  - name: kafka_repartitioning
    interval: 60s
    rules:
      # Broker request-handler threads starved (low idle) on at least one
      # broker, while Cruise Control reports the cluster is imbalanced.
      # RequestHandlerAvgIdlePercent is a MeanRate in [0,1]; < 0.2 == >80% busy.
      - alert: KafkaBrokerLoadImbalance
        expr: |
          (
            min by (kafka_broker) (
              kafka_server_kafkarequesthandlerpool_requesthandleravgidlepercent{job="kafka-jmx"}
            ) < 0.2
          )
          and on()
          (
            min(kafka_cruisecontrol_anomalydetector_balancedness_score{job="cruise-control-jmx"}) < 80
          )
        for: 15m
        labels:
          severity: warning
          automation_eligible: "true"
        annotations:
          summary: "Broker load imbalance detected"
          description: >
            At least one broker has sustained request-handler idle below 20%
            for 15m while the Cruise Control balancedness score is below 80.
            Automated rebalancing may be triggered if gating conditions pass.

      # Per-broker bytes-in skew for a topic: a broker carrying >2x the
      # topic's average inbound rate. Computed from real broker JMX metrics.
      - alert: KafkaTopicBytesInSkew
        expr: |
          max by (topic) (
            rate(kafka_server_brokertopicmetrics_bytesinpersec_total{
              topic=~"high-throughput-.*"
            }[5m])
          )
          /
          avg by (topic) (
            rate(kafka_server_brokertopicmetrics_bytesinpersec_total{
              topic=~"high-throughput-.*"
            }[5m])
          ) > 2
        for: 30m
        labels:
          severity: warning
          automation_eligible: "true"
        annotations:
          summary: "Per-broker bytes-in skew for topic {{ $labels.topic }}"
          description: >
            The busiest broker for topic {{ $labels.topic }} is taking more
            than 2x the per-broker average inbound rate for 30m. A leadership
            or replica rebalance should even this out.

      # Consumer lag hotspot. NOT automation-eligible: increasing partition
      # count is the only structural fix and that is a manual, irreversible
      # change. This alert pages a human.
      - alert: KafkaConsumerLagHotspot
        expr: |
          sum by (consumergroup, topic) (
            kafka_consumergroup_lag{topic=~"high-throughput-.*"}
          ) > 100000
        for: 10m
        labels:
          severity: critical
          automation_eligible: "false"
        annotations:
          summary: "Consumer lag hotspot on {{ $labels.topic }}"
          description: >
            Group {{ $labels.consumergroup }} on topic {{ $labels.topic }} has
            aggregate lag over 100k. Investigate consumer throughput and
            partition count manually before any reassignment.

The automation_eligible label is the gate: the controller acts only on alerts marked "true". Under-replicated or offline partitions, controller failures, and the lag hotspot above are deliberately excluded—those indicate infrastructure problems (or require a partition-count change) that a rebalance would not fix and could worsen.

Two correctness notes that matter in practice:

• RequestHandlerAvgIdlePercent is exported from a JMX MeanRate attribute as a fraction in [0,1], not a percentage. The Strimzi rules produce kafka_server_kafkarequesthandlerpool_requesthandleravgidlepercent with no _total suffix (it is a gauge, not a counter). Comparing < 0.2 means “less than 20% idle.”
• BrokerTopicMetrics BytesInPerSec is a Count and is exported as the counter kafka_server_brokertopicmetrics_bytesinpersec_total. You must wrap it in rate() to get a throughput; the topic label is present only on the per-topic MBeans, so this skew query is per-broker-per-topic.

Threshold Tuning and SLO Alignment

These thresholds are starting points, not constants. A cluster serving asynchronous analytics tolerates far more skew than one powering real-time fraud detection. To calibrate, scrape kafka_cruisecontrol_anomalydetector_balancedness_score alongside your application latency and error-rate SLIs for at least two weeks, and find the score below which user-facing degradation reliably appears. That crossover is your automation trigger point. Document the calibration in your runbooks.

The for: durations are deliberately long for a reason: they must outlast normal transients. A 15m window on KafkaBrokerLoadImbalance rides out batch jobs, partition-leader elections, and short producer bursts that resolve on their own; shorten it and automation will chase noise. The bytes-in skew alert uses 30m because topic-level skew that disappears in minutes is usually a hot key in a single producer, not a placement problem a rebalance can fix.

Gating Conditions and Storm Prevention

A rebalance temporarily raises broker load, which could re-fire an alert and trigger another rebalance indefinitely. Prevent that with a gating controller enforcing:

1. Cooldown — no automated rebalance within 30 minutes of the previous one completing.
2. Concurrency limit — at most one automated operation in flight.
3. Executor idle check — Cruise Control must report no execution in progress before starting (queried via /kafkacruisecontrol/state).
4. Maintenance-window awareness — automation suppressed during declared windows.
5. Throttle — replication_throttle set conservatively so reassignment never starves client traffic.

The controller is a small stateful service that consumes alerts via Alertmanager webhooks, checks cluster state through the Cruise Control REST API, and persists recent-operation history.

Wiring Cruise Control into Prometheus

There is no official “Cruise Control Prometheus exporter” image, and Cruise Control does not natively serve a /metrics endpoint. Two real, complementary integrations cover everything this pipeline needs.

1. Expose Cruise Control’s JMX metrics to Prometheus

Run the Prometheus JMX Exporter as a Java agent on the Cruise Control JVM, with the Strimzi Cruise Control rule file. This publishes kafka_cruisecontrol_* gauges—including kafka_cruisecontrol_anomalydetector_balancedness_score and kafka_cruisecontrol_executor_replica_action_in_progress—on the agent’s HTTP port.

# Snippet from the JMX exporter rule file used on the Cruise Control JVM.
# Produces e.g. kafka_cruisecontrol_anomalydetector_balancedness_score
lowercaseOutputName: true
rules:
  - pattern: "kafka.cruisecontrol<name=(.+)><>(\\w+)"
    name: "kafka_cruisecontrol_$1_$2"

If you run Cruise Control under Strimzi, this is wired automatically when you enable metricsConfig on the cruiseControl resource.

2. (Optional) Let Cruise Control sample from Prometheus

Instead of Cruise Control’s default Kafka-topic metrics reporter, you can have Cruise Control pull broker metrics from your existing Prometheus by setting its PrometheusMetricSampler:

# cruisecontrol.properties
metric.sampler.class=com.linkedin.kafka.cruisecontrol.monitor.sampling.prometheus.PrometheusMetricSampler
prometheus.server.endpoint=http://prometheus.kafka-platform.svc.cluster.local:9090
# Must match how often Prometheus scrapes broker JMX, used to build the iRate query:
prometheus.broker.metrics.scraping.frequency.seconds=60

This removes the need to deploy Cruise Control’s metrics-reporter on every broker if you already scrape broker JMX. Confirm samples are landing via /kafkacruisecontrol/state?substates=monitor and watch the Cruise Control logs for “invalid query results,” which usually means the broker JMX metric names in Prometheus do not match what the query supplier expects.

Configuring Prometheus to scrape both targets

With the Prometheus Operator, declare a ServiceMonitor per JMX-agent port (one for brokers, one for Cruise Control):

# servicemonitor-cruise-control-jmx.yaml
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: cruise-control-jmx
  namespace: kafka-platform
  labels:
    release: prometheus
spec:
  selector:
    matchLabels:
      app: cruise-control
  namespaceSelector:
    matchNames: [kafka-platform]
  endpoints:
    - port: jmx-metrics      # the JMX exporter agent port on the CC pod
      interval: 60s
      scrapeTimeout: 30s
      relabelings:
        - sourceLabels: [__meta_kubernetes_pod_name]
          targetLabel: instance

Verify by querying Prometheus for kafka_cruisecontrol_anomalydetector_balancedness_score; you should get a single time series. If it is absent, the JMX exporter agent is not attached to the Cruise Control JVM, or the rule file does not match the kafka.cruisecontrol domain.

Broker JMX metrics for correlation

The broker JMX agent gives you the raw telemetry the alerts above depend on. Useful real metric names (Strimzi rules): kafka_server_brokertopicmetrics_bytesinpersec_total and ..._bytesoutpersec_total (counters; use rate()), kafka_server_kafkarequesthandlerpool_requesthandleravgidlepercent (gauge, fraction), and kafka_server_replicamanager_underreplicatedpartitions (gauge). The dual-source design—broker load plus Cruise Control balancedness—keeps a transient CPU spike from a rogue client from triggering a rebalance unless the cluster is also structurally imbalanced.

Building the Automation Controller

With metrics flowing and alerts firing, a controller translates eligible alerts into Cruise Control API calls. It must be reliable, observable, and safe by default.

Architecture overview

A single-binary Go service (Python or Java work equally well) exposing:

• POST /webhooks/alerts — Alertmanager webhook receiver
• GET /health — liveness/readiness
• GET /metrics — the controller’s own Prometheus metrics

Internally it runs a finite state machine: IDLE, EVALUATING, EXECUTING, COOLDOWN, ERROR. Transitions are logged at INFO and emitted as counters so you can see how often automation fires.

Alert ingestion and gating

On each webhook, the controller keeps alerts with automation_eligible: "true" and evaluates gating:

// controller/gating.go (simplified)
func (c *Controller) evaluateGatingConditions(alert Alert) (bool, string) {
    if time.Since(c.lastOperationTime) < c.config.CooldownPeriod {
        return false, "cooldown period active"
    }
    if c.currentState == EXECUTING {
        return false, "another operation in flight"
    }
    // Reject if Cruise Control's executor is already moving replicas.
    busy, err := c.cruiseControlClient.ExecutionInProgress()
    if err != nil || busy {
        return false, "cruise control executor busy or unreachable"
    }
    if c.inMaintenanceWindow(time.Now()) {
        return false, "maintenance window active"
    }
    return true, ""
}

ExecutionInProgress() reads /kafkacruisecontrol/state?substates=executor&json=true and treats anything other than NO_TASK_IN_PROGRESS in executorState.state as busy. (json is a valid parameter on /state; the response is JSON.)

Requesting and validating a proposal

Cruise Control’s proposal endpoint is GET /kafkacruisecontrol/proposals—it retrieves a plan, it does not execute anything. Use a conservative, capacity-first goal list:

# Retrieve a proposal (read-only). Note: GET, not POST.
curl "http://cruise-control:9090/kafkacruisecontrol/proposals?verbose=true&ignore_proposal_cache=true&goals=com.linkedin.kafka.cruisecontrol.analyzer.goals.RackAwareGoal,com.linkedin.kafka.cruisecontrol.analyzer.goals.ReplicaCapacityGoal,com.linkedin.kafka.cruisecontrol.analyzer.goals.DiskCapacityGoal,com.linkedin.kafka.cruisecontrol.analyzer.goals.CpuCapacityGoal,com.linkedin.kafka.cruisecontrol.analyzer.goals.ReplicaDistributionGoal&excluded_topics=__.*"

RackAwareGoal, ReplicaCapacityGoal, DiskCapacityGoal, CpuCapacityGoal, and ReplicaDistributionGoal are all real analyzer goal classes (part of Cruise Control’s default goal list). The controller parses the proposal summary and validates before proceeding:

1. Data movement volume — total replica reassignments below a threshold (default 10% of replicas). Moving too much at once risks saturating the network.
2. Leadership changes — below a threshold (default 5% of partitions). Excessive leader moves cause brief unavailability for acks=all producers.
3. Excluded brokers — the plan must not move replicas onto brokers marked for decommissioning.

On failure the controller logs the reason, increments repartitioning_proposal_rejected_total, and returns to IDLE. That transparency is what lets you debug why automation did not fire when expected.

Executing the reassignment safely

To execute, call POST /kafkacruisecontrol/rebalance with dryrun=false and the same goals, throttled. Pass concurrent_partition_movements_per_broker, concurrent_leader_movements, and replication_throttle (bytes/second) to bound the impact:

# Execute a throttled rebalance. dryrun=false is required to move data;
# every POST defaults to dryrun=true as a safety measure.
curl -X POST "http://cruise-control:9090/kafkacruisecontrol/rebalance?dryrun=false&goals=com.linkedin.kafka.cruisecontrol.analyzer.goals.RackAwareGoal,com.linkedin.kafka.cruisecontrol.analyzer.goals.ReplicaCapacityGoal,com.linkedin.kafka.cruisecontrol.analyzer.goals.DiskCapacityGoal,com.linkedin.kafka.cruisecontrol.analyzer.goals.ReplicaDistributionGoal&concurrent_partition_movements_per_broker=2&concurrent_leader_movements=50&replication_throttle=10485760"

replication_throttle=10485760 caps inter-broker reassignment bandwidth at 10 MB/s so client traffic is not starved. The concurrent_* parameters provide backpressure on the number of simultaneous moves; if you omit them, Cruise Control falls back to the cluster defaults (num.concurrent.partition.movements.per.broker, num.concurrent.leader.movements).

Pick the throttle from headroom, not habit. If brokers run replication links at, say, 40 MB/s peak against a 125 MB/s (1 GbE) NIC, a 10 MB/s cap leaves client traffic untouched while still draining a moderate reassignment in minutes-to-hours; on 10 GbE you can afford far more. The failure signature of a too-high throttle is unmistakable: producer p99 latency climbs and bytesout/replication saturates the link the moment the rebalance starts—which is exactly the “rebalance caused degradation” runbook below.

If you enable two-step verification (two.step.verification.enabled=true), POSTs are not executed immediately—Cruise Control returns a review_id, and you (or an approver) re-issue the request with that review_id to run the approved command. The review_id therefore comes from the review workflow, never from the /proposals response. For unattended automation most teams leave two-step verification off and rely on the controller’s own gating instead.

The controller then polls GET /kafkacruisecontrol/state?substates=executor every 30 seconds, reading executorState.state. When it returns to NO_TASK_IN_PROGRESS, the operation is complete. The controller records the timestamp, moves to COOLDOWN, and emits repartitioning_completed_total with labels for partitions moved and duration.

Alertmanager Webhook Integration

Alertmanager bridges Prometheus alerting and the controller. Route only eligible alerts to it, and batch them so an incident storm cannot trigger overlapping evaluations.

# alertmanager.yml (excerpt)
route:
  receiver: default
  group_by: ['alertname', 'severity']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    - matchers:
        - automation_eligible="true"
      receiver: kafka-repartitioning-controller
      group_wait: 60s
      group_interval: 10m
      repeat_interval: 1h

receivers:
  - name: default
    # ... existing receivers ...
  - name: kafka-repartitioning-controller
    webhook_configs:
      - url: http://kafka-automation-controller.kafka-platform.svc.cluster.local:8080/webhooks/alerts
        send_resolved: false
        http_config:
          basic_auth:
            username: alertmanager
            password_file: /etc/alertmanager/secrets/controller_password

Key decisions:

• group_interval: 10m — the controller receives at most one webhook per group every 10 minutes, preventing repeated evaluations.
• repeat_interval: 1h — a still-firing alert is resent hourly, a retry path if the controller was down on first delivery.
• send_resolved: false — the cooldown handles backoff; resolved notifications add no value here.
• Basic auth — the webhook must authenticate; store the password in a Kubernetes Secret. (Recent Alertmanager versions use the matchers: list above; the legacy match: map is deprecated.)

The controller’s webhook handler verifies credentials and rate-limits to one request per minute, matching group_interval—defense in depth even if Alertmanager misbehaves.

Testing the Pipeline End-to-End

Validate the whole pipeline in a staging cluster that mirrors production topology before enabling live execution. Cover the happy path and the failure modes.

Generating broker load with Trogdor

The Kafka project ships Trogdor, a test framework whose ProduceBenchSpec generates real produce traffic—useful for driving a broker’s request-handler idle down and the balancedness score with it. The spec class is ProduceBenchSpec (not a “WorkloadSpec”), activeTopics/inactiveTopics are maps keyed by topic-name pattern, and producerNode names a Trogdor agent node, not a Kafka broker host:

{
  "class": "org.apache.kafka.trogdor.workload.ProduceBenchSpec",
  "durationMs": 1800000,
  "producerNode": "node0",
  "bootstrapServers": "broker-3:9092",
  "targetMessagesPerSec": 100000,
  "maxMessages": 50000000,
  "activeTopics": {
    "trogdor-load-[1-3]": { "numPartitions": 12, "replicationFactor": 3 }
  },
  "inactiveTopics": {}
}

Submit it to the Trogdor coordinator (default port 8889). The supported path is the bundled client:

./bin/trogdor.sh client createTask \
  -t localhost:8889 -i produce-load-0 --spec ./produce-bench.json

Then watch KafkaBrokerLoadImbalance fire within the expected window, and confirm from the controller logs that it received the alert, passed gating, retrieved a proposal, and (if everything held) executed a throttled rebalance.

Validating the dry-run path

Before allowing live execution, run the controller with EXECUTION_MODE=dryrun. It performs every step up to and including proposal validation but issues the rebalance with dryrun=true. Verify that the proposal is logged in full, the controller transitions EVALUATING → IDLE without hitting the live path, and repartitioning_proposal_validated_total increments. Only once dry-run proposals look sensible should you enable live execution.

Chaos: controller crash mid-execution

Cruise Control’s executor runs independently of the controller, so a reassignment continues if the controller dies. On restart the controller queries /kafkacruisecontrol/state?substates=executor, finds the in-progress execution, logs it, and goes straight to COOLDOWN instead of starting a new operation. Test by killing the controller mid-execution and confirming no duplicate rebalance is launched.

Operational Runbooks and Observability

Automation cuts toil but concentrates complexity. The on-call engineer needs clear runbooks and rich signals.

Key metrics to dashboard

• repartitioning_triggered_total (by alertname) — how often automation fires; a spike hints at misconfiguration or a traffic shift.
• repartitioning_proposal_rejected_total (by reason) — frequent volume rejections mean thresholds are too tight or the cluster is badly imbalanced.
• repartitioning_duration_seconds (histogram) — track p95 to set realistic recovery expectations.
• kafka_cruisecontrol_executor_replica_action_in_progress — Cruise Control’s own gauge for in-flight replica moves; alert if it stays above 0 for over an hour (stuck reassignment).
• controller_state — the controller’s FSM state; alert if ERROR persists beyond 5 minutes.

Failure modes and how to detect them

The four ways this pipeline fails in production, each with the signal that surfaces it and the first move:

Runbook: automation did not fire when expected

1. Is the alert still firing, and did it stay firing for the full for: duration? If it resolved early, the condition was not sustained.
2. Check gating rejections: grep "gating" /var/log/kafka-automation-controller.log—usually an active cooldown or a busy executor.
3. Check Cruise Control’s anomaly/executor state: curl "http://cruise-control:9090/kafkacruisecontrol/state?substates=anomaly_detector,executor&json=true". If the anomaly detector is disabled or the executor is stuck, automation will not proceed. (Cruise Control has no /anomalies endpoint; anomaly state lives under /state.)
4. Confirm webhook delivery: Alertmanager logs for send errors, and the controller’s /metrics for webhook_requests_total.

Runbook: rebalance caused performance degradation

1. Stop the executor immediately: curl -X POST "http://cruise-control:9090/kafkacruisecontrol/stop_proposal_execution". (This is a POST endpoint. In-progress batches may finish before the executor halts.)
2. Check the throttle: was replication_throttle set? An unset or too-high throttle lets reassignment saturate inter-broker links.
3. Check for overlap: a manual rebalance running alongside the automated one—the concurrency limit only guards automated operations.
4. Reverse if needed: Cruise Control has no native rollback, but you can generate and execute a proposals/rebalance request with the original placement (e.g. via --rebalance-disk off and a fixed goal set) or hand-craft a reverse assignment and apply it with kafka-reassign-partitions.sh.

Production Hardening and Gradual Rollout

Automating reassignment is high-stakes. Roll it out in stages, widening scope only after proving reliability.

Phase 1: alert-only

Deploy the Prometheus rules and the JMX exporters, but not the controller. Let alerts drive manual rebalances for at least two weeks to confirm thresholds are calibrated and alerts are actionable. If more than ~20% of alerts are false positives, retune before proceeding.

Phase 2: dry-run automation

Deploy the controller in dry-run mode. Every eligible alert generates and logs a proposal the controller does not execute; an engineer reviews each within one business day. This builds confidence that the controller’s choices match human judgment.

Phase 3: single-topic execution

Enable live execution scoped to one non-critical topic with stable traffic via excluded_topics on everything else (or topics= to constrain the rebalance). Watch its latency, throughput, and lag. After three clean automated rebalances with no user impact, proceed.

Phase 4: broad rollout with a circuit breaker

Expand to all topics outside the exclusion list. Add a circuit breaker that globally disables automation if the rebalance error rate exceeds 10% or any operation raises producer timeouts above a 5% threshold. Require manual reset so a flapping loop cannot silently degrade the cluster.

Across every phase, keep a single-toggle kill switch—an env var, feature flag, or ConfigMap value—so you can stop automation mid-incident without redeploying.

Conclusion

Done correctly, this pipeline turns reassignment from a reactive chore into a quiet background property: Prometheus and Cruise Control’s balancedness score detect structural imbalance, a gated controller retrieves and validates a proposal, and Cruise Control executes it under a throttle you control. The non-negotiables are accuracy and conservatism—real metric names, the correct HTTP verbs (GET /proposals, POST /rebalance), an executor-idle gate, and an incremental rollout from alert-only to dry-run to a single topic.

Invest in observability so every decision is logged, metered, and dashboarded. When it works, it should be boring. When it does not, the runbooks and failure-mode tables here point the on-call engineer to a fast resolution. The same alerting-and-gating pattern extends naturally to other Cruise Control operations—broker decommissioning via POST /kafkacruisecontrol/remove_broker, self-healing, and rack-awareness fixes—once you have earned operational confidence with reassignment.