Automated Provisioning of Kafka Clusters with Terraform and Ansible

Provisioning Apache Kafka clusters at scale means treating infrastructure as code to eliminate drift and manual configuration errors. Terraform declares the underlying compute, networking, and storage; Ansible configures the operating system, installs Kafka, and renders broker configuration. That split — cloud resources versus application configuration — is the natural seam to automate against.

The metadata management mode shapes everything downstream. As of Apache Kafka 4.0 (March 2025), KRaft is the only supported mode and ZooKeeper has been removed; clusters must be migrated to KRaft before upgrading to 4.0. New automation should therefore target KRaft, which removes the external ZooKeeper dependency but adds controller-quorum planning. ZooKeeper sequencing still matters only if you operate a pre-4.0 cluster. The trade-offs are covered in KRaft vs ZooKeeper: Choosing the Right Metadata Management, and that choice drives your Terraform resources, Ansible inventory, and bootstrap playbook.

This article gives a complete, doc-grounded framework for automating Kafka provisioning: infrastructure-as-code patterns for compute, networking, and storage; Ansible role design for broker configuration; secrets integration; and validation that confirms cluster readiness before production traffic. The broader Cluster Architecture & Provisioning section provides the context behind these decisions.

KRaft Topology: The Decision That Sizes Everything Else

Before any Terraform runs, decide your controller topology, because it sets your instance count, your security-group rules, and your inventory groups. KRaft nodes take a process.roles value of controller, broker, or broker,controller (combined). The controller quorum tolerates failures on a 2N+1 basis: 3 controllers survive 1 loss, 5 survive 2. More than 5 is rarely worth it — the Raft log replicates to every voter, so each added controller raises commit-latency cost without buying proportional safety.

The practical rule: run a fixed 3- or 5-node dedicated controller quorum in production and scale brokers freely against it, so growing throughput never changes your metadata-consensus latency. Combined mode is fine for a single test node or a cost-constrained dev cluster. This article’s inventory and templates support both; the only difference is which process.roles value a host inherits.

Infrastructure as Code Foundation with Terraform

Terraform provisions what brokers need: compute instances, network attachments, security groups, and persistent storage. Ansible then configures the OS and the Kafka application on those resources.

Compute Resource Declarations

The compute layer reflects capacity-planning decisions. Settle on broker instance types, storage profiles, and network throughput before writing modules; the Broker Sizing and Hardware Selection for Kafka Clusters guide provides the analytical framework. A broker module accepts variables for instance count, type, AMI, and subnet placement:

# variables.tf
variable "cluster_name" {
  description = "Logical name for the Kafka cluster"
  type        = string
}

variable "broker_count" {
  description = "Number of broker nodes"
  type        = number
  default     = 3
}

variable "instance_type" {
  description = "EC2 instance type for brokers"
  type        = string
  default     = "i3en.2xlarge"
}

variable "subnet_ids" {
  description = "List of subnet IDs for broker placement"
  type        = list(string)
}

variable "data_volume_size_gb" {
  description = "Size of broker data volume in GB"
  type        = number
  default     = 2500
}

variable "availability_zones" {
  description = "List of availability zones"
  type        = list(string)
}

Instance selection encodes hardware sizing. For sustained throughput above roughly 100 MB/s per broker, instances with local NVMe such as the i3en or i4i families give the low-latency disk I/O that Kafka’s log segments and page-cache flushes depend on. Note that instance-store NVMe is ephemeral — it does not survive a stop/start — so it is appropriate only where replication factor and rack awareness protect against single-node data loss; otherwise use EBS, as shown below. In KRaft, controller-node count drives quorum tolerance: an odd number (typically 3 or 5) tolerates 1 or 2 controller failures respectively.

Network and Security Group Design

Each broker exposes listeners for client connections and inter-broker traffic; KRaft adds a dedicated controller listener. The security group must allow inter-broker and controller traffic between brokers/controllers and expose the client listener to authorized subnets.

# security_groups.tf
resource "aws_security_group" "kafka_broker" {
  name        = "${var.cluster_name}-broker-sg"
  description = "Security group for Kafka brokers"
  vpc_id      = var.vpc_id

  ingress {
    description = "Inter-broker communication"
    from_port   = 9092
    to_port     = 9092
    protocol    = "tcp"
    self        = true
  }

  ingress {
    description = "KRaft controller quorum"
    from_port   = 9093
    to_port     = 9093
    protocol    = "tcp"
    self        = true
  }

  ingress {
    description = "Client access from trusted subnets"
    from_port   = 9094
    to_port     = 9094
    protocol    = "tcp"
    cidr_blocks = var.client_cidr_blocks
  }

  ingress {
    description = "Prometheus JMX exporter scrape"
    from_port   = 7071
    to_port     = 7071
    protocol    = "tcp"
    cidr_blocks = var.monitoring_cidr_blocks
  }

  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }

  tags = {
    Name      = "${var.cluster_name}-broker-sg"
    Cluster   = var.cluster_name
    ManagedBy = "terraform"
  }
}

The self = true rule restricts inter-broker (9092) and controller-quorum (9093) traffic to members of the same security group. Client traffic on 9094 is scoped to known CIDRs. Port 7071 is the conventional Prometheus JMX exporter agent port; pick whatever your exporter binds to and keep the monitoring CIDR tight so the metrics endpoint is not publicly reachable.

A common first-boot failure traces directly to this group: if the self = true rule on 9093 is missing or the controller listener is bound to the wrong interface, brokers start but never join — kafka-metadata-quorum.sh describe --status (below) shows them stuck as observers and the active controller never appears. Treat the 9093 rule as load-bearing, not optional.

Persistent Storage Configuration

Brokers are stateful: each holds segment files for every partition replica it leads or follows, and losing a data volume forces re-replication from surviving replicas. Provision block storage that survives instance termination, sized for throughput.

# storage.tf
resource "aws_ebs_volume" "kafka_data" {
  count             = var.broker_count
  availability_zone = element(var.availability_zones, count.index)
  size              = var.data_volume_size_gb
  type              = "gp3"
  iops              = 16000
  throughput        = 1000

  tags = {
    Name      = "${var.cluster_name}-broker-${count.index}-data"
    Cluster   = var.cluster_name
    ManagedBy = "terraform"
  }
}

resource "aws_volume_attachment" "kafka_data_attach" {
  count       = var.broker_count
  device_name = "/dev/sdf"
  volume_id   = aws_ebs_volume.kafka_data[count.index].id
  instance_id = aws_instance.kafka_broker[count.index].id
}

gp3 ships with a baseline of 3,000 IOPS and 125 MB/s, and lets you provision IOPS (up to 16,000) and throughput (up to 1,000 MB/s) independently of capacity — unlike gp2, where both scale with size. Set these from measured workload metrics rather than guesswork.

Ansible Inventory and Dynamic Host Generation

Terraform creates resources; Ansible configures them. The inventory bridges the two. Generate it from Terraform state so Ansible always acts on exactly what Terraform created, rather than a hand-maintained file that drifts.

Terraform Outputs as Inventory Source

After terraform apply, output broker (and, for legacy ZooKeeper clusters, ensemble) addresses in a structured shape Ansible can consume:

# outputs.tf
output "kafka_brokers" {
  value = {
    for idx, instance in aws_instance.kafka_broker :
    "broker-${idx}" => {
      private_ip = instance.private_ip
      public_ip  = instance.public_ip
      az         = instance.availability_zone
      rack_id    = instance.availability_zone
    }
  }
  description = "Kafka broker connection details"
}

A wrapper can run terraform output -json and feed it to an inventory plugin. The cloud.terraform.terraform_provider inventory plugin reads resources directly from state, and the Ansible dynamic inventory guide covers writing a custom inventory script when the built-in plugins do not fit.

Inventory Structure for Multi-AZ Deployments

Group hosts by rack — which maps to availability zone in the cloud — so Ansible can render broker.rack for rack-aware replica placement. Separate KRaft controllers from brokers so you can target each role:

# inventory/kafka.yml
all:
  children:
    kafka_cluster:
      children:
        controllers:
          hosts:
            controller-0: { ansible_host: 10.0.1.10, rack_id: us-east-1a, node_id: 1 }
            controller-1: { ansible_host: 10.0.2.10, rack_id: us-east-1b, node_id: 2 }
            controller-2: { ansible_host: 10.0.3.10, rack_id: us-east-1c, node_id: 3 }
        brokers:
          hosts:
            broker-0: { ansible_host: 10.0.1.20, rack_id: us-east-1a, node_id: 101 }
            broker-1: { ansible_host: 10.0.2.20, rack_id: us-east-1b, node_id: 102 }
            broker-2: { ansible_host: 10.0.3.20, rack_id: us-east-1c, node_id: 103 }
            broker-3: { ansible_host: 10.0.1.21, rack_id: us-east-1a, node_id: 104 }
            broker-4: { ansible_host: 10.0.2.21, rack_id: us-east-1b, node_id: 105 }
            broker-5: { ansible_host: 10.0.3.21, rack_id: us-east-1c, node_id: 106 }

This lets you run tasks against only brokers, only controllers, or the whole cluster. rack_id feeds broker.rack; node_id becomes the KRaft node.id and must be unique and stable across the cluster. Keeping controller IDs (1–3) in a separate numeric range from broker IDs (101+) is a deliberate convention — it makes a node’s role obvious in --describe and quorum output and prevents an accidental ID collision when you later scale brokers.

Ansible Role Design for Kafka Broker Configuration

The broker role handles OS tuning, the Java runtime, Kafka deployment, and configuration rendering. Factor these into distinct task files with clear ordering.

OS Tuning for Kafka Workloads

Kafka benefits from kernel tuning for network throughput and virtual-memory behavior:

# tasks/os-tuning.yml
- name: Configure sysctl parameters for Kafka
  ansible.posix.sysctl:
    name: "{{ item.name }}"
    value: "{{ item.value }}"
    state: present
    reload: true
  loop:
    - { name: "net.core.rmem_max", value: "134217728" }
    - { name: "net.core.wmem_max", value: "134217728" }
    - { name: "net.core.rmem_default", value: "134217728" }
    - { name: "net.core.wmem_default", value: "134217728" }
    - { name: "vm.swappiness", value: "1" }
    - { name: "vm.dirty_ratio", value: "60" }
    - { name: "vm.dirty_background_ratio", value: "5" }

- name: Increase file descriptor and process limits
  community.general.pam_limits:
    domain: kafka
    limit_type: "{{ item.limit_type }}"
    limit_item: "{{ item.limit_item }}"
    value: "{{ item.value }}"
  loop:
    - { limit_type: "soft", limit_item: "nofile", value: "100000" }
    - { limit_type: "hard", limit_item: "nofile", value: "100000" }
    - { limit_type: "soft", limit_item: "nproc",  value: "32768" }
    - { limit_type: "hard", limit_item: "nproc",  value: "32768" }

A note on FQCNs: sysctl lives in ansible.posix, not ansible.builtin — the bare-name and ansible.builtin.sysctl forms both resolve to the POSIX collection’s module, so use ansible.posix.sysctl explicitly. pam_limits is community.general.pam_limits.

The larger socket buffers serve Kafka’s high-throughput TCP connections. vm.swappiness=1 is preferred over 0: on some kernels 0 disables swap so aggressively that it invites the OOM killer under pressure. For the dirty-page ratios, Kafka relies heavily on the OS page cache flushing log segments asynchronously, so allowing a generous amount of dirty page cache (the Kafka docs reference values like vm.dirty_ratio in the 60–80 range with a low vm.dirty_background_ratio) lets writeback batch efficiently; tune to your disk throughput. The high file-descriptor limit is not optional headroom: a broker holds an open descriptor per log segment plus one per live socket, so a busy broker with thousands of partitions routinely exceeds the default 1024 limit and crashes with Too many open files — nofile: 100000 is sized to avoid that. These align with the Apache Kafka operations documentation.

Kafka Configuration Template

A Jinja2 template renders server.properties from inventory and Terraform-derived values. This template targets KRaft and falls back to a ZooKeeper connection only for legacy 3.x clusters:

{# templates/server.properties.j2 #}
{% if process_roles is defined %}
# --- KRaft mode ---
process.roles={{ process_roles }}
node.id={{ node_id }}
controller.quorum.voters={{ controller_quorum_voters }}
controller.listener.names=CONTROLLER
{% else %}
# --- Legacy ZooKeeper mode (pre-4.0) ---
broker.id={{ node_id }}
zookeeper.connect={{ zookeeper_connect }}
{% endif %}

{% if 'broker' in (process_roles | default('broker')) %}
listeners=INTERNAL://{{ ansible_host }}:9092,CLIENT://{{ ansible_host }}:9094{% if 'controller' in (process_roles | default('')) %},CONTROLLER://{{ ansible_host }}:9093{% endif %}

advertised.listeners=INTERNAL://{{ ansible_host }}:9092,CLIENT://{{ public_ip }}:9094
inter.broker.listener.name=INTERNAL
{% else %}
listeners=CONTROLLER://{{ ansible_host }}:9093
{% endif %}
listener.security.protocol.map=INTERNAL:PLAINTEXT,CLIENT:SASL_SSL,CONTROLLER:PLAINTEXT

num.network.threads={{ num_network_threads | default(8) }}
num.io.threads={{ num_io_threads | default(16) }}
socket.send.buffer.bytes=1048576
socket.receive.buffer.bytes=1048576
socket.request.max.bytes=104857600

log.dirs={{ data_mount_point }}/kafka-data
num.partitions={{ default_partition_count | default(12) }}
default.replication.factor={{ default_replication_factor | default(3) }}
min.insync.replicas={{ min_insync_replicas | default(2) }}

{% if rack_id is defined %}
broker.rack={{ rack_id }}
{% endif %}

auto.create.topics.enable=false
delete.topic.enable=true
log.retention.hours={{ log_retention_hours | default(168) }}
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000

offsets.topic.replication.factor=3
transaction.state.log.replication.factor=3
transaction.state.log.min.isr=2

Two correctness points the bare-bones version misses. First, KRaft uses node.id and process.roles, not broker.id; even a broker-only node must define controller.listener.names and include CONTROLLER in listener.security.protocol.map so it can reach the controller quorum. Second, the controller listener must not reuse the inter-broker listener name. Internal traffic stays plaintext inside the security-group boundary while the client listener enforces SASL_SSL. In production, secure the controller listener too (for example map CONTROLLER:SSL). For a single combined-role test node, process.roles=broker,controller is allowed but is not recommended for production.

The min.insync.replicas=2 paired with default.replication.factor=3 is the durability default worth making explicit: a producer using acks=all will keep accepting writes while any one of the three replicas is down, but a second failure stalls those partitions instead of silently losing acknowledged data. Setting min.insync.replicas equal to the replication factor would make any single replica loss block writes; setting it to 1 would let acks=all acknowledge writes that one disk failure can erase. Two-of-three is the standard balance, and rendering it from inventory keeps it consistent across every broker.

Secrets Management Integration

Brokers need credentials for SASL and SSL keystores. Pull them from a secrets manager instead of hardcoding:

# tasks/secrets.yml
- name: Retrieve Kafka broker credentials from Vault
  community.hashi_vault.vault_kv2_get:
    url: "{{ vault_addr }}"
    path: "kafka/{{ cluster_name }}/broker-credentials"
    auth_method: approle
    role_id: "{{ vault_role_id }}"
    secret_id: "{{ vault_secret_id }}"
  register: kafka_secrets
  no_log: true

- name: Write JAAS configuration for SASL
  ansible.builtin.template:
    src: kafka_server_jaas.conf.j2
    dest: /etc/kafka/kafka_server_jaas.conf
    owner: kafka
    group: kafka
    mode: "0600"
  no_log: true
  notify: restart kafka

- name: Set Kafka environment variables for JAAS
  ansible.builtin.lineinfile:
    path: /etc/default/kafka
    line: 'export KAFKA_OPTS="-Djava.security.auth.login.config=/etc/kafka/kafka_server_jaas.conf"'
    create: true
    owner: kafka
    group: kafka

no_log: true keeps secret values out of task output. The JAAS file is written with 0600 and only restarts the broker when it changes. For SSL, pull the PKCS12 keystore from the secrets manager and write it to the path referenced by ssl.keystore.location.

Bootstrap Sequencing and Cluster Initialization

KRaft Cluster Formation

A KRaft cluster needs a shared cluster ID and a formatted metadata log before first start:

# Generate the cluster ID once, on the first controller
KAFKA_CLUSTER_ID=$(kafka-storage.sh random-uuid)

# Format storage on every node (controllers and brokers) with that ID
kafka-storage.sh format \
  --cluster-id "$KAFKA_CLUSTER_ID" \
  --config /etc/kafka/server.properties \
  --ignore-formatted

random-uuid and format --ignore-formatted are real kafka-storage.sh subcommands/flags; --ignore-formatted makes the format step idempotent by skipping nodes that already have a meta.properties. The single most common bootstrap failure is a split cluster ID: if each node generates its own UUID instead of sharing one, formatting succeeds everywhere but the nodes refuse to form a quorum because their metadata logs belong to different clusters. Generate the ID once, set it as an Ansible fact on the first controller, and consume it on every other node so the whole cluster shares one ID. Then start controllers first, followed by brokers:

# playbooks/bootstrap-kafka.yml
- name: Format and start KRaft controllers
  hosts: controllers
  serial: 1
  tasks:
    - name: Start controller service
      ansible.builtin.systemd:
        name: kafka
        state: started
        enabled: true

    - name: Wait for controller listener
      ansible.builtin.wait_for:
        port: 9093
        timeout: 60

- name: Start Kafka brokers
  hosts: brokers
  serial: 1
  tasks:
    - name: Start Kafka broker
      ansible.builtin.systemd:
        name: kafka
        state: started
        enabled: true

    - name: Wait for broker to report leadership metrics
      ansible.builtin.uri:
        url: "http://{{ ansible_host }}:7071/metrics"
        return_content: true
      register: metrics
      until: "'kafka_server_replicamanager_leadercount' in metrics.content"
      retries: 60
      delay: 5

serial: 1 starts nodes one at a time. The readiness check scrapes the Prometheus JMX exporter and waits for kafka_server_replicamanager_leadercount to appear — the lowercased, mangled form the exporter produces from the kafka.server:type=ReplicaManager,name=LeaderCount MBean. Confirm the exact string against your exporter’s rendered /metrics output, since the name depends on your jmx_exporter rules.

Confirming the Quorum Formed

A listening port on 9093 proves a process is up, not that the quorum elected a leader. Kafka ships an authoritative check for that: kafka-metadata-quorum.sh describe --status queries the cluster’s own view of its Raft quorum.

kafka-metadata-quorum.sh \
  --bootstrap-server "${BROKER_ENDPOINT}:9094" \
  --command-config /etc/kafka/client.properties \
  describe --status

The output reports LeaderId (the active controller), CurrentVoters (every controller in the quorum), and MaxFollowerLag/MaxFollowerLagTimeMs (how far behind the trailing controller is). Gate the bootstrap on this rather than on a port check: a healthy quorum shows a non-negative LeaderId, a CurrentVoters list whose length equals your controller count, and a small MaxFollowerLag. A LeaderId of -1, a short voter list, or a lag that keeps climbing each poll all signal a quorum that has not converged — usually a split cluster ID or a blocked 9093 path, exactly the two failure modes above.

Legacy ZooKeeper Sequencing (pre-4.0 only)

If you still operate a 3.x cluster on ZooKeeper, the ensemble must reach quorum before brokers register:

- name: Verify ZooKeeper quorum
  ansible.builtin.shell: echo srvr | nc localhost 2181 | grep Mode
  register: zk_mode
  until: "'leader' in zk_mode.stdout or 'follower' in zk_mode.stdout"
  retries: 30
  delay: 2
  changed_when: false

The srvr four-letter word reports Mode: leader or Mode: follower once a node has taken its role. Note that four-letter words must be allowlisted via 4lw.commands.whitelist (e.g. srvr) in zoo.cfg, or the command returns nothing. Plan to migrate any such cluster to KRaft before upgrading to 4.0.

Validation Testing and Readiness Verification

A cluster is not production-ready until it passes checks for replication health, controller state, and client connectivity. Make these the final playbook phase. The table below maps each check to the failure it catches and the symptom that tells you it fired:

Topic Creation and Replication Verification

Create a test topic at production replication factor and confirm full ISR:

# tasks/validate-cluster.yml
- name: Create validation topic
  ansible.builtin.shell: |
    kafka-topics.sh --bootstrap-server {{ groups['brokers'][0] }}:9094 \
      --create --topic cluster-validation \
      --partitions 12 --replication-factor 3 \
      --command-config /etc/kafka/client.properties
  register: topic_create
  changed_when: "'Created topic' in topic_create.stdout"
  failed_when:
    - topic_create.rc != 0
    - "'already exists' not in topic_create.stderr"

- name: Describe validation topic
  ansible.builtin.shell: |
    kafka-topics.sh --bootstrap-server {{ groups['brokers'][0] }}:9094 \
      --describe --topic cluster-validation \
      --command-config /etc/kafka/client.properties
  register: topic_desc
  changed_when: false

- name: Assert every partition reports full ISR
  ansible.builtin.assert:
    that:
      - "topic_desc.stdout.count('Isr:') == 12"
      - "'Isr: \t' not in topic_desc.stdout"
    fail_msg: "Not all partitions have full in-sync replicas"

--command-config passes the admin-client properties file (security settings) to kafka-topics.sh, alongside --bootstrap-server. Asserting on a fixed Isr: 3 substring is brittle because kafka-topics.sh --describe prints replica IDs, not a count — count the Isr: lines (one per partition) and verify none are empty instead.

End-to-End Produce and Consume Test

Confirm the cluster actually accepts and delivers messages:

# Produce one test message
echo "validation-$(date +%s)" | kafka-console-producer.sh \
  --bootstrap-server "${BROKER_ENDPOINT}:9094" \
  --topic cluster-validation \
  --producer.config /etc/kafka/client.properties

# Consume it back
kafka-console-consumer.sh \
  --bootstrap-server "${BROKER_ENDPOINT}:9094" \
  --topic cluster-validation \
  --from-beginning --max-messages 1 \
  --consumer.config /etc/kafka/client.properties \
  --timeout-ms 10000

--producer.config and --consumer.config are the correct flags for the console tools (the admin tools instead use --command-config). Wrap this in a task that fails the run if no message round-trips within the timeout. This is the check that catches listener and auth misconfiguration that the metadata checks miss: if advertised.listeners points at an address clients cannot reach, or the SASL credentials are wrong, the topic exists and the quorum is healthy but the consumer still times out.

Metrics-Based Health Checks

Confirm exactly one active controller:

- name: Scrape broker metrics
  ansible.builtin.uri:
    url: "http://{{ ansible_host }}:7071/metrics"
    return_content: true
  register: broker_metrics

- name: Assert exactly one active controller
  ansible.builtin.assert:
    that:
      - "'kafka_controller_kafkacontroller_activecontrollercount 1.0' in broker_metrics.content"
    fail_msg: "Expected exactly one active controller"

ActiveControllerCount should sum to exactly 1 across the cluster: 0 means no controller, more than 1 means split brain. As with the leader-count metric, verify the exact exporter-rendered string for your jmx_exporter configuration.

Operationalizing the Provisioning Pipeline

Pipeline Structure and State Management

The pipeline runs: terraform plan, terraform apply, inventory generation, Ansible playbooks, validation. Keep Terraform state remote with locking — in S3, native S3 state locking (Terraform 1.10+) replaces the older DynamoDB lock table:

# backend.tf
terraform {
  backend "s3" {
    bucket       = "kafka-terraform-state"
    key          = "clusters/production/terraform.tfstate"
    region       = "us-east-1"
    use_lockfile = true
    encrypt      = true
  }
}

If you are on Terraform earlier than 1.10, use dynamodb_table = "terraform-locks" instead of use_lockfile. Version Ansible alongside the Terraform modules and trigger CI on changes to either. A Makefile gives one entry point:

# Makefile
.PHONY: provision validate destroy

provision:
	cd terraform && terraform init && terraform apply -auto-approve
	cd terraform && terraform output -json > ../ansible/inventory/terraform.json
	cd ansible && ansible-playbook -i inventory/kafka.yml playbooks/bootstrap-kafka.yml
	cd ansible && ansible-playbook -i inventory/kafka.yml playbooks/validate-cluster.yml

validate:
	cd ansible && ansible-playbook -i inventory/kafka.yml playbooks/validate-cluster.yml

destroy:
	cd ansible && ansible-playbook -i inventory/kafka.yml playbooks/decommission-brokers.yml || true
	cd terraform && terraform destroy -auto-approve

Handling Broker Decommissioning

Removing a broker safely means reassigning its partitions away first, then stopping the process, then letting Terraform destroy the instance. The ordering is the whole point: if you let Terraform destroy the instance first, the broker’s partitions go offline and any that had only min.insync.replicas copies block producers until they re-replicate. Drain, then destroy — never the reverse.

# playbooks/decommission-brokers.yml
- name: Drain and decommission brokers
  hosts: localhost
  vars:
    brokers_to_remove: "{{ target_broker_ids }}"
  tasks:
    - name: Generate reassignment plan
      ansible.builtin.shell: |
        kafka-reassign-partitions.sh --bootstrap-server {{ bootstrap_server }}:9094 \
          --topics-to-move-json-file /tmp/topics.json \
          --broker-list "{{ all_broker_ids | difference(brokers_to_remove) | join(',') }}" \
          --generate \
          --command-config /etc/kafka/client.properties
      register: reassignment_plan
      changed_when: false

    - name: Persist proposed assignment
      ansible.builtin.copy:
        content: "{{ reassignment_plan.stdout_lines[-1] }}"
        dest: /tmp/reassignment.json

    - name: Execute reassignment
      ansible.builtin.shell: |
        kafka-reassign-partitions.sh --bootstrap-server {{ bootstrap_server }}:9094 \
          --reassignment-json-file /tmp/reassignment.json \
          --execute \
          --command-config /etc/kafka/client.properties

    - name: Wait for reassignment completion
      ansible.builtin.shell: |
        kafka-reassign-partitions.sh --bootstrap-server {{ bootstrap_server }}:9094 \
          --reassignment-json-file /tmp/reassignment.json \
          --verify \
          --command-config /etc/kafka/client.properties
      register: verify_result
      until: "'is still in progress' not in verify_result.stdout"
      retries: 120
      delay: 10
      changed_when: false

With --generate, the tool reads --topics-to-move-json-file (the topics to relocate) plus --broker-list (the surviving target brokers) and prints a proposed assignment; capture the last line, persist it, then --execute and poll --verify against the same file. This drains data before the instance is destroyed, so partitions stay available throughout. On a large broker the reassignment can move terabytes and run for hours, which is why the poll is generous (120 retries × 10 s); throttle it with --throttle <bytes/s> on the --execute step if the move competes with production traffic.

Conclusion

Automating Kafka provisioning with Terraform and Ansible turns cluster deployment into a repeatable, reviewable process. The pattern is consistent: Terraform owns cloud resources, Ansible owns OS and application configuration, the inventory is generated from Terraform state, configuration is templated per role, secrets come from a manager, and validation gates production traffic.

The two details that most often break automation here are KRaft configuration correctness — node.id/process.roles, a dedicated controller listener, and a shared cluster ID formatted with kafka-storage.sh — and validation that asserts on real cluster state (kafka-metadata-quorum.sh describe --status, ISR line counts, a produce/consume round-trip) rather than assumed strings or bare port checks. Get those right, target KRaft (mandatory from Kafka 4.0), and one module plus one role can stand up consistent clusters across many environments. Pair this with the KRaft vs ZooKeeper and broker sizing guidance for the decisions this automation encodes.