Automating Kafka Cluster Provisioning with Terraform and Confluent for Kubernetes

Manual Kafka provisioning drifts, recovers slowly, and resists reproduction. Terraform plus Confluent for Kubernetes (CFK) replaces that with a declarative, GitOps-friendly path covering the full cluster lifecycle on Kubernetes. This guide walks the architecture and the concrete implementation patterns—from the Kubernetes foundation up to topics and role bindings—and calls out the field-name, ordering, and drift gotchas that break real deployments.

Understanding the Automation Stack

The stack has two layers, and the split is deliberate. Terraform owns the long-lived cloud foundation: the Kubernetes cluster, networking, storage classes, and IAM. CFK, deployed via its Helm chart, runs inside that cluster and continuously reconciles the desired state of Kafka clusters, topics, and role bindings as Kubernetes custom resources. Terraform provisions once and rarely changes; the operator reconciles constantly.

Before writing code, align provisioning with capacity. Broker count, CPU and memory, and storage class drive throughput and latency directly. A thorough Capacity Planning for Kafka Clusters: Throughput and Storage exercise dictates the resource requests, limits, and dataVolumeCapacity in your CFK manifests—skip it and you ship clusters that are either starved or wastefully over-provisioned.

CFK extends the Kubernetes API with CRDs for Kafka, KafkaTopic, Connect, ConfluentRolebinding, and other Confluent Platform components (API group platform.confluent.io/v1beta1). The cluster becomes just another Kubernetes resource, managed with kubectl and—critically—Terraform’s Kubernetes provider, so the entire stack version-controls in one repository. The broader discipline of Cluster Architecture & Provisioning gives context for how this fits your overall strategy.

Provisioning the Kubernetes Foundation with Terraform

The first step is a Kubernetes cluster that can run stateful workloads. The example uses Google Kubernetes Engine (GKE), but the pattern adapts to EKS or AKS with minimal change. The essentials are a dedicated node pool with fast SSDs, sized machine types, and workload identity.

# main.tf
terraform {
  required_providers {
    google = {
      source  = "hashicorp/google"
      version = "~> 5.0"
    }
    kubernetes = {
      source  = "hashicorp/kubernetes"
      version = "~> 2.23"
    }
    helm = {
      source  = "hashicorp/helm"
      version = "~> 2.11"
    }
  }
}

resource "google_container_cluster" "kafka" {
  name     = "kafka-cluster"
  location = "us-central1-a"

  initial_node_count       = 1
  remove_default_node_pool = true

  workload_identity_config {
    workload_pool = "${var.project_id}.svc.id.goog"
  }
}

resource "google_container_node_pool" "kafka_brokers" {
  name       = "kafka-broker-pool"
  cluster    = google_container_cluster.kafka.name
  location   = google_container_cluster.kafka.location
  node_count = 3

  node_config {
    machine_type = "n2-standard-8"
    disk_type    = "pd-ssd"
    disk_size_gb = 500

    labels = {
      role = "kafka-broker"
    }

    taint {
      key    = "dedicated"
      value  = "kafka-broker"
      effect = "NO_SCHEDULE"
    }
  }
}

The taint keeps general workloads off these nodes; brokers tolerate it (shown later) so only they land here, which protects broker page cache and disk I/O from noisy neighbors. Workload identity grants cloud access without static credentials. Two sizing decisions matter most here and are hard to change later: the node pool is us-central1-a (single zone) for simplicity, but production should spread brokers across at least three zones so a zone failure never takes down a majority of replicas; and the boot/data disk is SSD, because Kafka’s sequential-write pattern still suffers badly from the IOPS ceilings and tail latency of standard persistent disks. For production, define custom storage classes matching your throughput targets; the Kubernetes CSI driver documentation covers storage-class parameters per cloud provider.

Deploying the Confluent for Kubernetes Operator

With the foundation in place, deploy the CFK operator through the Helm provider so the operator itself is managed as code and upgraded through the same pipeline.

provider "kubernetes" {
  host                   = "https://${google_container_cluster.kafka.endpoint}"
  token                  = data.google_client_config.default.access_token
  cluster_ca_certificate = base64decode(google_container_cluster.kafka.master_auth[0].cluster_ca_certificate)
}

provider "helm" {
  kubernetes {
    host                   = "https://${google_container_cluster.kafka.endpoint}"
    token                  = data.google_client_config.default.access_token
    cluster_ca_certificate = base64decode(google_container_cluster.kafka.master_auth[0].cluster_ca_certificate)
  }
}

resource "helm_release" "confluent_operator" {
  name             = "confluent-operator"
  repository       = "https://packages.confluent.io/helm"
  chart            = "confluent-for-kubernetes"
  namespace        = "confluent"
  create_namespace = true

  depends_on = [
    google_container_node_pool.kafka_brokers
  ]
}

Configure the kubernetes and helm providers so they resolve before any resource that talks to the API server. Terraform evaluates provider configuration during planning, not apply, so a provider that depends on a not-yet-created cluster endpoint can fail the plan outright; wiring both providers to the cluster endpoint and CA certificate, plus depends_on on the node pool, is the reliable pattern. CFK is licensed under the Confluent Community License for development and ships a free 30-day evaluation; you do not set a license key as a Helm value, so no such line appears here.

After the operator is running, verify its health before creating Kafka resources. The Helm release deploys a Deployment named confluent-operator:

kubectl logs -n confluent deployment/confluent-operator -f

A clean operator log here is the gate for everything downstream: if its admission webhook or CRD installation has not finished registering, the very next step—a kubernetes_manifest that reads the Kafka CRD schema—will fail to plan. The Confluent for Kubernetes documentation covers operator configuration in depth, including multi-zone deployments and network policies.

Defining Kafka Clusters as Kubernetes Custom Resources

The Kafka custom resource encapsulates the broker configuration. Define it in Terraform with kubernetes_manifest, parameterizing broker count, resources, and storage from your capacity-planning outcomes.

resource "kubernetes_manifest" "kafka_cluster" {
  manifest = {
    apiVersion = "platform.confluent.io/v1beta1"
    kind       = "Kafka"
    metadata = {
      name      = "kafka"
      namespace = "confluent"
    }
    spec = {
      replicas = 3
      image = {
        application = "confluentinc/cp-server:7.9.0"
        init        = "confluentinc/confluent-init-container:2.11.0"
      }
      dataVolumeCapacity = "200Gi"
      storageClass = {
        name = "premium-rwo"
      }
      metricReporter = {
        enabled = true
      }
      listeners = {
        internal = {
          authentication = {
            type = "plain"
            jaasConfig = {
              secretRef = "kafka-credentials"
            }
          }
          tls = {
            enabled = true
          }
        }
      }
      podTemplate = {
        resources = {
          requests = {
            cpu    = "2"
            memory = "8Gi"
          }
          limits = {
            cpu    = "4"
            memory = "16Gi"
          }
        }
        tolerations = [
          {
            key      = "dedicated"
            operator = "Equal"
            value    = "kafka-broker"
            effect   = "NoSchedule"
          }
        ]
      }
    }
  }

  depends_on = [helm_release.confluent_operator]
}

This defines a three-broker cluster with a TLS-enabled internal listener and explicit resource boundaries; the tolerations place pods on the dedicated node pool. Three field-level details decide whether it actually starts:

  • Init-container tag must match the operator version. Pin image.init to the tag for your CFK release (here CP 7.9 with init container 2.11). A mismatched init tag is a common cause of pods that never leave Init or crash-loop before the broker process starts.
  • dataVolumeCapacity is immutable. CFK provisions it as a PersistentVolumeClaim; growing it later means an explicit volume-expansion procedure, not a one-line edit, so size it from capacity planning rather than guessing low.
  • CFK derives broker defaults from replicas. With three replicas it sets sane default.replication.factor and min.insync.replicas unless you override them—so the per-topic min.insync.replicas: "2" shown later is a deliberate durability choice, not boilerplate.

A subtler trap is drift. Because CFK’s defaulting webhook and the broker controller write back many spec and status fields, kubernetes_manifest can surface a perpetual diff or Error: Provider produced inconsistent result after apply when the server returns values you did not set. The fix is to declare those server-owned paths in computed_fields (which already defaults to metadata.labels and metadata.annotations) and, where another controller legitimately owns a field, set field_manager with force_conflicts = false so Terraform fails loudly instead of fighting the operator on every plan.

Finally, kubernetes_manifest reads the target CRD’s schema from the cluster at plan time, so the CFK CRDs must already be installed—which the helm_release above does—before this resource can be planned. In a single terraform apply that means the manifest resources for Kafka, KafkaTopic, and ConfluentRolebinding are typically applied in a second phase (for example with -target) or in a separate root module that runs after the operator is healthy.

Automating Topic and Role Binding Management

Managing topics manually across environments is a classic source of drift. CFK provides the KafkaTopic and ConfluentRolebinding CRDs so you declare data topology and access controls alongside infrastructure.

The topic resource kind is KafkaTopic (not Topic):

resource "kubernetes_manifest" "orders_topic" {
  manifest = {
    apiVersion = "platform.confluent.io/v1beta1"
    kind       = "KafkaTopic"
    metadata = {
      name      = "orders"
      namespace = "confluent"
    }
    spec = {
      replicas       = 3
      partitionCount = 12
      configs = {
        "cleanup.policy"      = "compact"
        "min.insync.replicas" = "2"
        "retention.ms"        = "604800000"
      }
      kafkaClusterRef = {
        name = "kafka"
      }
    }
  }

  depends_on = [kubernetes_manifest.kafka_cluster]
}

Two of these fields are one-way doors worth pausing on. partitionCount can be increased but never decreased, and for a compact or keyed topic a partition change re-routes keys to new partitions and breaks ordering guarantees—so partition count is a capacity-planning decision, not a knob to tune in review. The replicas: 3 with min.insync.replicas: "2" pairing is the standard durability floor: it tolerates one broker down while still acknowledging writes, where replicas: 3, minISR: 3 would block all producers the moment a single broker restarts.

For RBAC, the ConfluentRolebinding resource has two field shapes worth getting right. spec.principal.type is lowercase (user or group), and spec.resourcePatterns is a list of objects—each with name, resourceType, and patternType—not a list of "Type:name" strings. The cluster reference for a role binding is kafkaRestClassRef, which points at the KafkaRestClass CR used to reach the MDS/REST endpoint:

resource "kubernetes_manifest" "producer_rolebinding" {
  manifest = {
    apiVersion = "platform.confluent.io/v1beta1"
    kind       = "ConfluentRolebinding"
    metadata = {
      name      = "orders-producer-binding"
      namespace = "confluent"
    }
    spec = {
      principal = {
        type = "user"
        name = "orders-service"
      }
      role = "ResourceOwner"
      resourcePatterns = [
        {
          name        = "orders"
          resourceType = "Topic"
          patternType = "LITERAL"
        }
      ]
      kafkaRestClassRef = {
        name = "default"
      }
    }
  }

  depends_on = [kubernetes_manifest.kafka_cluster]
}

ResourceOwner is broad—it grants read, write, and administrative rights on the matched resource. For a service that only produces, prefer DeveloperWrite scoped to the topic and reserve ResourceOwner for the team that owns the topic’s schema and config. Note the role binding depends on the cluster but reaches Kafka over MDS, so it only reconciles once the REST/MDS endpoint is reachable—if a binding sits unreconciled, check the KafkaRestClass and MDS health before suspecting the binding itself. With this in version control, changes to partitions, retention, or access controls go through pull-request review before they apply. The Apache Kafka topic configuration reference lists every topic-level setting, and the CFK RBAC documentation covers the role-binding fields and valid resourceType values (Topic, Group, Subject, Cluster, TransactionalId, KsqlCluster).

Implementing GitOps with Terraform Cloud or Atlantis

Infrastructure as code compounds in value when paired with a GitOps pipeline. Terraform Cloud, Atlantis, or custom CI/CD can plan and apply on merge to main:

# atlantis.yaml
version: 3
projects:
  - name: kafka-infrastructure
    dir: terraform/kafka
    workspace: production
    autoplan:
      when_modified: ["*.tf", "*.tfvars"]
      enabled: true
    apply_requirements: [approved, mergeable]

For GitHub Actions, a workflow can run Terraform directly:

# .github/workflows/terraform-apply.yml
name: "Terraform Apply"
on:
  push:
    branches:
      - main
    paths:
      - 'terraform/kafka/**'
jobs:
  terraform:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: hashicorp/setup-terraform@v3
        with:
          terraform_version: 1.7.0
      - name: Terraform Apply
        run: |
          terraform -chdir=terraform/kafka init
          terraform -chdir=terraform/kafka apply -auto-approve
        env:
          GOOGLE_CREDENTIALS: ${{ secrets.GCP_SA_KEY }}

Every change is versioned, reviewed, and applied consistently, and rollbacks reduce to a git revert plus a pipeline run. One caveat the plan-time CRD behavior forces on this design: a fresh cluster cannot be stood up in a single monolithic run. Split it into a foundation stage (cluster + operator + CRDs) and a resources stage (Kafka, KafkaTopic, ConfluentRolebinding) that runs only after the operator is healthy. The cheap workaround—-target on the operator first—works for a one-off bootstrap but is fragile in a pipeline because it silently skips dependency checks; separate root modules (or separate Atlantis projects) make the ordering explicit and reviewable. Note this only affects the first apply; once the CRDs exist, day-two changes plan in a single run.

Monitoring and Validating Provisioned Clusters

Provisioning is not done until you have proven the cluster matches its spec. After Terraform completes, run a post-apply check on broker readiness, topic configuration, and end-to-end message flow.

The CFK Kafka CR exposes readiness through status.phase, which reads RUNNING once brokers are up; there is no built-in Ready condition to wait on, so poll the phase instead:

#!/usr/bin/env bash
# validate-kafka.sh
set -euo pipefail

NAMESPACE="confluent"
CLUSTER="kafka"

echo "Waiting for Kafka cluster to reach RUNNING..."
for _ in $(seq 1 60); do
  phase=$(kubectl get kafka "${CLUSTER}" -n "${NAMESPACE}" \
    -o jsonpath='{.status.phase}' 2>/dev/null || true)
  echo "  status.phase=${phase:-<none>}"
  [ "${phase}" = "RUNNING" ] && break
  sleep 10
done
[ "${phase:-}" = "RUNNING" ] || { echo "ERROR: Kafka did not reach RUNNING"; exit 1; }

echo "Verifying topic configuration..."
TOPIC_COUNT=$(kubectl get kafkatopic -n "${NAMESPACE}" --no-headers | wc -l)
if [ "${TOPIC_COUNT}" -lt 1 ]; then
  echo "ERROR: No KafkaTopic resources found"
  exit 1
fi

echo "Testing producer and consumer connectivity..."
kubectl exec -n "${NAMESPACE}" "${CLUSTER}-0" -- \
  kafka-console-producer --bootstrap-server localhost:9092 \
  --topic health-check <<< "ping"

kubectl exec -n "${NAMESPACE}" "${CLUSTER}-0" -- \
  kafka-console-consumer --bootstrap-server localhost:9092 \
  --topic health-check --from-beginning --max-messages 1 --timeout-ms 10000

echo "Validation complete."

A few things make this check trustworthy rather than theatrical. The phase loop caps at roughly ten minutes (60 × 10s)—if a cluster routinely needs longer, that is a signal about image-pull or PVC-provisioning latency, not a reason to raise the bound blindly. The cp-server image ships kafka-console-producer and kafka-console-consumer, both taking --bootstrap-server (the --broker-list alias is deprecated); --timeout-ms bounds the consumer’s wait and --max-messages 1 makes it exit after the first record, so the script never hangs in CI. If the listener enforces TLS or SASL—as the manifest above does—the plaintext localhost:9092 path will fail, and you must pass --producer.config/--consumer.config with matching client properties.

Wire this as a post-apply hook: on failure, alert on-call and halt further deployments. For ongoing observability, enable the cluster’s metricReporter and scrape broker JMX metrics into Prometheus, watching the few signals that actually predict incidents—under-replicated partitions (steady-state should be zero), consumer lag against your SLA, and disk utilization against the headroom set during capacity planning. A non-zero under-replicated-partition count that does not clear within a rolling restart window is the earliest reliable warning that a broker or its volume is failing.

Done this way, Kafka provisioning becomes repeatable and auditable: the same review velocity application teams have long enjoyed, with the durability and access controls a production streaming platform demands.

Back to Capacity Planning For Kafka Clusters Throughput And Storage