Deploying Kafka Connect on Kubernetes with Strimzi

Apache Kafka Connect is the standard framework for streaming data between Kafka and external systems, but running it reliably in production means managing worker clusters, plugin images, credentials, and rolling restarts. Strimzi handles this declaratively by extending the Kubernetes API with custom resources that the Cluster Operator reconciles. This guide walks through a production deployment of Kafka Connect on Kubernetes with Strimzi: the operator model, the KafkaConnect resource, TLS and SASL authentication, declarative connector management, building plugin images, and monitoring with Prometheus.

Understanding the Strimzi operator model

Strimzi uses the operator pattern. The Cluster Operator watches for custom resources such as KafkaConnect and reconciles the desired state against the running cluster. When you create a KafkaConnect resource, the operator provisions a Deployment, configures the pods with the required environment variables, secrets, and volumes, and performs rolling updates when the spec changes.

This implements Kafka Connect’s distributed deployment mode natively on Kubernetes. Workers connect to the Kafka cluster through its bootstrap address and form a Connect group; the operator exposes the REST API through a Kubernetes Service. Connect integrates with the Schema Registry & Connect Ecosystem by setting converter and schema registry URLs directly in the Connect configuration.

Before deploying Connect, install the Strimzi Cluster Operator. It can watch a single namespace, several namespaces, or the whole cluster, depending on your multi-tenancy needs. Create the namespace first, then apply the install files. The namespace query parameter rewrites the role bindings to match the target namespace, so it must equal the -n value:

kubectl create namespace kafka-connect
kubectl apply -f 'https://strimzi.io/install/latest?namespace=kafka-connect' -n kafka-connect

Quote the URL so the shell does not interpret the ?. Verify the operator is running before proceeding:

kubectl get pods -n kafka-connect -l name=strimzi-cluster-operator

Defining the KafkaConnect custom resource

The KafkaConnect resource defines the desired state of the Connect cluster: worker replica count, the Kafka bootstrap address, internal topic settings, converters, resource limits, and (optionally) a plugin build. The example below connects to a Strimzi-managed Kafka cluster over TLS with mutual TLS authentication:

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnect
metadata:
  name: production-connect-cluster
  namespace: kafka-connect
  annotations:
    strimzi.io/use-connector-resources: "true"
spec:
  version: 3.9.0
  replicas: 3
  bootstrapServers: production-kafka-bootstrap.kafka.svc:9093
  tls:
    trustedCertificates:
      - secretName: production-cluster-ca-cert
        certificate: ca.crt
  authentication:
    type: tls
    certificateAndKey:
      secretName: connect-tls-auth
      certificate: user.crt
      key: user.key
  config:
    group.id: production-connect-cluster
    offset.storage.topic: connect-cluster-offsets
    config.storage.topic: connect-cluster-configs
    status.storage.topic: connect-cluster-status
    config.storage.replication.factor: 3
    offset.storage.replication.factor: 3
    status.storage.replication.factor: 3
    key.converter: io.confluent.connect.avro.AvroConverter
    value.converter: io.confluent.connect.avro.AvroConverter
    key.converter.schema.registry.url: http://schema-registry.kafka-connect.svc:8081
    value.converter.schema.registry.url: http://schema-registry.kafka-connect.svc:8081
  resources:
    requests:
      memory: "2Gi"
      cpu: "1"
    limits:
      memory: "4Gi"
      cpu: "2"
  jvmOptions:
    -Xms: 1g
    -Xmx: 2g

A few production notes:

  • version pins the Kafka version of the Connect workers. Pinning it keeps reconciliation deterministic across operator upgrades; omit it and the operator uses its default for the installed version.
  • Internal topics (*.storage.topic) must be unique per Connect cluster. The operator does not create them for you with custom partition counts — Connect creates them on first start using the replication factors above, so ensure the broker allows topic auto-creation or pre-create them.
  • group.id must be unique across all Connect clusters that share the same Kafka cluster, or workers from different clusters will join the same group and corrupt each other’s assignments.
  • Avro converters (io.confluent.connect.avro.AvroConverter) ship in Confluent’s plugin, not in the base Strimzi image. Add them through a build (below) or a custom image, or the workers will fail to start a connector that uses them.
  • jvmOptions uses the -Xms/-Xmx keys exactly. If you omit them but set memory requests/limits, the operator derives a default heap from the limit.

The strimzi.io/use-connector-resources: "true" annotation tells the operator to manage connectors through KafkaConnector resources instead of the REST API, which is covered below.

Configuring security and authentication

Strimzi supports TLS encryption, mutual TLS authentication, and SASL mechanisms. The example above uses mutual TLS: the worker presents a client certificate and trusts the cluster CA, both stored in Kubernetes Secrets and mounted automatically. For a Strimzi-managed KafkaUser, the operator generates the user.crt/user.key secret for you.

For SASL/SCRAM-SHA-512 over a TLS-encrypted connection, replace the authentication block:

  tls:
    trustedCertificates:
      - secretName: production-cluster-ca-cert
        certificate: ca.crt
  authentication:
    type: scram-sha-512
    username: connect-user
    passwordSecret:
      secretName: connect-scram-credentials
      password: password

passwordSecret.password is the key within the named Secret that holds the password value, not the password itself. Keep tls configured alongside SASL so credentials are not sent in plaintext — SASL authenticates but does not encrypt. The Strimzi configuration guide covers TLS, authentication, and authorization options in full.

Managing connectors declaratively

With strimzi.io/use-connector-resources: "true" set on the KafkaConnect cluster, you manage each connector as a KafkaConnector resource. The operator translates it into Connect REST API calls and reconciles drift, so connectors live in Git alongside the rest of your infrastructure.

Credentials should never be inlined. Kafka’s EnvVarConfigProvider (in kafka-clients since 3.5.0) lets you expose a secret as a worker environment variable via externalConfiguration, then reference it from connector config. First, register the provider under an alias — here env — and mount the secret in the KafkaConnect spec. The alias you choose becomes the prefix in the placeholder (${env:...}):

spec:
  config:
    config.providers: env
    config.providers.env.class: org.apache.kafka.common.config.provider.EnvVarConfigProvider
  externalConfiguration:
    env:
      - name: MYSQL_PASSWORD
        valueFrom:
          secretKeyRef:
            name: mysql-credentials
            key: password

Then deploy a Debezium MySQL CDC connector that reads the password from that variable:

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnector
metadata:
  name: mysql-inventory-connector
  namespace: kafka-connect
  labels:
    strimzi.io/cluster: production-connect-cluster
spec:
  class: io.debezium.connector.mysql.MySqlConnector
  tasksMax: 1
  autoRestart:
    enabled: true
  config:
    database.hostname: mysql-primary.database.svc
    database.port: "3306"
    database.user: debezium
    database.password: ${env:MYSQL_PASSWORD}
    database.server.id: "184054"
    topic.prefix: inventory
    database.include.list: inventory
    schema.history.internal.kafka.bootstrap.servers: production-kafka-bootstrap.kafka.svc:9093
    schema.history.internal.kafka.topic: schema-changes.inventory

Two correctness points specific to this connector:

  • tasksMax: 1 — the Debezium MySQL connector reads a single binlog and always runs exactly one task. Setting a higher value does not parallelize it.
  • autoRestart — with it enabled, the Cluster Operator watches the connector and its tasks every reconciliation and restarts any that have failed, using an exponential back-off (immediate, then after 2, 6, 12, 20, 30, 42, 56 minutes, capped at 60). Set autoRestart.maxRestarts to give up after a fixed number of attempts; left unset, the operator retries indefinitely. This is genuine self-healing, not merely status reporting. The resource’s status still reflects connector and task health for external alerting.

For many connectors across environments, template the KafkaConnector resources with Kustomize or Helm to keep staging and production consistent.

Building custom container images with plugins

Strimzi’s build feature lets the operator build a Connect image containing your plugins, so you do not maintain a separate Dockerfile and CI pipeline for images. The operator runs a Kaniko-based builder pod that downloads each artifact, assembles the image, and pushes it to your registry. Add a build section to the KafkaConnect spec:

spec:
  build:
    output:
      type: docker
      image: my-registry.io/kafka-connect-plugins:3.9.0-1
      pushSecret: registry-credentials
    plugins:
      - name: debezium-mysql-connector
        artifacts:
          - type: zip
            url: https://repo1.maven.org/maven2/io/debezium/debezium-connector-mysql/2.5.0.Final/debezium-connector-mysql-2.5.0.Final-plugin.zip
            sha512sum: <archive-sha512>

Details that matter:

  • Artifact type must be one of jar, zip, tgz, maven, or other, and must match the URL. Debezium ships plugin archives as ...-plugin.zip, so use type: zip. Note the file name is debezium-connector-mysql-2.5.0.Final-plugin.zip (hyphen-joined), not a nested path.
  • sha512sum is optional but recommended in production: Strimzi verifies the downloaded archive against it, so a tampered or corrupted download fails the build instead of shipping silently.
  • output.image should be a stable, versioned tag (not latest) so that a rebuild produces a new tag and the workers actually roll to the new image.
  • pushSecret is a kubernetes.io/dockerconfigjson Secret with push credentials for the registry.

Watch the builder pod and stream its logs while the image is built:

kubectl get pods -n kafka-connect -l strimzi.io/kind=KafkaConnectBuild
kubectl logs -f production-connect-cluster-build -n kafka-connect

When the build completes, the operator updates the Connect Deployment to the new image and performs a rolling restart. Because plugins are declared in the same manifest as the cluster, the whole configuration fits a GitOps workflow.

Monitoring and operating Kafka Connect clusters

Connect workers expose JMX metrics. Strimzi converts them to Prometheus format with the JMX Prometheus Exporter when you set metricsConfig to reference a ConfigMap. Metrics are then served on port 9404. Add this to the KafkaConnect spec:

spec:
  metricsConfig:
    type: jmxPrometheusExporter
    valueFrom:
      configMapKeyRef:
        name: connect-metrics
        key: metrics-config.yml

Create the referenced ConfigMap with the exporter rules. The patterns below match Connect’s worker- and task-level MBeans:

apiVersion: v1
kind: ConfigMap
metadata:
  name: connect-metrics
  namespace: kafka-connect
data:
  metrics-config.yml: |
    lowercaseOutputName: true
    lowercaseOutputLabelNames: true
    rules:
      - pattern: "kafka.connect<type=connect-worker-metrics><>([a-z-]+)"
        name: kafka_connect_worker_metrics_$1
      - pattern: "kafka.connect<type=connect-worker-metrics, connector=([^:]+)><>([a-z-]+)"
        name: kafka_connect_worker_connector_metrics_$2
        labels:
          connector: "$1"
      - pattern: "kafka.connect<type=connect-metrics, client-id=([^:]+)><>([a-z-]+)"
        name: kafka_connect_metrics_$2
        labels:
          client_id: "$1"

Key signals to alert on:

  • connector-failed-task-count and task-startup-failure-percentage from connect-worker-metrics — a task entering the FAILED state.
  • connector-count dropping unexpectedly, which indicates connectors disappearing from a worker.
  • Sink connector offset lag — track consumer group lag on the connector’s group, since Connect reuses the consumer group machinery for sink tasks.

The garbled metric-exporter output from earlier drafts of this topic is replaced above with the structure from Strimzi’s official kafka-connect-metrics.yaml example. The Apache Kafka Connect monitoring reference lists every MBean.

Combine metrics with the resource status. The operator records readiness in the KafkaConnect resource’s status.conditions; inspect it directly:

kubectl describe kafkaconnect production-connect-cluster -n kafka-connect

For logging, override the Connect Log4j configuration through the logging property of the KafkaConnect spec (inline or via a ConfigMap) and forward worker logs to your aggregator — Elasticsearch, Loki, or similar — so connector failures can be correlated with worker restarts.

Handling upgrades and rollbacks

Strimzi performs a controlled rolling restart of the worker pods whenever you change the replicas, config, version, or build section. Connect’s group rebalancing redistributes tasks as workers cycle, so connectors keep running across the roll provided the cluster has enough replicas to absorb one pod at a time.

Before upgrading the operator itself, read the version-specific notes in the Strimzi deployment and upgrade guide — CRD API versions and defaults change between releases. Back up your KafkaConnect and KafkaConnector manifests and rehearse the upgrade in a staging cluster that mirrors production.

To roll back a bad change, revert the resource in Git and reapply; the operator reconciles back to the declared state. For a single misbehaving connector, set spec.state: stopped (or paused) on its KafkaConnector — which supersedes the older spec.pause field — or delete the resource without touching the cluster, then resume once fixed.

Treating Connect clusters this way — provisioned, scaled, and decommissioned through the same Kubernetes-native workflows as your stateless services — is what makes Strimzi worthwhile: the plugin image, credentials, connectors, and cluster topology all live in one reconciled, version-controlled place.

Back to Kafka Connect Architecture And Deployment Modes