Operations, Security & Observability for Apache Kafka in Production

Running Apache Kafka at scale couples three domains that are usually treated separately: day-to-day operations that keep the cluster healthy, security controls that protect data in transit and at rest, and observability pipelines that surface problems before they become incidents. The coupling is the hard part. A misconfigured security protocol breaks a monitoring scrape; an aggressive rebalance stalls clients when mTLS handshakes time out under load; a durability config tuned for availability silently widens your data-loss window. This article lays out the patterns — and the specific failure modes — for operating clusters that stay secure, observable, and resilient as they grow.

This guide assumes Apache Kafka 3.x or 4.x in production. Kafka 4.0 (March 2025) removed ZooKeeper support entirely and runs KRaft-only, so where behavior differs by metadata mode or version it is called out explicitly. Working familiarity with Kafka fundamentals is assumed; follow the inline links for deeper coverage of each subtopic.

The Operational Model for Production Kafka

Cluster Provisioning and Configuration Management

Provision production clusters through infrastructure-as-code — Ansible, Terraform, or a Kubernetes operator such as Strimzi. Broker configurations must be version-controlled, repeatable, and auditable. The properties that most directly impact operations, security, and observability fall into a few categories.

# Core broker identity and networking
broker.id=1
listeners=INTERNAL://0.0.0.0:9092,EXTERNAL://0.0.0.0:9093
advertised.listeners=INTERNAL://kafka-broker-1.internal:9092,EXTERNAL://kafka-broker-1.example.com:9093
listener.security.protocol.map=INTERNAL:SSL,EXTERNAL:SASL_SSL
inter.broker.listener.name=INTERNAL

# Log storage and retention
log.dirs=/data/kafka
num.partitions=12
default.replication.factor=3
min.insync.replicas=2
log.retention.hours=168
log.segment.bytes=1073741824

# Network and I/O threads
num.network.threads=8
num.io.threads=16
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600

advertised.listeners is a frequent source of misconfiguration. Each listener must advertise an address that clients can actually resolve and reach. In cloud environments with separate internal and external networks, getting this wrong means a client connects to the bootstrap server successfully but then receives unreachable broker addresses from the metadata response. Validate listener resolution from real client subnets before promoting a configuration to production.

The durability triad — default.replication.factor=3, min.insync.replicas=2, and producers using acks=all — is the single most consequential decision in this file, and the three settings only work together. acks=all makes the leader wait for the in-sync replica set; min.insync.replicas=2 defines how many that set must contain before a write is accepted. With replication factor 3 and min.insync.replicas=2, you tolerate the loss of one replica and keep accepting writes; lose two, and producers get NotEnoughReplicasException rather than acknowledging a write you cannot durably hold. Setting min.insync.replicas equal to the replication factor looks safer but means a single broker restart blocks all production to that partition — a common self-inflicted outage. Note also that unclean.leader.election.enable defaults to false (Kafka favors durability over availability): if every in-sync replica for a partition is lost, that partition stays offline until a known-good replica returns, rather than promoting a stale out-of-sync replica and silently dropping acknowledged records. Override to true only when uptime genuinely outranks consistency, and decide it per topic, not cluster-wide.

Partition Reassignment and Data Balancing

Over time, clusters develop hot spots: brokers join, others are decommissioned, and topic traffic shifts. Manual partition reassignment is error-prone and operationally expensive. The modern approach is to delegate it to Automated Cluster Rebalancing with Cruise Control, which models cluster workload and generates optimized reassignment plans.

# Dry-run rebalance proposal (dryrun defaults to true, but be explicit)
curl -X POST "http://cruise-control:9090/kafkacruisecontrol/rebalance?dryrun=true&goals=RackAwareGoal,ReplicaCapacityGoal,NetworkInboundCapacityGoal"

# Execute, throttling replica movement to ~30 MB/s of bandwidth
curl -X POST "http://cruise-control:9090/kafkacruisecontrol/rebalance?dryrun=false&replication_throttle=30000000"

The bandwidth-limiting parameter is replication_throttle (bytes/sec); concurrency is bounded separately by concurrent_partition_movements_per_broker (default 5). Even with automation, understand the underlying mechanics. Reassignment copies whole log segments between brokers, and Kafka itself enforces a throttle through the dynamic broker config leader.replication.throttled.rate / follower.replication.throttled.rate (set per-broker via kafka-configs --entity-type brokers) together with the per-topic leader.replication.throttled.replicas / follower.replication.throttled.replicas lists. Set throttles conservatively during business hours and relax them in maintenance windows.

The non-obvious failure mode: a throttle set too low can be worse than no throttle. If replication can’t keep pace with incoming writes, the moving replica falls further behind, never catches up, and the reassignment hangs indefinitely with the partition under-replicated. Watch UnderReplicatedPartitions during the move; if it climbs instead of draining, raise the throttle rather than waiting it out.

Security Architecture: Authentication, Authorization, and Encryption

Security in Kafka is layered: encryption of data in transit, authentication of every connection, and authorization of every operation. Production clusters must enforce all three consistently across brokers, clients, and tooling.

Transport-Layer Encryption and Mutual TLS

Kafka supports TLS on every listener. Production clusters should terminate plaintext listeners entirely. Mutual TLS (mTLS) — where both broker and client present certificates — provides authentication without an additional SASL mechanism and ties every connection to a verifiable identity. For a complete walkthrough, see Securing Kafka with mTLS and SASL/SCRAM.

# Generate a broker keystore with a key pair
keytool -keystore kafka.broker.keystore.jks -alias broker -validity 365 -genkeypair -keyalg RSA \
  -dname "CN=kafka-broker-1.internal" -storepass changeit -keypass changeit

# Export a certificate signing request
keytool -keystore kafka.broker.keystore.jks -alias broker -certreq -file broker.csr -storepass changeit

# Sign with your internal CA
openssl x509 -req -CA ca.crt -CAkey ca.key -in broker.csr -out broker-signed.crt -days 365 -CAcreateserial

# Import the CA root, then the signed certificate, back into the keystore
keytool -keystore kafka.broker.keystore.jks -alias CARoot -import -file ca.crt -storepass changeit -noprompt
keytool -keystore kafka.broker.keystore.jks -alias broker -import -file broker-signed.crt -storepass changeit

Certificate rotation is an operational reality: broker keystores must be refreshed before certificates expire, without cluster downtime. Kafka supports this through dynamic broker configuration (KIP-226). Point a broker at a new keystore file and update the keystore path as a per-broker dynamic config — the broker reloads it for new connections without a restart:

kafka-configs --bootstrap-server kafka-broker-1:9092 --alter \
  --entity-type brokers --entity-name 1 \
  --add-config 'ssl.keystore.location=/etc/kafka/ssl/kafka.broker.keystore.new.jks,ssl.keystore.password=changeit,ssl.key.password=changeit'

Two caveats decide how you schedule this. First, from KIP-226: dynamic updates apply to per-listener security stores, but you cannot dynamically change the inter-broker listener’s security protocol or listener name — plan inter-broker certificate changes accordingly. Second, because the reload affects only new connections, existing client connections keep using the old certificate until they reconnect; the cluster does not force a re-handshake. Rotate well ahead of expiry, and treat days-to-expiry as a monitored, alertable metric rather than a calendar reminder — an expired keystore that nobody caught is a cluster-wide outage that no config change can fix once connections start dropping.

SASL/SCRAM and Password-Based Authentication

Not every organization can run a full mTLS PKI for all clients. SASL/SCRAM provides a password-based alternative that still performs strong, salted-challenge authentication — but only over TLS, since the SCRAM exchange must not be observable. SCRAM-SHA-256 and SCRAM-SHA-512 are the supported mechanisms (minimum 4096 iterations). Credentials are stored in the metadata log (KRaft) or in ZooKeeper on older clusters; brokers never see the plaintext password.

# Create a SCRAM credential for a user principal
kafka-configs --bootstrap-server kafka-broker-1:9092 \
  --alter --add-config 'SCRAM-SHA-256=[iterations=8192,password=secure-password]' \
  --entity-type users --entity-name app-producer

# Describe the user to verify (the salted credential is shown, not the password)
kafka-configs --bootstrap-server kafka-broker-1:9092 \
  --describe --entity-type users --entity-name app-producer

Combine SCRAM with TLS for transport: set SASL_SSL for client-facing listeners in listener.security.protocol.map, and list SCRAM-SHA-256,SCRAM-SHA-512 in the broker’s sasl.enabled.mechanisms. Choose SCRAM over mTLS when you cannot distribute and rotate client certificates at scale; choose mTLS when you want identity bound to the transport with no shared secret to leak. Many production clusters run both — mTLS on the inter-broker and internal listeners, SASL/SCRAM over TLS for external application clients.

Authorization with ACLs

Once a connection is authenticated, authorization controls what the principal may do. The authorizer class depends on the metadata mode:

• KRaft clusters use org.apache.kafka.metadata.authorizer.StandardAuthorizer (introduced by KIP-801), which stores ACLs in the __cluster_metadata log.
• ZooKeeper clusters (Kafka 3.x only) use kafka.security.authorizer.AclAuthorizer, which stores ACLs in ZooKeeper.

Both implement the KIP-504 Authorizer interface and honor the same static configs (super.users, allow.everyone.if.no.acl.found). The older kafka.security.auth.SimpleAclAuthorizer was a separate, deprecated class and was removed in Kafka 3.0 — do not configure it. Set the class explicitly via authorizer.class.name.

Whatever the backend, the kafka-acls CLI is identical. Apply least privilege: each service account gets exactly the permissions it needs. See Kafka ACLs and Authorization Best Practices.

# Grant a producer principal write access to a specific topic
kafka-acls --bootstrap-server kafka-broker-1:9092 \
  --add --allow-principal User:app-producer \
  --operation Write --topic orders.v1

# Grant a consumer read access to a topic and a specific consumer group
kafka-acls --bootstrap-server kafka-broker-1:9092 \
  --add --allow-principal User:app-consumer \
  --operation Read --topic orders.v1 --group orders-consumer-group

# List all ACLs for auditing
kafka-acls --bootstrap-server kafka-broker-1:9092 --list

Deny rules take precedence over allow rules. Two settings cause most ACL incidents: leaving allow.everyone.if.no.acl.found=true (the permissive default) so that a topic with no ACLs is wide open rather than locked down, and granting wildcard principals (User:*). Set allow.everyone.if.no.acl.found=false in production and create explicit ACLs per service identity. Use prefixed patterns (--topic orders. --resource-pattern-type prefixed, KIP-290) to reduce rule sprawl for services that own many topics under a common prefix. A producer using transactions or idempotence needs more than Write: grant Describe and, for transactional producers, Write on the TransactionalId resource — a missing transaction ACL surfaces as a confusing authorization failure at initTransactions(), not at send time.

Observability: Metrics, Logging, and Tracing

A Kafka cluster that cannot be observed cannot be operated. Observability means the right telemetry at the right granularity, plus the ability to correlate signals across brokers, clients, and the network.

JMX Metrics with Prometheus and Grafana

A Kafka broker exposes hundreds of JMX metrics. The challenge is filtering, aggregating, and alerting on the subset that matters. The standard stack pairs the Prometheus JMX Exporter with Grafana dashboards. For a full setup, see Monitoring Kafka with JMX, Prometheus, and Grafana.

# jmx_exporter rules for a Kafka broker
lowercaseOutputName: true
rules:
  # Broker-level throughput
  - pattern: kafka.server<type=BrokerTopicMetrics, name=(BytesInPerSec|BytesOutPerSec)><>Count
    name: kafka_broker_throughput_bytes_total
    labels:
      direction: "$1"
  # Under-replicated partitions (should be 0)
  - pattern: kafka.server<type=ReplicaManager, name=UnderReplicatedPartitions><>Value
    name: kafka_under_replicated_partitions
  # Active controller count (cluster-wide should sum to 1)
  - pattern: kafka.controller<type=KafkaController, name=ActiveControllerCount><>Value
    name: kafka_active_controller_count
  # Request handler pool idle fraction (low = saturated I/O threads)
  - pattern: kafka.server<type=KafkaRequestHandlerPool, name=RequestHandlerAvgIdlePercent><>OneMinuteRate
    name: kafka_request_handler_idle_percent

A handful of signals carry most of the diagnostic value, and each has a clear bright line:

For security, watch authentication failures (failed-authentication-total on the listener metrics) and ACL-denial logging. For stream processing, consumer-group lag is the primary signal. ActiveControllerCount deserves special discipline: it is the one metric you should page on immediately, because a sum of zero means metadata changes — leader elections, topic creation, ACL updates — silently stop applying even though existing produce and consume traffic keeps flowing, so the symptom is invisible until something needs the controller.

Consumer Lag and End-to-End Latency

Consumer lag — the gap between the latest produced offset and the last committed offset for a group — is the canonical health signal for streaming pipelines. Kafka exposes it through the kafka-consumer-groups CLI and via the consumer client’s records-lag-max JMX metric.

# Per-partition lag for a consumer group
kafka-consumer-groups --bootstrap-server kafka-broker-1:9092 \
  --group orders-processor --describe

# Output columns include CURRENT-OFFSET, LOG-END-OFFSET, and LAG

Alert on the trend, not the instantaneous value. Lag in absolute records is misleading — 100,000 records is trivial for a batch sink and an emergency for a low-latency path. Two rules separate real incidents from noise: alert when lag is monotonically increasing over the burn-in window (the consumer is falling behind faster than it catches up, regardless of absolute value), and alert when the estimated time-to-drain (lag ÷ recent consume rate) exceeds your latency SLO. Lag spikes are normal during rebalances and brief consumer restarts; ignore anything shorter than your rebalance time. Sustained, climbing lag concentrated on one or a few partitions usually means a poison message blocking that partition’s consumer, not a capacity problem — check whether CURRENT-OFFSET is frozen for those partitions while others advance. Export lag to Prometheus via the consumer client metric or a tool such as Kafka Lag Exporter.

Broker and Client Logging

Broker logs are your first stop during incident response. The most useful loggers are kafka.authorizer.logger (ACL allow/deny decisions), kafka.request.logger (request-level tracing, expensive — enable selectively), and state.change.logger (leadership changes and controller events).

Logging configuration is version-dependent:

• Kafka 3.x ships Log4j 1.x and reads log4j.properties.
• Kafka 4.0 migrated to Log4j 2 (KIP-653) and reads log4j2.yaml / log4j2.properties. The old KafkaLog4jAppender was removed. Existing Log4j 1.x configs can be converted with the log4j-transform-cli tool.

A minimal Log4j 2 broker config (Kafka 4.0):

# log4j2.properties — console output plus targeted logger levels
rootLogger.level = INFO
rootLogger.appenderRef.stdout.ref = STDOUT

appender.stdout.type = Console
appender.stdout.name = STDOUT
appender.stdout.layout.type = PatternLayout
appender.stdout.layout.pattern = [%d] %p %m (%c)%n

logger.authorizer.name = kafka.authorizer.logger
logger.authorizer.level = INFO
logger.statechange.name = state.change.logger
logger.statechange.level = INFO

For structured JSON logs (to ship to Elasticsearch or Loki), Log4j 2 provides a built-in JsonTemplateLayout (layout.type = JsonTemplateLayout); on Kafka 3.x / Log4j 1.x, JSON output requires a third-party layout library. Prefer migrating to Log4j 2 over adding third-party 1.x dependencies.

Multi-Tenant Operations and Resource Governance

As Kafka becomes a centralized data backbone, multiple teams share one physical cluster. Without governance, a single misbehaving producer can saturate broker disk or network and degrade every tenant.

Quota Management and Throttling

Quotas limit the rate at which clients produce or consume. They are defined per client ID, per user principal, or per (user, client-id) pair, and are enforced per broker. See Quota Management and Throttling in Multi-Tenant Clusters.

# Produce-byte-rate quota of 10 MB/s for a client ID
kafka-configs --bootstrap-server kafka-broker-1:9092 \
  --alter --add-config 'producer_byte_rate=10485760' \
  --entity-type clients --entity-name team-payments-producer

# Consume-byte-rate quota of 20 MB/s for a user principal
kafka-configs --bootstrap-server kafka-broker-1:9092 \
  --alter --add-config 'consumer_byte_rate=20971520' \
  --entity-type users --entity-name team-analytics

# Request-rate quota: cap a client's share of broker I/O + network thread time
kafka-configs --bootstrap-server kafka-broker-1:9092 \
  --alter --add-config 'request_percentage=50' \
  --entity-type clients --entity-name batch-importer

request_percentage limits a client’s share of (num.io.threads + num.network.threads) * 100% of thread time; a separate controller_mutation_rate quota caps topic/partition create and delete operations. Because quotas are per broker, a producer_byte_rate of 10 MB/s allows up to 10 MB/s on each broker; across 6 brokers the effective aggregate is ~60 MB/s. Plan accordingly.

The detail that catches teams: quotas don’t reject — they delay. When a client exceeds its quota, the broker holds the response, applying back-pressure rather than returning an error, so a throttled client looks like a slow or laggy one, not a rejected one. Confirm a throttle from the client side via produce-throttle-time-avg / produce-throttle-time-max (and the fetch equivalents); a non-zero average there means the client is quota-bound, which is the difference between “the cluster is slow” and “this client hit its limit.” That distinction routes the incident to the right team.

Topic and Partition Governance

Unbounded topic creation leads to operational chaos. Enforce naming conventions, partition counts, and replication factors through a creation pipeline — a CLI wrapper, a GitOps workflow, or a platform API that validates requests before calling Admin.createTopics(). Set auto.create.topics.enable=false on every production broker so a misconfigured producer cannot create topics.

# Create a topic with explicit configuration
kafka-topics --bootstrap-server kafka-broker-1:9092 \
  --create --topic orders.v2 \
  --partitions 12 --replication-factor 3 \
  --config min.insync.replicas=2 \
  --config retention.ms=604800000

Partition count is the governance decision that is easy to get wrong and hard to undo: you can increase a topic’s partitions but never decrease them, and increasing them breaks keyed ordering (a key that hashed to partition 3 may now hash to partition 7, so its history is split across two partitions). Size partition counts for projected peak throughput up front rather than growing them reactively.

Cluster Maintenance and Upgrades

Production clusters must evolve without downtime: rolling restarts, version upgrades, and config changes that respect the distributed nature of the system.

Rolling Upgrades Without Downtime

Kafka’s wire protocol is backward-compatible, and the broker can be upgraded one node at a time. Restart each broker, wait for it to rejoin the ISR for every partition it leads or follows, and only then move to the next. On Kafka 3.x with ZooKeeper, the two-phase inter.broker.protocol.version (IBP) procedure still applies; on KRaft, feature levels are advanced separately (see below). The full procedure is in Rolling Upgrades and Maintenance Without Downtime.

# Inspect supported API versions before upgrading
kafka-broker-api-versions --bootstrap-server kafka-broker-1:9092 | head -20

# Under-replicated partitions must be zero before and after each restart
kafka-topics --bootstrap-server kafka-broker-1:9092 --describe --under-replicated-partitions

# Rolling restart of one broker (systemd example)
systemctl stop kafka
# Upgrade binaries, update configuration if needed
systemctl start kafka
# Re-check before proceeding to the next broker
kafka-topics --bootstrap-server kafka-broker-1:9092 --describe --under-replicated-partitions

The gate between brokers is non-negotiable: under-replicated partitions must return to zero before you touch the next node. Restarting a second broker while the first is still catching up can drop a partition below min.insync.replicas and halt production to it. On the IBP procedure specifically, leave inter.broker.protocol.version pinned to the old version until every broker runs the new binaries, then bump it in a second rolling restart — this is what lets you roll back cleanly if the new version misbehaves, because a bumped protocol version is a one-way door. For KRaft clusters, the controller quorum is upgraded with the same rolling discipline, and you must never lose a majority of controllers during the window. After all nodes run the new binaries, finalize the metadata version with kafka-features --bootstrap-server ... upgrade --metadata <version>. Monitor ActiveControllerCount throughout.

Maintenance Procedures and Disk Management

Broker disks fill up. Time- and size-based retention handle normal operation, but traffic spikes or consumer outages can overwhelm storage. Configure retention.bytes as a per-topic safety net and alert on disk usage at 70% and 85%. When a log directory fills, the broker marks it offline, taking its partitions unavailable — and on a JBOD broker, only the partitions on that disk fail, which makes the symptom partial and confusing unless you alert per log directory.

# Size-based retention limit on an existing topic (100 GiB per partition)
kafka-configs --bootstrap-server kafka-broker-1:9092 \
  --alter --entity-type topics --entity-name orders.v2 \
  --add-config retention.bytes=107374182400

Remember that retention.bytes is enforced per partition, not per topic: a 100 GiB limit on a 12-partition topic permits up to ~1.2 TiB across the cluster before deletion kicks in. Size it against per-broker disk, not the topic as a whole. Watch free space directly:

# Disk usage on the broker
df -h /data/kafka

To decommission a broker, move its partitions off with kafka-reassign-partitions before shutting it down, or use Cruise Control’s remove_broker endpoint, which generates a rack-aware, load-balanced reassignment plan (and supports dryrun and throttle_removed_broker).

Disaster Recovery and Geo-Replication

Production deployments must survive datacenter failures. That requires replicating data across geographic boundaries and a tested plan for failing over and back.

Geo-Replication with MirrorMaker 2

MirrorMaker 2 (MM2) is the standard tool for replicating topics between clusters. It runs on Kafka Connect: a source connector reads from the source cluster and produces to the target. Architecture and operations are detailed in Geo-Replication with MirrorMaker 2.

# connect-mirror-maker.properties
clusters = primary, secondary
primary.bootstrap.servers = kafka-primary-1:9092,kafka-primary-2:9092,kafka-primary-3:9092
secondary.bootstrap.servers = kafka-secondary-1:9092,kafka-secondary-2:9092,kafka-secondary-3:9092

primary->secondary.enabled = true
primary->secondary.topics = orders\..*, payments\..*

# Replication factor for mirrored data topics on the target
replication.factor = 3

# Replication factors for MM2 internal topics
primary->secondary.offset-syncs.topic.replication.factor = 3
checkpoints.topic.replication.factor = 3
heartbeats.topic.replication.factor = 3

# Emit heartbeats and checkpoints (offset translation)
primary->secondary.emit.heartbeats.enabled = true
primary->secondary.emit.checkpoints.enabled = true

MM2 records offset mappings in an offset-syncs topic and emits checkpoints, enabling consumer-group offset translation from source to target. This is what lets consumers resume from the right position after failover (use RemoteClusterUtils or the MirrorClient API to translate offsets) rather than replaying all retained data. Note the default DefaultReplicationPolicy prefixes mirrored topics with the source alias (e.g. primary.orders.v2).

Two operational realities define MM2’s limits. It is asynchronous, so the target always trails the source — your real-world RPO is whatever the replication lag is at the instant of failover, not zero; measure it from heartbeat latency rather than assuming it. And offset translation is approximate: a translated offset is guaranteed not to skip un-replicated records, but a failed-over consumer may reprocess some, so downstream consumers must tolerate at-least-once delivery across a failover.

Disaster Recovery Planning

Geo-replication is not, by itself, disaster recovery. A DR plan defines runbooks, RPO/RTO targets, and failover procedures per failure scenario. The full approach is in Disaster Recovery Strategies for Kafka. At minimum, address:

• Active-Passive vs. Active-Active: passive is simpler but wastes capacity; active-active needs conflict handling for topics written in both sites.
• Client Failover: DNS switching, load-balancer updates, or client-side reconfiguration to redirect producers and consumers.
• Data Validation: after failover, compare end offsets and message counts between clusters to detect replication gaps.
• Failback: reverse replication direction and resynchronize before switching clients back.

# Compare end offsets (sum of partition high watermarks) for a topic on each cluster.
# GetOffsetShell uses --bootstrap-server; --broker-list is deprecated/removed.
kafka-run-class kafka.tools.GetOffsetShell \
  --bootstrap-server kafka-primary-1:9092 \
  --topic orders.v2 --time -1

kafka-run-class kafka.tools.GetOffsetShell \
  --bootstrap-server kafka-secondary-1:9092 \
  --topic primary.orders.v2 --time -1

A DR plan you have not exercised is a hypothesis, not a plan. Schedule periodic game-day failovers: the recurring surprises are stale DNS TTLs that strand clients on the dead cluster, consumer ACLs that were never replicated to the secondary, and offset-translation gaps that only appear under real traffic. Each is cheap to find in a drill and expensive to discover during an actual outage.

Stream Processing Automation and Operational Integration

Kafka operations feed stream-processing frameworks — Kafka Streams, ksqlDB, Apache Flink. Operating these (state-store management, rebalance handling, exactly-once semantics) extends core Kafka operations.

Managing Kafka Streams Applications

A Kafka Streams app is a Java process embedding consumers and producers. It keeps local state stores backed by changelog topics. When an instance fails or scales, partitions rebalance across surviving instances, triggering state restoration from the changelog — which can take minutes for large state.

// Kafka Streams configuration with operational tuning
Properties props = new Properties();
props.put(StreamsConfig.APPLICATION_ID_CONFIG, "orders-processor");
props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "kafka-broker-1:9092,kafka-broker-2:9092");
props.put(StreamsConfig.NUM_STANDBY_REPLICAS_CONFIG, 1);
props.put(StreamsConfig.COMMIT_INTERVAL_MS_CONFIG, 10000);
props.put(StreamsConfig.STATESTORE_CACHE_MAX_BYTES_CONFIG, 10485760); // 10 MiB
props.put(StreamsConfig.producerPrefix(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG), true);
props.put(StreamsConfig.PROCESSING_GUARANTEE_CONFIG, StreamsConfig.EXACTLY_ONCE_V2);

num.standby.replicas is the central operational trade-off: standbys keep a warm copy of each state store so a failover restores in seconds instead of the minutes a cold changelog replay takes, at the cost of extra disk on each instance and continuous changelog read bandwidth. For any app with large state where restoration time is part of your availability budget, set at least 1. Monitor liveness with the kafka.streams:type=stream-metrics MBean (alive stream threads, rebalance activity) and track restoration progress via the per-task restore metrics under kafka.streams:type=stream-task-metrics (restore-records-rate, restore-total) and the state-store restore metrics. (Note: StreamsConfig.CACHE_MAX_BYTES_BUFFERING_CONFIG was deprecated in favor of STATESTORE_CACHE_MAX_BYTES_CONFIG; use the latter on recent versions.)

Schema Registry Operations

If your pipelines use Avro, Protobuf, or JSON Schema, the Schema Registry is critical infrastructure: highly available, backed up, and monitored. It persists schemas in a compacted Kafka topic (_schemas), so its durability rides on Kafka’s replication. Treat the Registry service itself as stateless and horizontally scalable, but ensure its backing topic has an appropriate min.insync.replicas.

# Create the _schemas topic explicitly (compacted, replicated)
kafka-topics --bootstrap-server kafka-broker-1:9092 \
  --create --topic _schemas \
  --partitions 1 --replication-factor 3 \
  --config min.insync.replicas=2 \
  --config cleanup.policy=compact

The constraint to respect: _schemas must stay single-partition and compacted, because the Registry relies on a totally ordered log to elect a primary and assign monotonic schema IDs. Never repartition it, and never set a time- or size-based retention that could delete schemas still referenced by live data — a lost schema makes every message written under that ID undeserializable.

Building an Operational Culture Around Kafka

The final layer of production readiness is organizational. Teams that run Kafka well invest in runbooks, chaos engineering, and blameless postmortems. They treat clusters as products and measure reliability with SLOs.

Defining SLOs and Error Budgets

Every cluster should have service-level objectives. Common examples:

• Availability: 99.9% of produce requests succeed within 100 ms over a 30-day window.
• Durability: zero acknowledged messages lost, verified by continuous end-to-end checks.
• Latency: P99 produce latency under 50 ms on the INTERNAL listener.

These SLOs define error budgets that govern change velocity: within budget, push configuration changes and rolling upgrades; over budget, freeze changes and invest in reliability.

Chaos Engineering for Kafka

Kafka’s distributed design makes it a strong candidate for chaos experiments. Regularly inject failures — broker termination, network latency, disk fill, KRaft controller loss — and verify that monitoring detects the issue and automated remediation responds. Chaos Mesh and LitmusChaos provide structured, Kubernetes-native frameworks for running these experiments against Kafka.

Continuous Learning and Community Resources

The Kafka operational landscape moves quickly. The Apache Kafka documentation is the authoritative reference for configuration and protocol details, the Kafka Improvement Proposals (KIPs) explain the rationale behind new features, and the Confluent Operations documentation offers vendor-specific but broadly applicable guidance.

Operating Kafka is a continuous practice, not a one-time setup: the cluster you run today will have more topics, more traffic, and more tenants in six months. The throughlines that let it scale are the ones this article keeps returning to — the durability triad enforced end to end, a handful of metrics with hard alert thresholds, and procedures (rolling upgrades, failovers, decommissions) gated on observable cluster state rather than wall-clock timing. When you hit a specific challenge, the linked guides carry the detailed procedures and configuration for your environment.