Cluster Architecture & Provisioning

Designing and provisioning an Apache Kafka cluster for production directly determines the resilience, scalability, and operational burden of your event streaming platform. The decisions made during architecture and provisioning echo through every incident, scaling event, and pager alert. This pillar covers the operationally important choices: metadata management, broker sizing, failure-domain placement, tiered storage, capacity planning, automated provisioning, and the monitoring that keeps it all healthy.

The journey begins with the metadata consensus mechanism. For years Apache ZooKeeper was the only option, but the Kafka Raft (KRaft) protocol has replaced it. As of Apache Kafka 4.0, ZooKeeper mode is removed entirely and KRaft is the only supported mode, so a new cluster is a KRaft cluster. Understanding the trade-offs in KRaft vs ZooKeeper: Choosing the Right Metadata Management still matters because most existing fleets are mid-migration. KRaft eliminates the external ZooKeeper ensemble, simplifying operations and enabling faster controller failover. A quorum of controller nodes manages the metadata log, which is itself a Kafka log replicated by Raft. Use an odd number of controllers (typically 3 or 5) so a clear majority survives a network partition. Controllers can be colocated with brokers or run on dedicated nodes, a decision that hinges on isolation requirements.

Once the metadata architecture is settled, the next layer is the infrastructure that hosts the brokers, where throughput and latency meet CPU, memory, disk I/O, and network. Broker Sizing and Hardware Selection for Kafka Clusters is an iterative practice tied to your workload. A common anti-pattern is over-provisioning CPU while starving storage and network. Kafka is I/O intensive: the bottleneck is usually disk for producers and network for consumers. For a high-throughput case such as ingesting 1 GB/s of clickstream data, prioritize multiple independent disks in a JBOD layout, high-throughput NICs (25 GbE or higher), and enough memory for the OS page cache. The page cache is Kafka’s lever: sized to hold the active segments of each partition, it serves most consumer reads from RAM. A reasonable baseline is 64 GB or more of RAM per broker, with most of it left to the page cache rather than the JVM heap. The heap should stay well under 32 GB to avoid long GC pauses, and G1GC is the standard collector for its predictable pauses.

With broker specs defined, the focus shifts to arranging brokers across failure domains. A single-AZ deployment is a liability for any production system. Multi-AZ Kafka Deployment Strategies for High Availability distributes brokers across at least three isolated zones in a region so the loss of an entire AZ does not cause data loss or a full outage. The key parameter is the replication factor. For a three-AZ deployment, replication factor 3 with min.insync.replicas=2 is standard: it tolerates the loss of one AZ while still allowing acks=all producers to write. This introduces cross-AZ traffic that adds latency and cost, so measure inter-AZ round-trip time and keep it inside your producers’ latency budget. The KRaft controller quorum follows the same logic: spread controllers across the same failure domains as the brokers so one AZ failure cannot take down metadata consensus.

Building on multi-AZ distribution, finer fault tolerance comes from rack awareness. Implementing Rack Awareness in Kafka for Fault Tolerance guards against correlated failures within a zone, such as a top-of-rack switch failure or a shared power bus trip. Kafka’s rack awareness places replicas of a partition on brokers in different racks, defined by broker.rack. You must assign each broker a rack identifier; when a topic is created or a partition reassigned, the placement algorithm spreads replicas across distinct racks. You can encode both the zone and the physical rack in a composite identifier such as us-east-1a-rack07. To validate placement, generate a proposed reassignment with kafka-reassign-partitions.sh --generate and inspect the replica distribution, or describe the topic and confirm each replica list spans distinct racks.

As data volumes grow, keeping all data on fast local disks becomes uneconomical. Tiered Storage in Kafka: Architecture and Configuration (production-ready since Kafka 3.9, after early access in 3.6 via KIP-405) offloads older, less-frequently-read segments to remote object storage such as Amazon S3, GCS, or Azure Blob. The architecture separates a hot tier (local disks) from a cold tier (remote storage). The broker moves closed log segments to the remote tier per retention policy and transparently fetches them back when a consumer reads tiered data. This changes capacity planning: local storage only needs to hold active segments plus enough history for typical consumer catch-up. A common setup retains 24 hours locally and 30 days remotely, cutting local storage cost substantially and speeding rebalances. The cost is a dependency on the remote store’s latency and availability, so configure timeouts and monitor remote-fetch latency.

No architecture survives an undersized cluster. Capacity Planning for Kafka Clusters: Throughput and Storage combines workload modeling, benchmarking, and trend analysis across two dimensions: throughput (MB/s or messages/s) and storage (GB/TB). Throughput depends on broker count, network bandwidth, and disk I/O. Storage is a function of retention, ingress, and replication factor: Storage = Ingress_Throughput * Retention_Seconds * Replication_Factor, plus roughly 30% overhead for indices, internal topics, and filesystem slack. Partition count matters too: each partition consumes memory for indices and file handles, and the controller tracks leadership and ISR state for every partition. KRaft raises the practical partition ceiling well beyond ZooKeeper-era limits, but it is still finite, and per-broker partition count remains a real constraint.

With the architecture designed and capacity modeled, provision it reliably and repeatably. Ad-hoc console clicks and one-off scripts breed configuration drift. Automated Provisioning of Kafka Clusters with Terraform and Ansible treats infrastructure as code: Terraform declares cloud resources (instances, disks, security groups, load balancers) and Ansible configures the OS, installs Kafka, renders config from templates, and starts services. This separates provisioning from configuration management and makes scaling a matter of changing a count variable and reapplying.

Once the cluster serves traffic, the focus is visibility. Monitoring Cluster Health and Performance Metrics is the feedback loop that keeps the platform healthy. Kafka exposes metrics via JMX, typically scraped into Prometheus and visualized in Grafana. The essential signals include throughput (BytesInPerSec/BytesOutPerSec), request latency (TotalTimeMs), durability (UnderReplicatedPartitions), controller health (ActiveControllerCount), JVM heap and GC, and OS-level disk metrics. Alert on under-replicated partitions above zero for more than a few minutes, disk usage past ~85%, and sudden drops in bytes-in that may signal a producer outage.

KRaft Metadata Management in Depth

The move from ZooKeeper to KRaft is a re-architecture of Kafka’s control plane, not just a consensus swap. Metadata describing topics, partitions, ISRs, and configs lives in an internal single-partition topic, __cluster_metadata, replicated across the controller quorum via Raft. The active controller is the Raft leader and the only node that writes to the metadata log. This removes the old failover penalty where a new controller had to reload the full metadata state from ZooKeeper, which could take minutes on large clusters. With KRaft the metadata log is already in memory on every controller, so failover takes seconds.

A KRaft quorum is defined by controller.quorum.voters, which lists each controller by ID, host, and port:

controller.quorum.voters=1@controller-1.example.com:9093,2@controller-2.example.com:9093,3@controller-3.example.com:9093

(On Kafka 3.9+ with dynamic quorums from KIP-853, you can instead bootstrap with controller.quorum.bootstrap.servers and add or remove voters at runtime; controller.quorum.voters remains valid for a static quorum.)

Each node needs a unique node.id and a controller.listener.names value naming the listener used for inter-controller Raft traffic. The process.roles property sets a node’s role: controller, broker, or broker,controller for combined nodes. The choice is operational, not cosmetic:

• Combined (broker,controller) — fewer machines, simpler to run, and fine for small clusters and dev. The risk is shared fate: a GC pause in the broker’s data plane can make the node miss Raft heartbeats and trigger an unnecessary controller election, and a broker OOM takes a controller voter down with it.
• Dedicated controllers — three controllers on smaller, isolated hardware give a more stable control plane and predictable failover. This is the default recommendation for any cluster large enough that a control-plane wobble has real blast radius. Combined mode also cannot be converted to a dedicated topology in place, so deciding up front avoids a later migration.

The metadata log itself needs care. metadata.log.dir sets where the Raft log is stored; place it on a fast disk to minimize fsync latency. (If unset, the metadata log goes in the first directory listed in log.dirs.) metadata.log.segment.bytes and metadata.log.segment.ms control segment rolling. The metadata log is compacted, retaining the latest value per key, so monitor its size and segment-creation rate to catch misconfiguration or bugs that cause unbounded growth.

Migrating an existing ZooKeeper cluster to KRaft (KIP-866) is a sequenced, online process. You run a bridge release that speaks both modes — 3.5 or later is recommended, and 3.9 is the final bridge release, since 4.0 removed ZooKeeper. You provision KRaft controllers (formatted with kafka-storage.sh format using the existing cluster ID), enable migration, and let the cluster enter dual-write mode, where metadata is written to both KRaft and ZooKeeper so a rollback is possible. You then roll the brokers from ZooKeeper mode into KRaft mode, finalize the migration, and decommission ZooKeeper. Note that the metadata version cannot be upgraded during migration, so inter.broker.protocol.version must already be at 3.9 before you start. Test the full procedure in a staging environment that mirrors production before touching live data; follow the official Apache Kafka migration guide step by step.

Broker Hardware Specification and Sizing

Translating workload requirements into hardware is what separates reliable Kafka platforms from those that buckle under load. The four resources — CPU, memory, disk, network — must be balanced, because one starved resource bottlenecks every partition a broker hosts.

CPU sizing is often misunderstood. Brokers do not compute on payloads, but CPU is consumed by request handling, compression/decompression, TLS, and log compaction. A broker handling 10,000 TLS connections with Snappy compression uses far more CPU than one with 100 plaintext, uncompressed connections. A practical starting point is 16 to 32 vCPUs, the higher end for TLS-heavy or high-connection workloads. Hyper-threading yields diminishing returns for Kafka’s I/O-bound profile, so prefer physical cores where you can choose.

Memory is the most important resource for read performance. Kafka leans on the OS page cache to serve consumer reads from memory. Size total memory to hold the active segments of every hosted partition plus the JVM heap and OS overhead. Keep the heap between 8 GB and 32 GB with G1GC and leave the rest to the page cache. For example, a 128 GB broker might give 16 GB to the heap and leave ~112 GB for page cache, comfortably covering the most recent hours of a workload. The heap is set via KAFKA_HEAP_OPTS and JVM options:

export KAFKA_HEAP_OPTS="-Xms16g -Xmx16g"
export KAFKA_JVM_PERFORMANCE_OPTS="-XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:+ExplicitGCInvokesConcurrent"

MaxGCPauseMillis=20 is aggressive but achievable for many workloads, and InitiatingHeapOccupancyPercent=35 starts concurrent marking early to avoid full collections. (Kafka’s own start scripts set its default G1GC options through KAFKA_JVM_PERFORMANCE_OPTS; prefer ExplicitGCInvokesConcurrent over DisableExplicitGC so any explicit System.gc() runs concurrently instead of being silently dropped.)

Disk is the most common bottleneck for write-heavy workloads. Kafka writes sequentially, which suits both SSDs and HDDs, but concurrent reads from lagging consumers create a random read pattern that HDDs handle poorly. The modern default is SSD or NVMe, which keeps catch-up reads fast and avoids seek-induced lag spikes. If HDDs are unavoidable, use JBOD with multiple independent disks, each mounted as its own log directory; Kafka spreads partitions across them, increasing aggregate throughput. JBOD also lets a broker survive a single data-disk failure rather than losing everything — this is fully supported in KRaft mode as of Kafka 3.7 (KIP-858). Note that if the directory holding the metadata log fails, the whole broker stops. XFS is the recommended filesystem for Kafka data directories for its performance with large files; mount with noatime to cut metadata writes:

mount -o noatime /dev/sdb1 /data/kafka-logs-1

Network bandwidth is the final dimension. A NIC must carry produce traffic, consume traffic, and inter-broker replication. With replication factor 3, each produced byte is written to the leader and replicated to two followers, so leader-side write-path traffic is roughly Produce_Throughput * 3 + Consume_Throughput. A 10 Gbps NIC tops out near 1.25 GB/s, but plan for 70–80% utilization to leave headroom for bursts and rebalances. In the cloud, bandwidth is often tied to instance type, and burstable network tiers are unsuitable for sustained Kafka load.

Multi-AZ and Rack Awareness Implementation

Deploying across availability zones is standard for production resilience but requires care in both infrastructure and Kafka configuration. The goal is that any single AZ failure causes neither data loss nor an outage, achieved through replica placement, client configuration, and network design.

Assign each broker a broker.rack that encodes its AZ:

broker.rack=us-east-1a

For a more granular model that also captures the physical rack inside the AZ:

broker.rack=us-east-1a-rack07

Kafka’s placement algorithm uses this when creating topics or reassigning partitions, spreading replicas across distinct rack IDs. For replication factor 3 across three AZs, each AZ holds one replica, so losing an AZ still leaves two in-sync replicas.

min.insync.replicas defines durability semantics. With acks=all, a write is acknowledged only after all in-sync replicas confirm it. With min.insync.replicas=2 and replication factor 3, the cluster tolerates one failed replica (and thus one AZ) and keeps accepting writes. If a second replica is lost, the partition drops below min-ISR and acks=all producers receive NotEnoughReplicas errors and must retry or buffer. That is the deliberate trade-off: brief unavailability for a subset of partitions in exchange for no data loss. Setting min.insync.replicas=1 keeps writes flowing with two replicas down but risks loss if the last replica fails before the others recover.

Two defaults reinforce that durability stance and are worth confirming explicitly. First, unclean.leader.election.enable is disabled by default: Kafka will not elect an out-of-sync replica as leader, so it favors a brief partition outage over silently dropping committed records. Leave it disabled for any topic where data loss is unacceptable; enabling it trades durability for availability. Second, on Kafka 4.0+ the controller supports Eligible Leader Replicas (ELR, KIP-966), which tracks replicas that fell out of the ISR but are still known to hold all committed data, so it can promote one of them after a min-ISR shortfall without resorting to an unclean election. ELR is enabled by default on new clusters from 4.1 (in 4.0 you opt in with eligible.leader.replicas.version=1); it narrows the window where a min-ISR drop forces a choice between outage and data loss, but it is not a substitute for replication factor 3 and min.insync.replicas=2.

Create a topic with the right durability settings:

kafka-topics.sh --bootstrap-server broker-1:9092 --create --topic critical-events \
  --partitions 12 --replication-factor 3 \
  --config min.insync.replicas=2

Alter an existing topic dynamically:

kafka-configs.sh --bootstrap-server broker-1:9092 --entity-type topics \
  --entity-name critical-events --alter --add-config min.insync.replicas=2

Inter-AZ latency affects producer throughput and replication. Measure round-trip time between AZs and keep it inside your latency budget. In most cloud regions inter-AZ RTT is under ~2 ms, fine for most workloads; across regions or hybrid links it can be 50 ms or more. In high-latency cases, raise the producer’s batch.size and linger.ms to amortize latency over larger batches, and tune follower fetch behavior with replica.fetch.min.bytes and replica.fetch.wait.max.ms so followers fetch larger batches.

Place the KRaft controller quorum on the same multi-AZ footprint: one controller per AZ in a three-AZ deployment, with broker.rack on controllers reflecting their AZ. Raft needs a majority available to elect a leader and commit entries, so three controllers tolerate one AZ failure.

Tiered Storage Architecture and Configuration

Tiered storage changes the economics of long-term retention. Without it, retention is bounded by the local disk you can afford per broker; with it, you keep recent data on fast local disks and offload the rest to cheap object storage, retaining months or years of history.

The RemoteStorageManager (RSM) interface defines how Kafka reads and writes the remote tier, and RemoteLogMetadataManager (RLMM) tracks remote-segment metadata. Kafka ships a default RLMM backed by an internal topic, plus a LocalTieredStorage RSM intended for testing; production deployments use a pluggable RSM such as Aiven’s open-source implementation or a vendor offering. Broker-side configuration enables the feature and points at the RSM/RLMM implementations. The official sample broker configuration looks like this:

remote.log.storage.system.enable=true
remote.log.metadata.manager.listener.name=PLAINTEXT
remote.log.storage.manager.class.name=org.apache.kafka.server.log.remote.storage.LocalTieredStorage
remote.log.storage.manager.class.path=/path/to/rsm-implementation.jar

Implementation-specific settings are passed under a configurable prefix. The RSM prefix defaults to remote.log.storage.manager.impl.prefix=rsm.config. and the RLMM prefix to remote.log.metadata.manager.impl.prefix=rlmm.config., so a third-party S3 RSM would read keys like rsm.config.s3.bucket.name that it defines itself. These rsm.config.* keys are not invented; they are the documented mechanism for forwarding configuration to the plugin.

Tiered storage is also a per-topic switch. After enabling it cluster-wide, turn it on for a topic and set the local retention window so recent data stays on disk while the full retention lives remotely:

kafka-configs.sh --bootstrap-server broker-1:9092 --entity-type topics \
  --entity-name events --alter --add-config \
  'remote.storage.enable=true,retention.ms=2592000000,local.retention.ms=86400000'

Here retention.ms (30 days) governs total retention, while local.retention.ms (24 hours) governs how long closed segments stay on local disk before deletion after upload. A segment becomes eligible for upload once it is closed (it reaches segment.bytes or segment.ms and is no longer active); the broker uploads eligible segments and only removes the local copy after local.retention.* expires, so data briefly lives in both tiers.

Two constraints catch teams off guard. Compacted topics cannot use tiered storage — it applies to delete-retention topics only — and you cannot shrink local.retention.* below the active segment, since the active segment is never uploaded. When a consumer reads tiered data, the broker fetches it from the remote store, adding tens to hundreds of milliseconds depending on the object store, so a fleet of consumers replaying from the cold tier drives remote-read load and cost, not local disk. Monitor tiered storage through the kafka.log.remote:type=RemoteLogManager MBeans, which expose copy/fetch counts, lag, and error rates. Alert on a rising rate of upload or fetch failures, which point to credential, permission, or connectivity problems, and watch local disk usage to confirm local.retention.* is expiring segments as intended.

Capacity Planning Methodology

Capacity planning is iterative: theoretical modeling plus empirical measurement, sized for steady-state and peak traffic and for failure scenarios where a broker or an AZ is down.

First characterize the workload: produce throughput in MB/s, average message size, connection counts, retention period, replication factor, and consumption pattern. Real-time (tail) consumers mostly hit the page cache; catch-up consumers hit disk. The mix drives disk I/O requirements.

Then compute storage:

Total_Storage = Ingress_MBps * Retention_Seconds * Replication_Factor * Overhead_Factor

An Overhead_Factor of 1.3 (30%) covers indices, internal topics, and filesystem slack. For 100 MB/s ingress, 7-day retention, and replication factor 3:

100 MB/s * 604800 s * 3 * 1.3 ≈ 235,872,000 MB ≈ 225 TB

Across 15 brokers that is ~15 TB usable per broker; with a 12-disk JBOD layout, ~1.25 TB per disk, easily met by 2 TB drives. Size the disks to the steady-state number above your alert threshold, not to 100% — at the recommended ~85% disk alert, that 15 TB working set wants closer to 18 TB of raw capacity per broker so a single broker failure, which shifts its replica leadership and catch-up reads onto the survivors, does not push them over the line.

Throughput is more nuanced. A 10 Gbps NIC caps near 1.25 GB/s. With replication factor 3, the leader’s replication egress alone is Ingress * (Replication_Factor - 1), so total leader traffic is Ingress + Ingress * (Replication_Factor - 1) + Consume_Throughput. If consume equals produce, that is roughly Ingress * 3, leaving a 10 Gbps broker around 400 MB/s of produce in a balanced scenario. Disk is the other constraint: a single SSD handles several hundred MB/s of sequential writes, and a 12-SSD JBOD reaches several GB/s aggregate, so network is usually the binding limit before disk.

Partition count is a frequently overlooked dimension. Each partition consumes memory for indices and file handles, so a broker with many partitions and segments can blow past the OS file-descriptor limit. Raise LimitNOFILE/ulimit -n to about 1,000,000 and monitor open handles. High partition counts also raise controller memory and election time. As a guideline, keep partitions per broker under ~4,000 for very high-throughput clusters, allowing more for lighter workloads. Size the partition count per topic up front to the target peak consumer parallelism — partitions are the unit of consumer concurrency, and while you can add partitions later, doing so breaks key-based ordering by remapping keys to new partitions, so over-splitting a key-ordered topic after the fact is not a clean fix.

Validate the plan with kafka-producer-perf-test.sh and kafka-consumer-perf-test.sh, which generate synthetic load and report throughput and latency:

kafka-producer-perf-test.sh --topic test-topic --num-records 100000000 \
  --record-size 1024 --throughput -1 --producer-props \
  bootstrap.servers=broker-1:9092,broker-2:9092,broker-3:9092 \
  acks=all compression.type=snappy

This writes 100 million 1 KB records with no rate limit, acks=all, and Snappy compression, reporting achieved throughput and latency percentiles. Run it with concurrency matching production and long enough to fill the page cache and reach steady state, so results validate the plan or expose a bottleneck.

Automated Provisioning with Terraform and Ansible

Manual provisioning breeds inconsistency. Terraform for infrastructure plus Ansible for configuration gives a repeatable, version-controlled, auditable build-and-scale process.

Terraform declares the cloud resources. A simplified AWS broker definition:

resource "aws_instance" "kafka_broker" {
  count         = var.broker_count
  ami           = var.kafka_ami
  instance_type = "i3en.3xlarge"
  subnet_id     = element(var.subnet_ids, count.index % length(var.subnet_ids))

  root_block_device {
    volume_size = 100
    volume_type = "gp3"
  }

  tags = {
    Name = "kafka-broker-${count.index + 1}"
    Role = "broker"
  }
}

resource "aws_ebs_volume" "data_disks" {
  count             = var.broker_count * var.data_disk_count
  availability_zone = aws_instance.kafka_broker[floor(count.index / var.data_disk_count)].availability_zone
  size              = 2000
  type              = "st1"

  tags = {
    Name = "kafka-data-${floor(count.index / var.data_disk_count) + 1}-${count.index % var.data_disk_count + 1}"
  }
}

resource "aws_volume_attachment" "data_disk_attachments" {
  count       = var.broker_count * var.data_disk_count
  device_name = "/dev/sd${element(split("", "fghijklmnop"), count.index % var.data_disk_count)}"
  volume_id   = aws_ebs_volume.data_disks[count.index].id
  instance_id = aws_instance.kafka_broker[floor(count.index / var.data_disk_count)].id
}

The subnet_id line spreads brokers round-robin across the subnets in var.subnet_ids; supply one subnet per AZ so the broker count distributes evenly across zones and the broker.rack rendered later matches the instance’s actual AZ. The i3en.3xlarge type includes NVMe instance storage usable for log directories alongside or instead of EBS. Instance storage is faster but ephemeral; EBS persists independently, so a failed broker can be replaced without losing data. Production clusters usually prefer EBS for that reason.

After Terraform provisions infrastructure, Ansible configures the OS and installs Kafka, driven by a dynamic inventory generated from Terraform output. A task to format and mount a data disk:

- name: Format data disk
  community.general.filesystem:
    fstype: xfs
    dev: /dev/sdf

- name: Mount data disk
  ansible.posix.mount:
    path: /data/kafka-logs-1
    src: /dev/sdf
    fstype: xfs
    opts: noatime
    state: mounted

The broker config is rendered from a Jinja2 template with variables for node ID, KRaft voters, listeners, and log directories:

node.id={{ node_id }}
process.roles=broker
controller.quorum.voters={{ kraft_voters }}
controller.listener.names=CONTROLLER
listeners=PLAINTEXT://{{ ansible_default_ipv4.address }}:9092,SSL://{{ ansible_default_ipv4.address }}:9093
advertised.listeners=PLAINTEXT://{{ ansible_default_ipv4.address }}:9092,SSL://{{ ansible_fqdn }}:9093
log.dirs={% for disk in data_disks %}/data/kafka-logs-{{ loop.index }}{% if not loop.last %},{% endif %}{% endfor %}

num.network.threads=8
num.io.threads=16
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
num.partitions=12
default.replication.factor=3
min.insync.replicas=2

Before first start, format the storage directory with the cluster ID (KRaft requires this):

KAFKA_CLUSTER_ID="$(kafka-storage.sh random-uuid)"
kafka-storage.sh format --cluster-id "$KAFKA_CLUSTER_ID" \
  --config /opt/kafka/config/server.properties

Generate the cluster ID once and reuse it across every node in the cluster — formatting each broker with a fresh ID produces nodes that refuse to join the same quorum. Then start Kafka under systemd:

[Unit]
Description=Apache Kafka Broker
After=network.target

[Service]
Type=simple
User=kafka
Group=kafka
ExecStart=/opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/server.properties
ExecStop=/opt/kafka/bin/kafka-server-stop.sh
Restart=on-failure
RestartSec=10
LimitNOFILE=1000000

[Install]
WantedBy=multi-user.target

Scaling becomes a matter of incrementing broker_count, reapplying Terraform, and rerunning Ansible against the new inventory. The whole flow fits a CI/CD pipeline where a pull request triggers terraform plan and a merge triggers apply. Tune kernel parameters in the same playbook — vm.swappiness=1 to avoid swapping the page cache and a raised net.core.rmem_max/net.core.wmem_max for network buffers.

Monitoring and Alerting for Cluster Health

Effective monitoring is the difference between catching a problem early and hearing about it from a support ticket. Kafka’s JMX MBeans are scraped by a JMX exporter that runs as a Java agent and exposes a Prometheus endpoint, configured with a YAML rules file:

lowercaseOutputName: true
rules:
- pattern: kafka.server<type=(.+), name=(.+)PerSec\w*><>Count
  name: kafka_server_$1_$2_per_sec
  type: COUNTER
- pattern: kafka.server<type=ReplicaManager, name=(.+)><>Value
  name: kafka_server_replicamanager_$1
  type: GAUGE
- pattern: kafka.network<type=RequestMetrics, name=(.+), request=(.+)><>(\d+)thPercentile
  name: kafka_network_request_$1
  labels:
    request: "$2"
    quantile: "0.$3"
  type: GAUGE

Start the broker with the agent:

export KAFKA_OPTS="-javaagent:/opt/jmx_exporter/jmx_prometheus_javaagent.jar=7071:/opt/jmx_exporter/config.yml"

Prometheus scrapes /metrics on port 7071 of each broker, and Grafana visualizes it. The essential panels:

• Throughput: kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec and BytesOutPerSec, aggregated cluster-wide.
• Request Latency: kafka.network:type=RequestMetrics,name=TotalTimeMs,request={Produce|FetchConsumer|FetchFollower}, at the 50th, 95th, and 99th percentiles.
• Under-Replicated Partitions: kafka.server:type=ReplicaManager,name=UnderReplicatedPartitions, which should stay at zero.
• Active Controller Count: kafka.controller:type=KafkaController,name=ActiveControllerCount. Summed across the cluster this must equal exactly 1; 0 means no active controller, and a sustained sum above 1 indicates a split-brain.
• Disk Usage: OS metrics via the Node Exporter; alert past ~85% on any data disk.
• JVM Heap and GC: heap usage and GC pause time, to catch memory pressure.

Why these specific signals: UnderReplicatedPartitions > 0 is the earliest broad indicator that a broker is failing, falling behind, or partitioned, and it fires before clients see errors; ActiveControllerCount != 1 is the one metric that distinguishes a healthy control plane from a stalled or split-brained one; and request-latency percentiles catch the slow-degradation failures (page-cache misses, GC, a saturated disk) that throughput alone hides. Define alerts for the conditions that need a human now:

groups:
- name: kafka_alerts
  rules:
  - alert: UnderReplicatedPartitions
    expr: kafka_server_replicamanager_UnderReplicatedPartitions > 0
    for: 5m
    labels:
      severity: critical
    annotations:
      summary: "Broker {{ $labels.instance }} has under-replicated partitions"
  - alert: NoActiveController
    expr: sum(kafka_controller_KafkaController_ActiveControllerCount) != 1
    for: 1m
    labels:
      severity: critical
    annotations:
      summary: "Cluster does not have exactly one active controller"
  - alert: BrokerDiskFull
    expr: (node_filesystem_avail_bytes{mountpoint="/data"} / node_filesystem_size_bytes{mountpoint="/data"}) < 0.15
    for: 5m
    labels:
      severity: critical
    annotations:
      summary: "Broker {{ $labels.instance }} disk usage above 85%"

Also monitor consumer group lag — the gap between the latest produced offset and the last committed offset. The kafka-consumer-groups.sh --describe command reports lag, and exporters such as Kafka Exporter or Burrow expose it to Prometheus for alerting when a group falls persistently behind.

Operational Procedures and Day-2 Management

Building the cluster is the start; the ongoing “Day-2” work needs documented, rehearsed procedures for scaling, upgrading, and rebalancing.

Scaling horizontally is common as throughput grows. Provision new brokers with the same automation and start them; they join the cluster but receive no partitions automatically. Use kafka-reassign-partitions.sh to move partitions onto them. First generate a plan:

kafka-reassign-partitions.sh --bootstrap-server broker-1:9092 \
  --topics-to-move-json-file topics.json \
  --broker-list "1,2,3,4,5" --generate

The topics.json file lists the topics to move:

{"topics": [{"topic": "events"}], "version": 1}

Review the proposed plan, save it, and execute it — throttling replication so the move does not starve live traffic:

kafka-reassign-partitions.sh --bootstrap-server broker-1:9092 \
  --reassignment-json-file reassignment.json --throttle 50000000 --execute

Reassignment runs in the background. Check progress with --verify, which reports whether each partition has reached its target replica set and, once the move is complete, removes the throttle that --execute applied (an important cleanup — a leftover throttle keeps limiting the cluster):

kafka-reassign-partitions.sh --bootstrap-server broker-1:9092 \
  --reassignment-json-file reassignment.json --verify

Run reassignments in a maintenance window or with a conservative throttle, since data movement adds load. The --throttle value is bytes/second per broker for replication moved by the reassignment; set it well below the network headroom you reserved in capacity planning, watch UnderReplicatedPartitions and request latency while it runs, and raise it gradually rather than starting unthrottled.

Upgrading Kafka uses rolling restarts, one broker at a time, with no full outage. Upgrade the binaries on a broker, restart it, wait for it to rejoin and for all under-replicated partitions to heal, then move on. In a KRaft cluster the feature level is governed by metadata.version, not the old ZooKeeper-era inter.broker.protocol.version/log.message.format.version (both removed in 4.0). Before and after, read the current feature levels with the describe subcommand:

kafka-features.sh --bootstrap-server broker-1:9092 describe

After all brokers run the new binaries and the cluster is verified healthy, finalize the upgrade to enable new features. Current Kafka favors upgrade --release-version, which raises every server feature to that release’s level in one step:

kafka-features.sh --bootstrap-server broker-1:9092 upgrade --release-version 4.1

metadata.version upgrades are a one-way operation in practice, so verify cluster health first; always follow the version-specific upgrade notes in the official Apache Kafka upgrade guide.

Rebalancing is also periodic maintenance: as topics churn and traffic shifts, some brokers go hot while others idle. Cruise Control automates this by analyzing CPU, disk, network, and partition-count metrics and generating an optimal, rack-aware reassignment plan, exposed through a REST API. Integrating it removes much of the manual effort of keeping a cluster balanced.

Backup and disaster recovery are easy to overlook because replication provides intra-cluster HA. But replication does not protect against accidental deletion, a corrupting bug, or loss of the whole cluster. MirrorMaker 2 — a Kafka Connect-based tool — replicates data, consumer offsets, and topic configuration to a secondary cluster in another region for failover. Note that MirrorMaker 2 is asynchronous, so a region-loss failover can lose the records not yet mirrored, and consumer-offset translation across clusters is approximate — plan the failover runbook around at-least-once, possibly-duplicated delivery, not exactly-once. Tiered storage can complement this if the object store is versioned and cross-region replicated, though it never holds the newest, not-yet-uploaded data and so is not a complete backup on its own.

Security spans every layer. Kafka supports TLS for encryption in transit, SASL for authentication, and ACLs for authorization. Enable TLS for client-to-broker and inter-broker traffic, authenticate with SASL/SCRAM or SASL/OAUTHBEARER against an external identity provider, and apply least-privilege ACLs so producers only write to their topics and consumers only read theirs. Manage ACLs with kafka-acls.sh:

kafka-acls.sh --bootstrap-server broker-1:9092 --add \
  --allow-principal User:producer-app --operation Write --topic events

ACLs also cover consumer groups, transactional IDs, and cluster resources. At scale, manage them through automation tied to your identity provider rather than by hand.

The last piece is documentation. Every procedure — scaling, upgrading, rebalancing, replacing a failed disk or broker — belongs in a runbook accessible to on-call, with step-by-step commands, expected output, and troubleshooting for common failures. Exercise these runbooks through game days: terminate an AZ’s instances and confirm the cluster recovers and the alerts fire. That practice builds confidence and surfaces gaps before a real incident does.

The Apache Kafka ecosystem evolves quickly, so staying current with releases, security patches, and KIPs is ongoing work. The Apache Kafka documentation is the authoritative reference for configuration and upgrades, and the Kafka Improvement Proposals (KIPs) explain the design behind protocol and broker changes. Pairing a solid architectural foundation with rigorous operations gives you a Kafka platform that is resilient, scalable, and manageable.