Managing Topic Configuration at Scale

In production Apache Kafka environments, topic counts grow from a handful to hundreds or thousands. Each topic carries its own settings—partition count, replication factor, retention, compaction, compression—and managing them by hand quickly becomes unsustainable. The failure mode is drift: configs that no longer match any documented intent, applied inconsistently across topics with very different requirements. This article covers the operational patterns, automation, and governance needed to manage topic configuration at scale for platform, SRE, and backend teams.

Topic configuration does not exist in isolation. The settings you choose shape how data flows through your Topics, Partitions & Data Modeling architecture, and they interact directly with Partitioning Strategies for High-Throughput Topics and Keying Strategies for Ordered Message Processing. A misconfigured topic can silently break ordering guarantees or create hot partitions that throttle an entire pipeline.

The Configuration Surface Area

Before automating anything, understand what you are managing. Kafka exposes roughly two dozen topic-level configs (see the topic configs reference), and they fall into a few operationally distinct groups.

Durability and availability: replication.factor is set at creation; min.insync.replicas and unclean.leader.election.enable are alterable at runtime. With a replication factor of 3, set min.insync.replicas=2 so a single broker failure does not block acks=all writes. Keep unclean.leader.election.enable=false (the broker default) to trade availability for durability—an out-of-sync replica must never be elected leader on a topic where you cannot afford data loss.

Retention and storage: retention.ms and retention.bytes bound how much data a partition keeps; segment.ms and segment.bytes control segment rolling, which in turn gates when old data becomes eligible for deletion. For compacted topics, cleanup.policy, min.cleanable.dirty.ratio, delete.retention.ms, and min.compaction.lag.ms govern compaction behavior. A common scale mistake is applying one retention policy across topics with wildly different volumes, producing lopsided disk usage across brokers.

Performance: compression.type, segment.bytes, segment.index.bytes, and the batch-flush controls (flush.messages, flush.ms). The compression.type default is producer, which preserves whatever codec the producer used and avoids broker-side recompression. Setting it to an explicit codec such as zstd, lz4, snappy, or gzip forces the broker to recompress on write, costing CPU—do that deliberately, not by accident.

Replication throttling: leader.replication.throttled.replicas and follower.replication.throttled.replicas mark which replicas are throttled; the actual rate caps (leader.replication.throttled.rate and follower.replication.throttled.rate) are broker-level configs. Together they keep partition reassignment traffic from saturating links and degrading client performance.

To inspect the real surface area for a topic, including both overrides and the defaults they shadow:

kafka-configs --bootstrap-server kafka-broker-1:9092 \
  --entity-type topics --entity-name orders.v3 \
  --describe --all

The --all flag lists every config with its synonym chain, so you can see exactly which override is winning and what it shadows:

All configs for topic orders.v3 are:
  retention.ms=604800000 sensitive=false synonyms={DYNAMIC_TOPIC_CONFIG:retention.ms=604800000, DEFAULT_CONFIG:log.retention.ms=604800000}
  compression.type=zstd sensitive=false synonyms={DYNAMIC_TOPIC_CONFIG:compression.type=zstd, DEFAULT_CONFIG:compression.type=producer}

Synonyms are listed in precedence order—the first entry is the effective value—which makes this command the fastest way to confirm whether a setting is a topic override, a broker default, or the hard-coded Kafka default.

Configuration as Code: Declarative Topic Management

The foundational pattern at scale is to treat topic definitions as code. Rather than running ad-hoc kafka-topics and kafka-configs commands, define topics declaratively in version-controlled YAML or JSON and apply them through a reconciliation pipeline.

topics:
  - name: orders.v3
    partitions: 24
    replication_factor: 3
    configs:
      retention.ms: 604800000
      compression.type: zstd
      min.insync.replicas: 2
      cleanup.policy: delete
      unclean.leader.election.enable: false
  - name: inventory-snapshots
    partitions: 12
    replication_factor: 3
    configs:
      cleanup.policy: compact
      min.cleanable.dirty.ratio: 0.5
      segment.ms: 86400000
      min.insync.replicas: 2

A reconciler applies these definitions, handling three cases: create new topics, update configs on existing ones, and flag topics present in the cluster but absent from the desired state for human review. The example below uses kafka-python. Note the API details—they are a frequent source of bugs:

import yaml
from kafka.admin import KafkaAdminClient, NewTopic, ConfigResource, ConfigResourceType
from kafka.errors import TopicAlreadyExistsError

def reconcile_topics(bootstrap_servers, desired_state_file):
    with open(desired_state_file) as f:
        desired = yaml.safe_load(f)

    admin = KafkaAdminClient(bootstrap_servers=bootstrap_servers)
    existing_topics = set(admin.list_topics())

    new_topics, alter_resources = [], []
    for topic_def in desired['topics']:
        name = topic_def['name']
        configs = {k: str(v) for k, v in topic_def['configs'].items()}
        if name not in existing_topics:
            new_topics.append(NewTopic(
                name=name,
                num_partitions=topic_def['partitions'],
                replication_factor=topic_def['replication_factor'],
                topic_configs=configs,
            ))
        else:
            alter_resources.append(
                ConfigResource(ConfigResourceType.TOPIC, name, configs=configs)
            )

    if new_topics:
        admin.create_topics(new_topics)
    if alter_resources:
        admin.alter_configs(alter_resources)

A few non-obvious points that trip people up:

• alter_configs takes a list of ConfigResource objects, not a {topic: configs} dict. Each resource’s configs is a dict of key -> value (or key -> (op, value) tuples).
• In kafka-python, alter_configs defaults to the incremental path when the broker supports it (Kafka 2.3+ via KIP-339); you can force it with incremental=True. There is no separate incremental_alter_configs method. This matters because the legacy (full-replace) AlterConfigs API overwrites every dynamic override on the resource—any key you omit is reset to its default. If you ever target a pre-2.3 broker or pass incremental=False, you must send the complete config set.
• The CLI equivalent of the incremental path is kafka-configs --alter with --add-config and --delete-config, which only touch the keys you name.

Every change now goes through code review, is auditable via git history, and is reversible by reverting a commit. This pattern is independent of the metadata backend: KIP-500 has been delivered—KRaft was marked production-ready in Kafka 3.3 (KIP-833) and ZooKeeper mode was removed in Kafka 4.0—but topic configuration is applied through the Admin API either way.

Automating Partition Count Decisions

Partition count is the most consequential topic decision and one of the hardest to walk back. Under-partitioning caps throughput and consumer parallelism; over-partitioning inflates metadata, increases end-to-end latency, and lengthens failover and recovery. At hundreds of topics you cannot eyeball this—you need a data-driven default.

A workable starting formula sizes partitions from target throughput and a conservative per-partition ceiling:

partitions = max(ceil(target_throughput_mbps / per_partition_mbps), min_partitions)

A per-partition target of around 10 MB/s is a defensible default, but it depends heavily on hardware, message size, replication factor, and consumer processing cost—measure rather than assume. Enforce a floor (often 6 or 12) so even low-volume topics retain headroom for consumer parallelism.

To feed real numbers into that formula, read the per-topic ingress rate from JMX. The BytesInPerSec MBean exposes a per-topic variant via a topic= key:

# Sample the bytes-in rate for one topic every 60s.
# kafka-jmx.sh ships in bin/ and wraps org.apache.kafka.tools.JmxTool.
kafka-jmx.sh \
  --object-name 'kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec,topic=orders.v3' \
  --jmx-url service:jmx:rmi:///jndi/rmi://kafka-broker-1:9999/jmxrmi \
  --attributes OneMinuteRate \
  --reporting-interval 60000

Note the tooling change: the old kafka-run-class kafka.tools.JmxTool invocation was removed in Kafka 4.0. JmxTool now lives at org.apache.kafka.tools.JmxTool, and the supported wrapper is kafka-jmx.sh. The --reporting-interval value is in milliseconds, so 60000 samples once a minute. For real sizing, don’t trust a one-off sample—pull historical peak throughput from your monitoring system (Prometheus, Datadog) and size to peak, not average.

When you do increase partitions, remember that ordering is guaranteed only within a partition. Adding partitions changes the key-to-partition mapping for the default partitioner, so keyed events that used to land together can scatter. If your application depends on ordering, coordinate the increase with consumer logic or accept a one-time reordering window—see Keying Strategies for Ordered Message Processing. A poorly chosen key turns repartitioning into a breaking change.

Governance Through Tiered Topic Policies

Not all topics deserve the same guarantees. Payment events need stronger durability and retention than ephemeral clickstream. A small set of pre-approved tiers reduces decision fatigue and prevents one-off misconfiguration.

Define three to five tiers, each a fixed configuration template:

Enforce tiers in CI. When a team submits a topic definition, a pipeline step validates that its configs match an approved tier before the change can merge:

import yaml, sys

TIER_POLICIES = {
    'critical': {
        'replication_factor': 3,
        'min.insync.replicas': 2,
        'unclean.leader.election.enable': 'false',
    },
    'standard': {
        'replication_factor': 3,
        'min.insync.replicas': 2,
    },
}

def validate_topic(topic_def):
    tier = topic_def.get('tier', 'standard')
    policy = TIER_POLICIES.get(tier)
    if not policy:
        print(f"ERROR: unknown tier '{tier}' for topic {topic_def['name']}")
        return False
    flat = {**topic_def, **topic_def.get('configs', {})}
    ok = True
    for key, expected in policy.items():
        actual = flat.get(key)
        if str(actual) != str(expected):
            print(f"ERROR: {topic_def['name']} {key}={actual}, expected {expected}")
            ok = False
    return ok

with open(sys.argv[1]) as f:
    topics = yaml.safe_load(f)['topics']

sys.exit(0 if all(validate_topic(t) for t in topics) else 1)

(Note that replication_factor is a top-level field in the definition while the others live under configs, so the check flattens both before comparing.) The same tier metadata also drives auditing: periodic drift detection can compare live configs against both the desired state and the tier policy, and alert on any deviation. Confluent’s topic configuration reference is a useful companion for reasoning about broker-default versus topic-override precedence.

Handling Configuration Drift at Runtime

Even with declarative management and CI validation, drift happens. Someone applies an emergency override directly via CLI, or a config is changed in one environment and never backported. Catching it requires continuous reconciliation.

Run a drift detector on a schedule (say every 15 minutes) that diffs the live cluster against the desired state in git, alerts on any discrepancy, and—for low-risk tiers only—optionally reverts. A minimal version:

#!/bin/bash
set -euo pipefail
DESIRED_STATE="topic-definitions/production.yaml"
DRIFT_REPORT="/tmp/kafka-drift-$(date +%Y%m%d-%H%M).txt"

# Dump live topic configs (omit --entity-name to describe all topics).
kafka-configs --bootstrap-server kafka-broker-1:9092 \
  --entity-type topics --describe --all > /tmp/live-configs.txt

# Diff against your rendered desired state. kafka-gitops is one third-party
# option; its plan output is human-readable and its exit code is non-zero on drift.
kafka-gitops plan -f "$DESIRED_STATE" > "$DRIFT_REPORT" || true

if [ -s "$DRIFT_REPORT" ]; then
  echo "Drift detected. See $DRIFT_REPORT"
  # Auto-remediation is appropriate only for low-risk topics; gate it carefully.
  # kafka-gitops apply -f "$DESIRED_STATE"
fi

kafka-gitops (by devshawn) is a real GitOps tool for Kafka, but treat its exact subcommands and flags as version-specific and confirm them against its docs—do not copy a flag from a blog post into production. In particular, it has no built-in --tier selector; it operates on the topics declared in its state file, and it reads its broker connection from configuration/environment rather than a --bootstrap-server flag. Scope auto-remediation by maintaining separate state files (or branches) per tier, not by inventing a filter flag that does not exist.

For critical topics, automated reversion is usually too risky: a “drift” might be a deliberate emergency fix. Route those alerts to on-call with full context so an engineer can decide whether to revert or to promote the change into the desired state.

Scaling Topic Management with Self-Service

As the org grows, a platform team that hand-approves every topic becomes the bottleneck. A self-service model with guardrails lets teams create topics within policy while the platform retains control.

Expose self-service as a CLI, a CI/CD template, or an internal UI. A team specifies tier, estimated throughput, and retention; the tooling renders a topic definition, validates it against tier policy, and opens a pull request against the topic-definitions repo. A thin wrapper CLI might look like this:

# Custom platform tool — NOT a bundled Kafka command. It renders a topic
# definition and opens a PR; CI runs the tier validator before merge.
ktopic request \
  --name team-a.clickstream \
  --tier standard \
  --estimated-throughput-mbps 25 \
  --retention-days 3 \
  --owner team-a@company.com

There is no bundled Kafka CLI for this; the wrapper is something your platform team builds. Under the hood it computes partition count from throughput, applies the tier template, and—on merge—the reconciler from the first section calls the Admin API (create_topics, alter_configs) to realize it. Platform engineers review only the unusual requests (a critical topic for a non-prod service, an outsized partition count); standard requests merge automatically once CI passes.

This scales topic management to hundreds of developers without losing consistency, and it closes a feedback loop: when a topic hits a throughput limit, monitoring can recommend a partition increase that the owning team submits through the same flow.

Monitoring and Alerting on Topic Health

Configuration management is incomplete without observability into how those configs behave. Track both the configuration state and its runtime outcomes. Per-topic signals worth alerting on:

• Under-replicated partitions — durability risk, usually from broker failure or throttled replication.
• Offline partitions — page immediately.
• Producer request rate and throughput — confirms partition counts are still adequate.
• Consumer lag — surfaces under-partitioning or slow consumers.
• Disk usage per topic — confirms retention is doing its job and no topic is hogging storage.

Tie alerts back to configuration decisions. The two rules below illustrate the pattern:

groups:
  - name: kafka_topic_health
    rules:
      - alert: TopicConsumerLagHigh
        expr: sum(kafka_consumergroup_lag{consumergroup="critical-services"}) by (topic) > 100000
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "Consumer lag for topic {{ $labels.topic }} exceeds 100k"
      - alert: UnderReplicatedPartitions
        expr: kafka_server_replicamanager_underreplicatedpartitions > 0
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Cluster has under-replicated partitions"

Metric names depend on which exporter you run, and the two rules above deliberately come from two different exporters:

• Under-replicated partitions comes from the broker’s own JMX (kafka.server:type=ReplicaManager,name=UnderReplicatedPartitions). Scraped through the Prometheus JMX Exporter with the Kafka sample config, it becomes kafka_server_replicamanager_underreplicatedpartitions. This is a broker-wide gauge, not per-topic.
• Consumer-group lag is not a broker JMX metric. The JMX Exporter cannot produce per-topic, per-group lag. Use the separate Kafka Exporter, which emits kafka_consumergroup_lag labeled by consumergroup, topic, and partition. Pin your alert expressions to whatever your deployment actually scrapes.

The Kafka monitoring documentation is the authoritative list of broker MBeans and their meanings. Group dashboards by tier and owner so that when something breaks, you can immediately see which class of topic and which team are affected.

Conclusion

Managing topic configuration at scale is a continuous practice, not a one-time setup. Declarative definitions give you review and rollback; automated partition sizing replaces guesswork with measurement; tiered policies and CI validation keep configs within known-good bounds; drift detection and self-service workflows keep the system honest as it grows. The most important habit is the one this article keeps returning to: verify each command, flag, config key, and metric name against current documentation before it reaches production—the tooling moves (JmxTool’s package, KRaft replacing ZooKeeper, incremental config APIs), and stale commands are how outages start.