Topics, Partitions & Data Modeling

The Operational Anatomy of a Kafka Topic

A Kafka topic is a distributed, partitioned, append-only log. For platform and SRE engineers, understanding the physical layout of a topic is the prerequisite for every operational decision, from capacity planning to failure recovery. Logically a topic is a named feed of records; physically it is a collection of partitions spread across the broker cluster. Each partition is an ordered, immutable sequence of records that is continually appended to, and each record within a partition is assigned a sequential offset that uniquely identifies it within that partition.

The append-only nature of the partition log is the cornerstone of Kafka’s performance. Because writes are always sequential, brokers achieve high throughput on modest hardware, often saturating disk or network I/O before CPU becomes the bottleneck. This design also simplifies replication: each partition has one leader broker and zero or more follower brokers that replicate the leader’s log. All producer writes and consumer reads for a partition go through the leader, which is what makes per-partition ordering possible. When you create a topic with a replication factor of three, you instruct the cluster to maintain three copies of each partition’s log across distinct brokers, giving you resilience against the loss of up to two of those brokers.

The relationship between topics, partitions, and the storage layer is critical for capacity planning. Each partition replica maps to a directory on the broker’s filesystem containing segment files (.log), index files (.index and .timeindex), and related metadata. Segment size is controlled by the broker-level log.segment.bytes configuration (default 1 GiB, 1073741824) or the topic-level segment.bytes override. When the active segment reaches this size, or segment.ms elapses, Kafka rolls a new segment, and rolled segments become eligible for deletion or compaction based on the topic’s cleanup.policy. Because every replica holds open file handles for its segments and indexes, partition count directly drives the number of open file descriptors on a broker, a common source of production incidents when teams create thousands of partitions without raising OS nofile limits.

# Create a topic with explicit partition count, replication factor, and segment size
kafka-topics.sh --bootstrap-server kafka-broker-1:9092 \
  --create \
  --topic orders.v2.ecommerce \
  --partitions 12 \
  --replication-factor 3 \
  --config segment.bytes=536870912 \
  --config retention.ms=604800000

The operational burden of a partition extends beyond file handles. Each partition leader maintains in-memory offset indexes, tracks producer idempotency state, and coordinates fetch requests from followers and consumers. The controller, elected via ZooKeeper in older deployments or via KRaft in modern ones, tracks the leadership and ISR state of every partition and orchestrates leader elections during broker failures. The cost of a high partition count shows up most painfully when the controller itself fails: it must rebuild cluster metadata and re-elect leaders for every affected partition, and that recovery window grows with total partition count. For ZooKeeper-based clusters, Confluent’s long-standing guidance is a budget of roughly 4,000 partitions per broker and 200,000 partitions per cluster — limits driven mainly by controller failover time, not steady-state throughput. KRaft (KIP-500) raises these ceilings by an order of magnitude, because the controller keeps metadata in a replicated log and reads it incrementally rather than reloading it all from ZooKeeper on failover. Either way, the lesson is the same: treat partition count as a budgeted resource and size it against parallelism needs, not as something to over-provision by reflex.

Partition Mechanics: Parallelism, Ordering, and Durability

Partitions are the unit of parallelism in Kafka, and their behavior governs three system properties: throughput scalability, message ordering, and durability. Within a consumer group, each partition is assigned to at most one consumer instance, so the maximum effective parallelism of a group is bounded by the partition count of the topics it consumes. If a topic has 6 partitions and you deploy a group with 12 instances, 6 instances sit idle. With 12 partitions and 6 consumers, each consumer is assigned 2 partitions, and you can scale to 12 consumers before hitting the ceiling.

This consumer-to-partition mapping is managed by the group coordinator and the partition assignment strategy configured on the consumer. The default since the classic rebalance protocol is RangeAssignor, which assigns partitions on a per-topic basis and can produce uneven load when a group subscribes to multiple topics. StickyAssignor tries to preserve existing assignments across rebalances to reduce state migration. CooperativeStickyAssignor implements incremental cooperative rebalancing (KIP-429): instead of every consumer revoking all partitions at once, only the partitions that must move are revoked, so consumers keep processing the rest while the rebalance proceeds. For stateful Kafka Streams applications, where each revoked partition can trigger expensive state restoration, the assignment strategy directly affects recovery time and tail latency during scaling events.

# Consumer configuration for incremental cooperative rebalancing
group.id=order-processor-v3
partition.assignment.strategy=org.apache.kafka.clients.consumer.CooperativeStickyAssignor
session.timeout.ms=45000
heartbeat.interval.ms=15000
max.poll.interval.ms=300000

Message ordering is guaranteed only within a single partition; Kafka makes no ordering promises across partitions of the same topic. This is a deliberate trade-off: by relaxing global ordering, Kafka scales writes and reads horizontally. When an application requires strict ordering for related events — for example, all state changes for one account must be processed in the sequence they occurred — those events must be routed to the same partition. That is achieved through message keying, covered in Keying Strategies for Ordered Message Processing.

Durability is a function of the replication protocol and the producer acknowledgment setting. With acks=all (equivalently acks=-1), the leader does not acknowledge a produce request until all replicas currently in the in-sync replica (ISR) set have written the record. This is the strongest durability guarantee: the record survives leader loss because at least one in-sync follower holds a committed copy. The cost is latency bounded by the slowest in-sync replica. The min.insync.replicas configuration sets the minimum ISR size required to accept an acks=all write. Setting min.insync.replicas=2 on a replication-factor-3 topic lets the topic tolerate one ISR member being down while still accepting writes. If the ISR shrinks below min.insync.replicas, the leader rejects acks=all writes with a NotEnoughReplicas (or NotEnoughReplicasAfterAppend) error, halting writes to protect consistency while reads and already-committed data remain available.

# Describe a topic to inspect partition leadership and in-sync replica status
kafka-topics.sh --bootstrap-server kafka-broker-1:9092 \
  --describe \
  --topic orders.v2.ecommerce

# Example output:
# Topic: orders.v2.ecommerce  PartitionCount: 12  ReplicationFactor: 3
#   Topic: orders.v2.ecommerce  Partition: 0  Leader: 3  Replicas: 3,1,2  Isr: 3,1,2
#   Topic: orders.v2.ecommerce  Partition: 1  Leader: 1  Replicas: 1,2,3  Isr: 1,2

In this output, Partition 1 has an ISR of [1,2] while its assigned replicas are [1,2,3]; broker 3 has fallen behind and dropped out of the ISR. With min.insync.replicas=2, writes to Partition 1 continue. With min.insync.replicas=3, acks=all writes are blocked until broker 3 rejoins the ISR. This availability-versus-consistency tension is a recurring theme in Kafka operations, and the right setting depends on whether the workload prioritizes write availability or data safety.

Data Modeling Patterns for Stream Processing

Data modeling in Kafka differs fundamentally from relational modeling. Instead of normalizing state into tables with foreign keys, you model events as immutable facts that capture what happened at a point in time. Each event should be self-contained enough for downstream consumers to interpret it. This shift from state-oriented to event-oriented thinking forces deliberate choices about topic granularity, schema design, and the relationships between events across topics.

Three patterns dominate: event notification, event-carried state transfer, and event sourcing. Event notification emits a lightweight record signaling that something happened; consumers call back to the producing service’s API for the full state. This minimizes duplication but couples services and creates availability dependencies. Event-carried state transfer embeds the relevant state (snapshot or delta) in the event, letting consumers operate without callbacks. Messages are larger, but consumers become autonomous. Event sourcing treats the log as the system of record: all state changes are events, and current state is derived by replaying them. It yields a complete audit trail and temporal queries but demands disciplined schema evolution and idempotent processing.

When designing event schemas, consider both the logical structure and the serialization format. Apache Avro is common in the Kafka ecosystem for its compact binary encoding, well-defined schema evolution rules, and integration with the Confluent Schema Registry. Protobuf and JSON Schema are also supported by the registry, with different trade-offs around readability, performance, and tooling. Regardless of format, events typically carry envelope metadata such as event_id, event_timestamp, event_type, and schema_version, which enable deduplication, temporal ordering, content-based routing, and compatibility checks. For deeper treatment, see Designing Event-Driven Schemas for Kafka Topics.

{
  "event_id": "evt_9a7b3c4d",
  "event_timestamp": "2025-03-15T14:22:31.442Z",
  "event_type": "order.placed",
  "schema_version": "2.1.0",
  "payload": {
    "order_id": "ord_8821",
    "customer_id": "cust_4409",
    "items": [
      {"sku": "PRD-001", "quantity": 2, "unit_price": 29.99},
      {"sku": "PRD-087", "quantity": 1, "unit_price": 149.50}
    ],
    "total_amount": 209.48,
    "currency": "USD",
    "payment_method": "credit_card"
  }
}

A key decision is one topic per event type versus a shared topic carrying multiple types. Single-type topics give clear ownership boundaries, independent retention, and simpler consumer logic, at the cost of topic proliferation that complicates governance and discovery. Shared topics with a well-defined envelope reduce topic count but require consumers to filter by type, wasting bandwidth and CPU on uninteresting records. Many teams adopt a hybrid: coarse-grained domain topics (orders, payments, shipments) with an event_type field for routing, plus dedicated topics for high-volume or latency-sensitive streams. This is where Topic Naming Conventions and Governance Best Practices become essential as the architecture scales.

Partitioning Strategies and Key Design

The partitioning strategy determines how evenly load spreads across partitions and how effectively consumers parallelize. For a keyed record, the producer hashes the key with murmur2 and maps it to a partition, so records sharing a key always land on the same partition (as long as the partition count is unchanged). This keyed behavior has been stable across versions and is what gives you per-key ordering.

The handling of null-key (keyless) records has changed, and it is worth getting right. The original DefaultPartitioner and UniformStickyPartitioner are deprecated as of Kafka 3.0 (KIP-794). Since 3.3, when you leave partitioner.class unset, the producer uses a built-in partitioner that applies sticky partitioning to keyless records: it sends records to one partition until the current batch is sent, then picks another partition, rather than rotating per-record as a naive round-robin would. KIP-794 made this sticky logic adaptive to broker load. If you want the older strict-uniform behavior for keyless records, set partitioner.ignore.keys=true rather than naming a deprecated partitioner class. In practice this means keyless throughput is distributed evenly across partitions over time, with better batching than per-record round-robin.

When you need to scale beyond the write throughput a single partition leader can absorb (bounded largely by the leader broker’s network and disk), you add partitions. Throughput scales roughly linearly with partition count until you hit broker resource limits. A deeper exploration is in Partitioning Strategies for High-Throughput Topics.

Custom partitioners implement the org.apache.kafka.clients.producer.Partitioner interface. They are useful for routing on a composite key, tenant isolation, or applying business logic to placement. A multi-tenant platform might partition by tenant_id so all of a tenant’s events share a partition, giving per-tenant ordering while parallelizing across tenants. The trade-off is operational: a custom partitioner must be deployed to every producer, and changing its logic can break ordering for records already in flight.

// Custom partitioner that routes by tenant_id extracted from the record value
public class TenantAwarePartitioner implements Partitioner {
    @Override
    public int partition(String topic, Object key, byte[] keyBytes,
                         Object value, byte[] valueBytes, Cluster cluster) {
        // Parse tenant_id from the serialized value (simplified for illustration)
        String tenantId = extractTenantId(valueBytes);
        int numPartitions = cluster.partitionCountForTopic(topic);
        return Math.floorMod(tenantId.hashCode(), numPartitions);
    }

    @Override
    public void close() {}

    @Override
    public void configure(Map<String, ?> configs) {}
}

cluster.partitionCountForTopic(topic) returns an Integer (null if metadata is absent); using Math.floorMod avoids the negative-result edge case that Math.abs(hashCode()) % n hits when hashCode() is Integer.MIN_VALUE.

Partition count is a one-way door. You can increase a topic’s partitions but never decrease them. Increasing partitions changes the key-to-partition mapping for records produced after the change, so the same key may move to a new partition, breaking per-key ordering across the boundary. This is why you provision partitions for expected peak throughput plus a growth buffer rather than starting at one partition and growing later. When broker load (not partition count) becomes uneven, use Automated Repartitioning with Kafka Cruise Control to rebalance partition leadership and placement across brokers.

Retention Policies and Log Compaction

Retention determines how long records remain in a topic and when they become eligible for cleanup. The two delete-policy strategies, time-based and size-based, combine to bound a partition’s log. Time-based retention (retention.ms) caps record age; size-based retention (retention.bytes) caps per-partition log size, deleting the oldest segments first. When both are set, a segment is eligible once either limit is exceeded. Both apply only to topics with cleanup.policy=delete (the default).

Delete policies suit streams whose value decays over time: metrics, logs, clickstream, transient notifications. For stateful use cases where you need the latest value per key, log compaction (cleanup.policy=compact) offers a different model: a compacted topic retains at least the latest record for each key and reclaims superseded records. This lets a topic act as a durable, replayable key-value snapshot. The choice is not strictly binary; you can set cleanup.policy=compact,delete to compact and also age out old data. For a full comparison, see Retention vs Compaction: When to Use Each.

# Create a compacted topic for a stateful stream-processing changelog
kafka-topics.sh --bootstrap-server kafka-broker-1:9092 \
  --create \
  --topic customer-profiles.changelog \
  --partitions 24 \
  --replication-factor 3 \
  --config cleanup.policy=compact \
  --config min.cleanable.dirty.ratio=0.5 \
  --config segment.ms=3600000 \
  --config delete.retention.ms=86400000

Compaction operates on closed (non-active) segments. The log cleaner, a pool of background threads on each broker, scans these segments and removes records whose key has a newer record elsewhere in the log. min.cleanable.dirty.ratio controls aggressiveness: at 0.5, a log becomes eligible for compaction once roughly half of it (by size) consists of records that could be cleaned. Lower values compact more often, trading I/O for lower disk usage. delete.retention.ms controls how long a tombstone (a record with a null value, signaling key deletion) is retained after compaction before final removal. This matters for correctness: if a tombstone is removed before a lagging or restarting consumer reads it, that consumer never learns the key was deleted and keeps stale state.

Compacted topics underpin stateful processing in Kafka Streams and ksqlDB. When a Streams application runs a stateful operation such as an aggregation or join, it materializes state into a local store (RocksDB by default) backed by a compacted changelog topic. If an instance fails and restarts elsewhere, it replays the changelog to rebuild local state. See Using Log Compaction for Stateful Stream Processing. Operationally, this means compacted topics need monitoring: if the cleaner falls behind, changelog topics grow and state-restoration times climb. Kafka exposes cleaner health via JMX, including kafka.log:type=LogCleaner,name=max-clean-time-secs and kafka.log:type=LogCleanerManager,name=uncleanable-partitions-count. A non-zero, sustained uncleanable-partitions-count is the strongest signal that compaction is stuck on a specific partition.

# Prometheus alert rule for stuck log compaction.
# Metric names assume the standard jmx_exporter mapping
# (MBean -> lowercased, dots/dashes -> underscores); verify against your exporter config.
groups:
  - name: kafka_log_cleaner
    rules:
      - alert: LogCleanerUncleanablePartitions
        expr: kafka_log_logcleanermanager_uncleanable_partitions_count > 0
        for: 15m
        labels:
          severity: warning
        annotations:
          summary: "Log cleaner cannot compact one or more partitions"
          description: "Broker {{ $labels.instance }} reports {{ $value }} uncleanable partition(s); changelog/compacted topics may grow unbounded"

Producer and Consumer Configuration for Production Workloads

Producer and consumer behavior is governed by dozens of parameters whose defaults favor safety over peak performance. In production you tune them to the workload’s latency, throughput, and durability needs. The most impactful producer settings are acks, linger.ms, batch.size, compression.type, and max.in.flight.requests.per.connection.

acks sets the durability floor. acks=0 waits for nothing and does not retry on send failure, acceptable only where occasional loss is fine. acks=1 waits for the leader’s local write, surviving producer failures but not the loss of the leader before replication. acks=all waits for every in-sync replica, the strongest guarantee, at the cost of latency. For business-critical data, acks=all with min.insync.replicas=2 and replication.factor=3 is the standard durable-yet-available baseline. Note that since Kafka 3.0, the producer defaults to acks=all and enable.idempotence=true; set them explicitly to document intent.

Throughput tuning centers on batching and compression. linger.ms adds a small delay so more records accumulate per batch; 5–10 ms often raises throughput substantially with negligible latency impact. batch.size caps batch size in bytes per partition. Compression shrinks batches on the wire and on disk in exchange for CPU: lz4 favors speed, snappy is a balanced default, and zstd (KIP-110) gives the best ratio at higher CPU cost.

# High-throughput producer configuration
bootstrap.servers=kafka-broker-1:9092,kafka-broker-2:9092,kafka-broker-3:9092
acks=all
compression.type=zstd
linger.ms=10
batch.size=131072
max.in.flight.requests.per.connection=5
enable.idempotence=true
max.request.size=2097152
delivery.timeout.ms=120000
request.timeout.ms=30000

With enable.idempotence=true, max.in.flight.requests.per.connection may be at most 5; the idempotent producer preserves ordering and deduplicates retries within that bound. On the consumer side the impactful settings are fetch.min.bytes, fetch.max.wait.ms, max.partition.fetch.bytes, and max.poll.records. Raising fetch.min.bytes above its default of 1 makes the broker wait until enough data accrues (or fetch.max.wait.ms elapses) before responding, cutting round-trips at the cost of some latency. max.poll.records bounds how many records poll() returns, which in turn bounds how long the application works between polls. If processing a batch exceeds max.poll.interval.ms, the consumer is evicted and the group rebalances, a frequent cause of rebalance storms.

# Consumer configuration optimized for throughput with moderate latency tolerance
bootstrap.servers=kafka-broker-1:9092,kafka-broker-2:9092,kafka-broker-3:9092
group.id=order-processor-v3
fetch.min.bytes=1048576
fetch.max.wait.ms=500
max.partition.fetch.bytes=10485760
max.poll.records=500
enable.auto.commit=false
isolation.level=read_committed
auto.offset.reset=earliest

isolation.level matters with transactions. read_committed makes consumers skip records from aborted transactions and hold back records from open ones until their commit/abort marker arrives, which is required for end-to-end exactly-once but adds latency. For non-transactional workloads, the default read_uncommitted is lower latency.

Topic Configuration Management at Scale

As clusters grow to hundreds or thousands of topics, configuration management becomes a first-order operational problem. A topic has dozens of tunable properties, and the right values depend on the workload: a high-throughput telemetry topic wants different retention and segment settings than a compacted changelog. Without systematic management you get configuration drift, where topics created at different times by different teams behave inconsistently under failure.

The foundation of scalable management is configuration tiers mapped to workload profiles. A typical scheme: high-throughput-transient for metrics and logs (short time-based retention, aggressive segment rolling, snappy compression); business-events for domain events (moderate retention, idempotent producers, Avro plus Schema Registry); stateful-changelog for processor state (compacted, short segment time, tuned dirty ratio); and audit-log for compliance (long retention, no compaction, encryption at rest). Each tier defines default configs applied to new topics and enforced on existing ones. This is covered in Managing Topic Configuration at Scale.

# Example configuration tier definition for infrastructure-as-code
topic_tiers:
  high-throughput-transient:
    defaults:
      retention.ms: 86400000        # 1 day
      segment.bytes: 536870912      # 512 MiB
      compression.type: snappy
      cleanup.policy: delete
      min.insync.replicas: 1
    overrides_allowed:
      - retention.ms
      - segment.bytes

  business-events:
    defaults:
      retention.ms: 604800000       # 7 days
      segment.bytes: 1073741824     # 1 GiB
      compression.type: zstd
      cleanup.policy: delete
      min.insync.replicas: 2
      max.message.bytes: 2097152
    overrides_allowed:
      - retention.ms
      - max.message.bytes

  stateful-changelog:
    defaults:
      cleanup.policy: compact
      min.cleanable.dirty.ratio: 0.5
      segment.ms: 1800000           # 30 minutes
      delete.retention.ms: 86400000 # 1 day
      min.insync.replicas: 2
    overrides_allowed:
      - min.cleanable.dirty.ratio
      - delete.retention.ms

Enforcing these standards needs both tooling and process. The Kafka Admin API allows programmatic inspection and modification of topic configs, so you can build auditors that run on a schedule and flag non-compliant topics. More mature setups validate configs at creation time in CI/CD and reject topics that do not match an approved tier. Apache Kafka also supports a server-side create.topic.policy.class.name (and alter.config.policy.class.name) plugin that intercepts topic creation and config changes at the broker, letting you enforce policy regardless of which client makes the request. The auditor below uses the confluent-kafka Python client, whose describe_configs return shape is stable and documented: it returns a dict of futures keyed by ConfigResource, and each future resolves to a {config_name: ConfigEntry} mapping.

# Configuration auditor that checks topics against tier standards (confluent-kafka)
from confluent_kafka.admin import AdminClient, ConfigResource

admin = AdminClient({"bootstrap.servers": "kafka-broker-1:9092"})

TIER_STANDARDS = {
    "business-events": {
        "min.insync.replicas": "2",
        "cleanup.policy": "delete",
    },
}

def audit_topic_configs(topic_name, expected_tier):
    resource = ConfigResource(ConfigResource.Type.TOPIC, topic_name)
    futures = admin.describe_configs([resource])
    entries = futures[resource].result()  # dict: {config_name: ConfigEntry}

    for key, expected_value in TIER_STANDARDS[expected_tier].items():
        actual = entries[key].value if key in entries else None
        if actual != expected_value:
            print(f"DRIFT: {topic_name} {key}={actual} expected={expected_value}")

audit_topic_configs("orders.v2.ecommerce", "business-events")

To change a topic config from the CLI, use kafka-configs.sh with --entity-type topics and --entity-name <topic> (the --topic shorthand is not valid here):

# Alter and then describe a single topic-level configuration
kafka-configs.sh --bootstrap-server kafka-broker-1:9092 \
  --alter \
  --entity-type topics \
  --entity-name orders.v2.ecommerce \
  --add-config retention.ms=259200000

kafka-configs.sh --bootstrap-server kafka-broker-1:9092 \
  --describe \
  --entity-type topics \
  --entity-name orders.v2.ecommerce

Operational Monitoring and Troubleshooting

Effective Kafka operations require visibility at several levels: broker, producer, consumer, and end-to-end latency. The broker exposes hundreds of JMX metrics; at minimum capture kafka.server:type=ReplicaManager,name=UnderReplicatedPartitions, kafka.controller:type=KafkaController,name=OfflinePartitionsCount, the per-topic kafka.server:type=BrokerTopicMetrics rates (BytesInPerSec, BytesOutPerSec, MessagesInPerSec, TotalProduceRequestsPerSec, TotalFetchRequestsPerSec), and request-latency percentiles from kafka.network:type=RequestMetrics. These reveal hot partitions, under-replicated data, and throughput anomalies in near real time.

Under-replicated partitions are a leading indicator of broker trouble. A partition is under-replicated when one or more followers fall out of the ISR. Common causes are inter-broker network saturation, disk I/O contention on a follower, or a broker still catching up after a restart. Persistent under-replication erodes durability and, with a strict min.insync.replicas, can block writes. The CLI gives a point-in-time view; production monitoring should alert when UnderReplicatedPartitions stays above zero for more than a few minutes.

# Check for under-replicated partitions across the cluster
kafka-topics.sh --bootstrap-server kafka-broker-1:9092 \
  --describe \
  --under-replicated-partitions

# Inspect consumer group membership, assignment, and lag
kafka-consumer-groups.sh --bootstrap-server kafka-broker-1:9092 \
  --describe \
  --group order-processor-v3

Consumer lag, the gap between a partition’s log-end offset and a group’s committed offset, is the most important consumer-side signal. High or growing lag means consumers cannot keep up, whether from too few instances, slow processing, or network limits. Monitor lag per partition, not just per group, because one slow partition can hide inside a healthy-looking aggregate. The kafka-consumer-groups.sh --describe output gives lag per partition on demand; for continuous monitoring, export it to a time-series database. The consumer client also exposes records-lag-max under the kafka.consumer:type=consumer-fetch-manager-metrics MBean, which the jmx_exporter typically surfaces as kafka_consumer_fetch_manager_records_lag_max. Burrow, the lag evaluator open-sourced by LinkedIn, classifies consumer health by lag trend rather than a single absolute threshold, which reduces false alarms on bursty workloads.

# Prometheus rules for consumer lag (recording + alerting)
groups:
  - name: kafka_consumer_lag
    rules:
      - record: job:kafka_consumer_records_lag_max:rate5m
        expr: rate(kafka_consumer_fetch_manager_records_lag_max[5m])
      - alert: ConsumerLagIncreasing
        expr: job:kafka_consumer_records_lag_max:rate5m > 10
        for: 15m
        labels:
          severity: warning
        annotations:
          summary: "Consumer lag is increasing for {{ $labels.consumer_group }}"
          description: "records-lag-max is rising at {{ $value }}/s over 5m"

Troubleshooting partition-level issues usually means correlating metrics across sources. A common scenario: producers report higher latency and the broker shows elevated produce-request time in kafka.network:type=RequestMetrics,name=TotalTimeMs,request=Produce. Drilling into per-partition rates may reveal one partition absorbing a disproportionate share of traffic, a hot partition caused by key skew. Remedies include reworking the key, increasing partition count (accepting the ordering implications), or shedding load at the application layer. The Confluent reference for topic configuration parameters documents every tunable that affects partition behavior.

Lifecycle Management and Governance

Topics have a lifecycle from creation through active use to deprecation and deletion. Without governance, topics accumulate indefinitely, consuming disk, file handles, and attention. A mature platform implements lifecycle management spanning naming conventions, ownership metadata, usage monitoring, and automated cleanup of abandoned topics.

Naming conventions encode ownership, environment, data classification, and structure in a parseable form. A common pattern is <domain>.<subdomain>.<event-type>.<environment>, for example ecommerce.orders.placed.prod. Structured names let tooling route topics to config tiers, apply ACLs, and generate documentation. Enforce conventions at creation time through automation, not documentation, ideally via a broker-side create.topic.policy.class.name plugin or a gateway service that rejects non-conforming names.

Ownership metadata is essential for long-term maintenance. Every topic should map to an owning team and a contact channel. You can stash a small amount of this in a topic config (topic configs accept arbitrary keys via --add-config, though brokers may log warnings for unknown keys), but the durable approach is an external registry keyed by topic name. Encoding ownership in the topic name (team-payments.orders.placed.prod) is tempting but brittle: it complicates ownership transfer and invites naming collisions.

# List non-internal topics for an ownership/audit sweep, cross-referencing an external registry
kafka-topics.sh --bootstrap-server kafka-broker-1:9092 \
  --list \
  --exclude-internal | while read -r topic; do
    echo "topic=$topic"
    # Look up owner in your governance registry, e.g.:
    #   curl -s "https://kafka-governance.internal/owners/$topic"
done

Deprecation is the most neglected phase. Topics nobody produces to or consumes from still consume resources and clutter the cluster. A practical “death certificate” process: identify topics with no producer or consumer activity over a window (say 30 days), notify registered owners, quarantine by revoking produce/consume ACLs, and finally delete after a grace period. Automate it; manual cleanup is rarely done consistently. A scheduled job that joins broker activity metrics with the ownership registry can produce a weekly candidate list.

# Identify topics with no committed consumer offsets as deprecation candidates (confluent-kafka)
from confluent_kafka.admin import AdminClient, ConsumerGroupTopicPartitions

admin = AdminClient({"bootstrap.servers": "kafka-broker-1:9092"})

def topics_with_active_consumers():
    groups = admin.list_consumer_groups().result()  # ListConsumerGroupsResult
    active_topics = set()
    for listing in groups.valid:                     # ConsumerGroupListing
        req = ConsumerGroupTopicPartitions(listing.group_id)
        offsets_future = admin.list_consumer_group_offsets([req])[listing.group_id]
        result = offsets_future.result()             # ConsumerGroupTopicPartitions
        for tp in result.topic_partitions or []:
            active_topics.add(tp.topic)
    return active_topics

def find_abandoned_topics():
    all_topics = set(admin.list_topics(timeout=10).topics)  # ClusterMetadata.topics
    all_topics = {t for t in all_topics if not t.startswith("__")}
    return all_topics - topics_with_active_consumers()

for topic in sorted(find_abandoned_topics()):
    print(f"Candidate for deprecation: {topic}")

This heuristic flags topics with no group offsets; before deletion, confirm there is also no recent produce traffic (via BytesInPerSec per topic or log-end-offset movement), since a topic can be actively produced to yet have no consumer groups.

The Apache Kafka documentation on log compaction is the definitive reference for the cleaner’s algorithm and the interaction between compaction and retention. For broader operational context, “Kafka: The Definitive Guide” by Gwen Shapira, Todd Palino, Rajini Sivaram, and Krit Petty remains a strong reference on topics, partitions, and the controller’s role in partition leadership.