Enforcing Topic Naming Standards in Kafka and Confluent Cloud

In a self-service Kafka platform, topics are the shared vocabulary, and an unenforced naming convention degrades fast: cryptic names, accidental collisions between teams, and a data-discovery problem no wiki page can fix. The durable solution is to make the convention a gate — reject non-compliant names at the point of creation instead of policing them after the fact.

There are two honest ways to do this, and which one applies depends on where your brokers run:

This guide covers both, with the exact interfaces, configs, and CLI commands that actually exist. A naming convention is only as good as its design, which we cover in Topic Naming Conventions and Governance Best Practices; the broader architectural context lives in Topics, Partitions & Data Modeling. What follows is the operational realization of those design principles.

Note on a common confusion: Confluent Cloud does not expose a managed “topic policy” object with a confluent kafka topic-policy CLI command, a TopicPolicy YAML descriptor, or a confluent_kafka_topic_policy Terraform resource. None of those exist. The real, supported mechanisms are the two below.

Designing a Convention That a Machine Can Enforce

Before any enforcement, codify the convention as a single regular expression. A well-designed convention is hierarchical and delimiter-separated so each segment carries meaning:

<domain>.<subdomain>.<entity>.<event-type>.<data-format>.<environment>

Concrete examples:

• inventory.warehouse.stock.level-changed.avro.prod
• payments.billing.invoice.generated.protobuf.staging
• iot.fleet.telemetry.raw.json.dev

The domain and subdomain drive ownership and discovery; entity and event-type describe the business event; data-format signals the serialization strategy; and the environment suffix prevents staging data from leaking into production consumers.

A regex that enforces an approved enumeration on the bounded segments:

^(inventory|payments|iot|users)\.[a-z0-9-]+\.[a-z0-9-]+\.[a-z-]+\.(avro|protobuf|json)\.(dev|staging|prod)$

This requires an approved domain, lowercase alphanumeric-plus-hyphen for the free segments, an approved serialization format, and an explicit environment. Two constraints to bake in deliberately:

• Stay under the length limit. Kafka caps topic names at 249 characters (a filesystem constraint, since each partition becomes a directory named <topic>-<partition>). A six-segment hierarchical name is well within budget, but generated suffixes (timestamps, UUIDs) can push past it. Note that . and _ are both legal characters yet collide in metric names, which is the reason this convention uses . consistently and never mixes the two as delimiters.
• Anchor both ends. The ^…$ anchors are load-bearing. Without them, Bad-Name.payments.billing.x.avro.prod would partially match and slip through.

Validate the pattern in a real engine (grep -P, or a Java Pattern test if you are writing a broker plugin) before wiring it into enforcement, because a regex bug here either blocks every topic or silently lets everything through.

Self-Managed: The CreateTopicPolicy Plugin

On clusters where you control the brokers, Kafka gives you a first-class hook for this. KIP-108 introduced the org.apache.kafka.server.policy.CreateTopicPolicy interface. When you set the broker config

create.topic.policy.class.name=com.example.kafka.NamingConventionPolicy

the controller instantiates your class, calls configure() with the broker configs, and invokes validate() for every topic in every CreateTopics request before any metadata is written. This is inline, synchronous enforcement that an AdminClient call cannot bypass, because it runs inside the broker, not in a CI step.

Implementing the interface

The interface is small. validate() receives a RequestMetadata describing the requested topic and throws PolicyViolationException to reject it:

package com.example.kafka;

import org.apache.kafka.common.errors.PolicyViolationException;
import org.apache.kafka.server.policy.CreateTopicPolicy;
import java.util.Map;
import java.util.regex.Pattern;

public class NamingConventionPolicy implements CreateTopicPolicy {

    private static final Pattern NAME =
        Pattern.compile("^(inventory|payments|iot|users)\\."
            + "[a-z0-9-]+\\.[a-z0-9-]+\\.[a-z-]+\\."
            + "(avro|protobuf|json)\\.(dev|staging|prod)$");

    @Override
    public void configure(Map<String, ?> configs) { /* read tunables if needed */ }

    @Override
    public void validate(RequestMetadata metadata) throws PolicyViolationException {
        String topic = metadata.topic();
        // Allow Kafka-internal topics (e.g. __consumer_offsets) and other reserved names.
        if (topic.startsWith("_")) {
            return;
        }
        if (!NAME.matcher(topic).matches()) {
            throw new PolicyViolationException(
                "Topic '" + topic + "' must match "
                + "<domain>.<subdomain>.<entity>.<event-type>.<format>.<env> "
                + "with approved domains, formats, and environments.");
        }
    }

    @Override
    public void close() { /* release resources if any */ }
}

RequestMetadata exposes exactly five accessors you can validate against: topic() (String), numPartitions() (Integer), replicationFactor() (Short), replicasAssignments() (Map<Integer, List<Integer>>), and configs() (Map<String, String>). That means the same hook can also enforce a minimum replication factor or a sane partition count, not just the name. One sharp edge: when a request uses explicit replicasAssignments, numPartitions() and replicationFactor() return null — so derive the replication factor from the assignment map in that case rather than dereferencing a null Short.

Package the class on the broker’s classpath, set create.topic.policy.class.name, and do a rolling restart for the config to take effect. On a KRaft cluster the policy runs on the active controller; in older ZooKeeper-based deployments it runs on the broker handling the request — in both cases it is broker-side, not client-side, which is the whole point.

What the client sees

When validate() throws, the broker returns Kafka error code 44, POLICY_VIOLATION, carrying your exception message. The failure is per-topic: other topics in the same batch request are still processed independently, so one bad name in a batch does not roll back the compliant ones. A client attempting a bad name gets:

org.apache.kafka.common.errors.PolicyViolationException:
Topic 'TestTopic' must match <domain>.<subdomain>.<entity>.<event-type>.<format>.<env>
with approved domains, formats, and environments.

Two important scope limits from KIP-108: the policy only runs on CreateTopics, not on AlterConfigs, CreatePartitions, or topic deletion, and it is a creation-time gate, so it does not rename or retroactively reject topics that already exist. If you also need to gate config changes (for example, blocking a later retention.ms override), pair this with the separate alter.config.policy.class.name hook; the two policies are independent and neither covers the other’s surface.

Confluent Cloud: Prefixed ACLs and RBAC

On Confluent Cloud you have no broker access, so a CreateTopicPolicy plugin is not an option. The supported, Cloud-native way to constrain what a principal can name a topic is the authorization layer itself. A key Kafka rule makes this work: a resource cannot be created unless a principal has CREATE granted on a name or a prefix of that name. By granting CREATE only on the prefixes you sanction, you make non-compliant names un-creatable.

Grant the payments service account the right to create only payments.-prefixed topics:

confluent kafka acl create \
  --allow \
  --service-account sa-12345 \
  --operations create \
  --topic "payments." \
  --prefix \
  --cluster lkc-def456

The --prefix flag interprets --topic as a prefix rather than a literal name. (On self-managed Confluent Platform the equivalent uses --operation singular and --principal instead of --service-account.) Repeat per team to build a per-namespace authorization model; a principal with no matching prefixed CREATE grant simply cannot create a topic at all.

What ACLs can and cannot do here, stated plainly:

• They can confine each principal to a domain prefix (payments., inventory.), which is the single most effective guardrail in a multi-team Cloud cluster.
• They cannot evaluate a full regex over the entire name. Prefix matching does not enforce that the <format> segment is one of avro|protobuf|json, nor that the suffix is a valid environment. payments.GARBAGE.x.txt.qa matches the payments. prefix and would be allowed.

Confluent Cloud RBAC layers cleanly on top: assign the DeveloperManage role scoped to a prefixed topic resource for the same effect with role-based ergonomics, and reserve CloudClusterAdmin for the platform team. Prefer RBAC roles for everyday access and use ACLs for the fine-grained prefix-creation rules. Mixing both works because Confluent’s authorizer grants access if either an RBAC role binding or an ACL allows it — useful, but it also means a stray broad ACL can silently widen what an RBAC scope was meant to restrict, so audit both layers together.

Enforcing the full pattern: pre-creation validation

Because prefix ACLs only constrain the leading segment, enforce the complete regex one step earlier, in the pipeline that provisions topics. Two practical options:

1. Terraform. Provision topics with the real confluent_kafka_topic resource and validate the name with a validation block on the input variable, so a non-compliant name fails terraform plan — before any API call reaches Confluent:

  variable "topic_name" {
    type = string
    validation {
      condition = can(regex(
        "^(inventory|payments|iot|users)\\.[a-z0-9-]+\\.[a-z0-9-]+\\.[a-z-]+\\.(avro|protobuf|json)\\.(dev|staging|prod)$",
        var.topic_name))
      error_message = "Topic name must follow <domain>.<subdomain>.<entity>.<event-type>.<format>.<env>."
    }
  }

  resource "confluent_kafka_topic" "this" {
    topic_name    = var.topic_name
    kafka_cluster { id = confluent_kafka_cluster.primary.id }
    rest_endpoint = confluent_kafka_cluster.primary.rest_endpoint
    credentials {
      key    = confluent_api_key.app.id
      secret = confluent_api_key.app.secret
    }
  }

Confirm the resource and argument names against the Confluent Terraform provider docs for your provider version. There is no policy resource to reach for here; the gate is the variable validation plus the prefix ACLs above.

2. CI/CD check. If topics are created from a script or GitOps repo, run the regex in CI before the confluent kafka topic create call. Pair it with the prefix ACLs so the authorization layer remains the backstop even if someone creates a topic outside the pipeline.

This is a defense-in-depth posture: ACLs/RBAC make wrong prefixes impossible at the broker, and pipeline validation enforces the rest of the structure with a fast, human-readable error. The two layers fail in different places — that redundancy is the point. Treat the pipeline check as the convenience gate (good error messages) and the ACL as the authority gate (cannot be bypassed).

Observability and Audit

Topic creations and authorization decisions are recorded in the Confluent Cloud audit log. Stream it to your observability platform and alert on kafka.CreateTopics events that are denied (authorizationResult: DENY), which indicate either a misconfigured pipeline or someone trying to bypass the convention. Audit records include the principal, the requested resource, and a timestamp — enough for rapid diagnosis of who tried to create what.

To inspect the audit-log configuration from the CLI on Confluent Cloud, use confluent audit-log describe, which describes the audit-log configuration for your organization. (On self-managed Confluent Platform the equivalent commands live under confluent audit-log config describe and confluent audit-log route list.) For long-term analysis, route the log to a dedicated cluster and sink it to Elasticsearch, Splunk, or your SIEM.

For self-managed clusters, surface policy rejections from broker logs and track the POLICY_VIOLATION (error code 44) responses your clients receive. A spike usually means a deploy shipped a non-compliant topic definition; a slow trickle usually means one team never adopted the convention. Watching the rate distinguishes the two.

Troubleshooting

Regex behaves differently than expected. Test the pattern in the same engine that will run it. A Java CreateTopicPolicy uses java.util.regex, while a grep -P smoke test uses PCRE — they agree on this pattern, but lookarounds and character-class edge cases can diverge. In a Terraform validation block, remember that HCL strings require \\. for a literal dot. If a name you expect to pass is rejected, check the anchors first: a missing $ lets a trailing segment through, a missing ^ lets a bad prefix through.

Internal topics get blocked. Kafka-internal and many framework topics start with _ (__consumer_offsets, __transaction_state) or are created by Kafka Streams and Connect with their own naming (changelog and repartition topics, connect-configs, connect-offsets, connect-status). Exempt them explicitly — the topic.startsWith("_") guard in the policy above, or a dedicated prefix ACL for the framework’s service account on Confluent Cloud — rather than trying to encode every exception into one regex.

ACL grant isn’t taking effect. Confirm the --prefix flag was actually set (without it, --topic is a literal exact-match name), that the principal is the service account that issues the create request (not the human running the CLI), and that you targeted the right cluster. Authorization is deny-by-default: if no prefixed CREATE ACL matches, creation fails with TOPIC_AUTHORIZATION_FAILED, not POLICY_VIOLATION — the error code tells you which layer rejected it.

Policy plugin not loaded (self-managed). create.topic.policy.class.name is read at broker startup, so a rolling restart is required after changing it. A ClassNotFoundException in the broker log means the JAR is not on the classpath; a topic that should have been rejected but went through usually means the restart didn’t actually reload the config on every node — check each broker, not just one.

Conclusion

Enforcing topic naming turns a convention document into a control the platform actually applies. On self-managed Kafka, the CreateTopicPolicy interface (KIP-108) gives you a true broker-side gate that returns a POLICY_VIOLATION for any non-compliant name. On Confluent Cloud, where broker plugins are off the table, prefixed ACLs and RBAC confine each team to its namespace, and a regex check in Terraform or CI enforces the rest of the structure before the topic is ever requested.

Treat the convention as a living standard: when domains or formats change, update the regex and the ACLs through the same version-controlled, peer-reviewed process you use for any infrastructure change. Pair this enforcement with the design principles in Topic Naming Conventions and Governance Best Practices, and revisit the modeling context in Topics, Partitions & Data Modeling.

References: KIP-108: Create Topic Policy, the CreateTopicPolicy Javadoc, Confluent Cloud ACLs, and the Confluent Terraform provider.