Setting Up ACLs for Multi-Tenant Kafka Clusters

A single Apache Kafka cluster often serves many application teams, each requiring strict isolation of its topics, consumer groups, and transactional IDs. Access Control Lists (ACLs) are the mechanism that turns a shared cluster from a security liability into a governed, multi-tenant data backbone. This guide gives Platform, SRE, and backend engineers a concrete, verifiable methodology for designing, applying, and validating ACLs so that tenants can only touch their own resources.

Authorization is one pillar of a secure deployment, sitting alongside transport encryption and authentication. For the broader picture of Kafka’s authorization mechanisms, see the dedicated guide on Kafka ACLs and Authorization Best Practices. This article focuses on the multi-tenancy implementation and assumes a working understanding of Kafka’s pluggable authorizer and its role in the wider Operations, Security & Observability domain.

Prerequisites and Cluster Configuration

ACLs are identity-based: without a reliable way to identify the principal behind a request, authorization is meaningless. In a production multi-tenant environment that means moving beyond PLAINTEXT to SASL- or mTLS-based authentication, and enabling an authorizer on every node.

Set authorizer.class.name in server.properties on all nodes. The choice depends on your deployment mode:

  • KRaft clusters (the only mode supported in Kafka 4.0, and the recommended mode since 3.3) use the built-in StandardAuthorizer, which stores ACLs in the __cluster_metadata log. It is the default authorizer in KRaft and is a drop-in replacement for the legacy one.
  • Legacy ZooKeeper clusters (Kafka 3.x and earlier) use kafka.security.authorizer.AclAuthorizer, which stores ACLs in ZooKeeper. ZooKeeper mode was removed in Kafka 4.0.
# server.properties (KRaft)
authorizer.class.name=org.apache.kafka.metadata.authorizer.StandardAuthorizer
super.users=User:admin;User:kafka-broker

super.users is a semicolon-separated list of principals that bypass all ACL checks. The principal type (User) is case-sensitive. Keep this list as small as possible — typically just the cluster’s own operational service accounts.

Next, configure listeners to enforce authentication. A common pattern is SASL/SCRAM for application clients and mTLS for inter-broker traffic. The snippet below defines a SASL_SSL client listener on 9093 and an SSL internal listener on 9092; SASL_SSL encrypts both credentials and data in transit.

# server.properties
listeners=INTERNAL://:9092,CLIENTS://:9093
advertised.listeners=INTERNAL://kafka-broker-0.internal:9092,CLIENTS://kafka-broker-0.example.com:9093
listener.security.protocol.map=INTERNAL:SSL,CLIENTS:SASL_SSL
inter.broker.listener.name=INTERNAL
ssl.truststore.location=/etc/kafka/secrets/kafka.truststore.jks
ssl.truststore.password=truststore-password
ssl.keystore.location=/etc/kafka/secrets/kafka.keystore.jks
ssl.keystore.password=keystore-password
sasl.enabled.mechanisms=SCRAM-SHA-512

After applying these changes, perform a rolling restart and confirm each node comes back up cleanly. The official Apache Kafka Authorization and ACLs documentation details every supported authentication mechanism and authorizer configuration parameter.

Designing a Tenant Namespace Convention

A workable multi-tenant ACL strategy depends on a predictable, enforceable naming convention. Without one, ACL rules degrade into an unmanageable sprawl of per-resource exceptions. Least privilege demands that each tenant be scoped to a specific resource prefix.

Mandate that every tenant operates within a dedicated prefix for topics, consumer groups, and transactional IDs. A tenant named orders owns all resources beginning with orders.. Enforce this at the organizational level — ideally through a self-service provisioning tool that validates resource names against the authenticated principal.

Tenant: orders
  Topics:            orders.checkout, orders.payment, orders.fulfillment.internal
  Consumer Groups:   orders.checkout-processor, orders.payment-auditor
  Transactional IDs: orders.txn-checkout-0

Prefix-based isolation is the cornerstone of the model. A single compact ACL with --resource-pattern-type prefixed grants a principal access to all of its current and future resources without ever touching another tenant’s data, and it scales to hundreds of tenants with negligible broker overhead.

Creating and Applying Tenant ACLs

With the convention in place, create the ACLs with kafka-acls.sh. A typical tenant principal needs:

  • Read and Write on its topics (prefixed). Write on a topic is also all that an idempotent producer needs since Kafka 2.8 — the old separate IdempotentWrite cluster permission is no longer required when producing to 2.8+ brokers (KIP-679 made idempotence the producer default in 3.0).
  • Read on its consumer groups (prefixed). Read on a group covers the join/heartbeat/sync flow; offset commits are authorized by Read on the topic.
  • Write and Describe on its transactional IDs (prefixed), if it uses transactions. Write authorizes the transaction itself; Describe is checked when the producer locates the transaction coordinator. Omitting Describe breaks transactional producers.

Note what is not needed: a tenant does not need any Cluster-level permission for normal produce/consume. Topic and group metadata is authorized by Describe (implied by Read/Write) on the topic and group resources, not on the cluster.

The commands below build the access profile for the orders tenant, authenticating as User:orders-app:

# Read + Write on all topics under the orders. prefix
kafka-acls.sh --bootstrap-server kafka-broker-0.example.com:9093 \
  --command-config admin-client.conf \
  --add \
  --allow-principal User:orders-app \
  --operation Read --operation Write \
  --topic orders. \
  --resource-pattern-type prefixed

# Read on all consumer groups under the orders. prefix
kafka-acls.sh --bootstrap-server kafka-broker-0.example.com:9093 \
  --command-config admin-client.conf \
  --add \
  --allow-principal User:orders-app \
  --operation Read \
  --group orders. \
  --resource-pattern-type prefixed

# Write + Describe on all transactional IDs under the orders. prefix
kafka-acls.sh --bootstrap-server kafka-broker-0.example.com:9093 \
  --command-config admin-client.conf \
  --add \
  --allow-principal User:orders-app \
  --operation Write --operation Describe \
  --transactional-id orders. \
  --resource-pattern-type prefixed

If a tenant does not use transactions, skip the third command entirely.

The admin-client.conf file holds the credentials for an administrative principal authorized to manage ACLs (Alter on the cluster). Never distribute it to tenants. Note that --operation All is a wildcard that matches every operation, including Delete and Alter on the topic — granting topic management rights you usually want to withhold. Prefer enumerating Read and Write explicitly and reserving Delete/Alter for a separate tenant-admin principal.

Validating and Auditing ACL Enforcement

After applying ACLs, verify they behave as intended. kafka-acls.sh --list shows the currently applied ACLs and can be filtered by principal or resource.

# List all ACLs for the orders-app principal
kafka-acls.sh --bootstrap-server kafka-broker-0.example.com:9093 \
  --command-config admin-client.conf \
  --list \
  --principal User:orders-app

Beyond listing, run positive and negative integration tests with a client using the tenant’s own credentials. A successful read from an authorized topic, plus a TopicAuthorizationException when consuming from another tenant’s topic, confirms enforcement is working in both directions. The Confluent reference on Kafka ACLs lists the error codes you should expect.

For ongoing auditability, configure the kafka.authorizer.logger. By default this logger emits denied decisions at INFO and allowed decisions at DEBUG; each line includes the principal, client host, operation, and resource. Leaving it at INFO gives you a low-volume audit trail of every rejected request without the firehose of allowed traffic. Only drop to DEBUG for short, targeted debugging sessions — on a busy cluster, logging every allowed request can produce gigabytes per hour.

Kafka 3.x ships log4j 1.x (reload4j); the appender configuration below uses log4j.properties:

# log4j.properties (Kafka 3.x / reload4j)
log4j.logger.kafka.authorizer.logger=INFO, authorizerAppender
log4j.additivity.kafka.authorizer.logger=false
log4j.appender.authorizerAppender=org.apache.log4j.DailyRollingFileAppender
log4j.appender.authorizerAppender.File=/var/log/kafka/kafka-authorizer.log
log4j.appender.authorizerAppender.layout=org.apache.log4j.PatternLayout
log4j.appender.authorizerAppender.layout.ConversionPattern=[%d] %p %m (%c)%n

Kafka 4.0 migrated logging to Log4j 2 (KIP-653); there, configure the same kafka.authorizer.logger through log4j2.yaml/log4j2.properties rather than the log4j 1.x appender classes above.

Managing Cross-Tenant Access and Shared Topics

Strict prefix isolation covers most cases, but platforms frequently need controlled data sharing — for example a shared platform.events topic that several tenants produce to or consume from. Model this with explicit, literal ACLs rather than broadening any prefix.

Grant Write or Read on the specific shared topic to exactly the principals that need it, while each tenant’s prefix-based rules continue to protect its private resources.

# orders-app can write to the shared platform.events topic
kafka-acls.sh --bootstrap-server kafka-broker-0.example.com:9093 \
  --command-config admin-client.conf \
  --add \
  --allow-principal User:orders-app \
  --operation Write \
  --topic platform.events \
  --resource-pattern-type literal

# fulfillment-app can read from the shared platform.events topic
kafka-acls.sh --bootstrap-server kafka-broker-0.example.com:9093 \
  --command-config admin-client.conf \
  --add \
  --allow-principal User:fulfillment-app \
  --operation Read \
  --group fulfillment.events-consumer \
  --resource-pattern-type prefixed

A consumer of the shared topic still needs Read on its own (prefixed) consumer group, shown in the second command — granting topic Read alone is not enough to join a group. Keeping each shared topic as an explicit, literal exception makes it visible in ACL listings and audit logs, and avoids the accidental leakage a broad * prefix would cause. For more complex flows, consider a dedicated bridge principal that consumes from one tenant’s topic and produces to another’s, with ACLs scoped to that bridge alone.

Operationalizing ACLs with Infrastructure as Code

Running kafka-acls.sh by hand is error-prone and unsustainable across environments. Treat ACL definitions as code: version them, review them, and apply them through CI/CD. Define the desired state declaratively and reconcile it on each deploy so that diffs, rollbacks, and audit trails live in Git alongside application code.

For teams on Kubernetes, the Strimzi operator exposes a KafkaUser custom resource with a simple authorization type that maps directly onto Kafka ACLs:

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaUser
metadata:
  name: orders-app
  labels:
    strimzi.io/cluster: my-cluster
spec:
  authentication:
    type: scram-sha-512
  authorization:
    type: simple
    acls:
      - resource:
          type: topic
          name: orders.
          patternType: prefix
        operations: [Read, Write]
      - resource:
          type: group
          name: orders.
          patternType: prefix
        operations: [Read]
      - resource:
          type: transactionalId
          name: orders.
          patternType: prefix
        operations: [Write, Describe]

Strimzi provisions both the SCRAM credentials and the matching ACLs from this single resource. The Strimzi KafkaUserAuthorizationSimple reference documents the exact field names and supported operations. Whatever tool you choose, the principle holds: ACLs are infrastructure, not ad-hoc administrative tweaks.

Troubleshooting Common ACL Pitfalls

Principal mismatch. The ACL principal must exactly match what the authentication layer extracts. For SASL/SCRAM it is User:<username>; for mTLS it is User:<full DN>, e.g. User:CN=orders-app,OU=apps,O=example,C=US. A frequent mistake is using the CN alone when the full Distinguished Name is what the broker derives (the mapping is governed by ssl.principal.mapping.rules). Confirm the exact principal from the authorizer log after a failed request.

Pattern-type confusion. --resource-pattern-type literal on orders. matches only a topic literally named orders., not orders.checkout. Use prefixed for prefix matching, and verify the pattern type in your ACL listings.

Missing group permission for consumers. A consumer needs Read on the consumer group and Read (which implies Describe) on the topic. Missing the group ACL surfaces as GroupAuthorizationException; missing the topic ACL surfaces as TopicAuthorizationException. Neither requires any cluster-level permission.

Transactional producers missing Describe. Granting only Write on the transactional ID lets the transaction start but fails coordinator discovery. Grant Write and Describe on the transactional ID.

Deny rules win. The authorizer evaluates deny before allow: a single matching deny ACL overrides every allow. A broad --deny-principal User:* --operation All --topic * blocks all of your carefully crafted allow rules. Use deny ACLs sparingly and only for specific, high-risk resources.

When debugging, temporarily set kafka.authorizer.logger to DEBUG, reproduce the failing operation, and read the resulting line — it states the exact principal, host, operation, resource, and whether it was Allowed or Denied. As the broader Operations, Security & Observability discipline stresses, observability into authorization decisions is as important as monitoring broker health.

Back to Kafka ACLs And Authorization Best Practices