Kafka ACLs and Authorization Best Practices

Authentication verifies who is connecting; authorization decides what they can do once inside. In a cluster shared across teams, applications, or business units, the Access Control List (ACL) layer is what enforces data isolation, prevents one tenant from reading or clobbering another’s topics, and satisfies compliance audits. This guide covers the authorization model, the multi-tenant patterns that keep ACL management tractable, the automation that makes it auditable, and the metrics that tell you when a policy is silently breaking a pipeline.

Authorization is one pillar of a broader Operations, Security & Observability strategy, and it is useless on its own. ACLs are evaluated against the principal established during the authentication handshake, so a working authentication layer—mutual TLS or SASL/SCRAM, as covered in Securing Kafka with mTLS and SASL/SCRAM—is a hard prerequisite. Without it, every connection authenticates as User:ANONYMOUS and your carefully written ACLs have nothing meaningful to match against.

The Kafka Authorization Model: Principals, Resources, and Operations

Kafka authorization is attribute-based. Every ACL binding ties a principal to a resource with a specific operation, a permission type (Allow or Deny), and a host. The granularity of each component is what makes—or breaks—a policy.

A principal is an identity in the format User:<name>. The name comes from the authentication mechanism: with SASL/SCRAM it is the username from the handshake; with mTLS it is the certificate’s Distinguished Name (DN), usually mapped to a shorter name via ssl.principal.mapping.rules or a custom KafkaPrincipalBuilder. The wildcard User:* matches all authenticated principals—powerful, and dangerous.

A resource is a typed object. Kafka defines these resource types, each addressed by a name:

• Topic — a topic name, or the wildcard *.
• Group — a consumer group ID, or *.
• Cluster — the cluster itself, addressed by the fixed resource name kafka-cluster.
• TransactionalId — the transactional.id used for exactly-once semantics (EOS).
• DelegationToken — the token ID, for delegation-token operations.
• User — the target principal, used to authorize creating and describing delegation tokens on behalf of another user.

Operations are not interchangeable across resource types—a request fails authorization if the operation it needs is granted on the wrong resource. The combinations that matter most in practice:

The ALL pseudo-operation grants every operation on a resource. Be deliberate with it—ALL on a topic also confers DELETE and ALTER_CONFIGS, which most application principals should never hold.

The permission type is Allow or Deny. Deny takes precedence over Allow, giving you an explicit exclusion mechanism. Overusing Deny produces policies that are hard to reason about and audit; prefer a default-deny posture built from explicit Allow rules, and reserve Deny for narrow carve-outs (for example, blocking one principal from a prefix that an Allow * otherwise opens).

The host field restricts an ACL to connections from a specific IP. In dynamic environments—Kubernetes especially—pod IPs churn, so host-based ACLs are fragile and a frequent cause of “it worked yesterday” denials after a reschedule. Most production deployments set the host to * and enforce network segmentation through firewalls, security groups, or service-mesh policy instead.

Enabling and Configuring the Authorizer

Kafka’s authorization is pluggable via authorizer.class.name. On modern KRaft-based clusters the built-in authorizer is StandardAuthorizer (introduced in KIP-801), which stores ACLs in the __cluster_metadata log rather than ZooKeeper. It is the default authorizer implementation in KRaft mode but is not active unless you set the class explicitly:

# Enable the KRaft ACL authorizer (Kafka 3.3+ / required path in 4.0)
authorizer.class.name=org.apache.kafka.metadata.authorizer.StandardAuthorizer

# Super users bypass all ACL checks (semicolon-separated)
super.users=User:admin;User:kafka-broker

Note on versions: the older ZooKeeper-backed kafka.security.authorizer.AclAuthorizer was the authorizer for ZooKeeper clusters. ZooKeeper mode and AclAuthorizer were removed in Kafka 4.0, which is KRaft-only. If you are migrating from a ZooKeeper cluster, switch authorizer.class.name to StandardAuthorizer as part of the migration; it honors the same super.users and allow.everyone.if.no.acl.found settings.

Super users bypass authorization entirely. At minimum, the broker’s own inter-broker principal and a designated admin principal should be super users—otherwise an over-tightened ACL set can lock you out of your own cluster, and recovering means editing broker config and restarting. The list is semicolon-separated (not comma-separated), because principal names derived from mTLS DNs routinely contain commas; using a comma here silently splits one DN into several bogus principals.

The single most consequential setting for a multi-tenant cluster is allow.everyone.if.no.acl.found. When it is true, a resource that has no matching ACL is open to every authenticated principal; resources that have at least one ACL still operate default-deny. When it is false, a resource with no matching Allow is denied. For a strict, predictable posture, set it explicitly to false rather than relying on the default—make the cluster’s stance a declared part of its config:

# Deny any operation that lacks an explicit Allow ACL
allow.everyone.if.no.acl.found=false

With this set, every principal—including internal system actors and the consumers of internal topics like __consumer_offsets—needs an explicit grant. That is more work up front, but it eliminates the most common authorization gap: a newly created resource that nobody remembered to lock down is silently world-accessible.

Designing a Multi-Tenant ACL Strategy

In a multi-tenant cluster the goal is strong isolation between tenants with minimal operational toil. The most scalable pattern is prefix-based naming with prefixed-pattern ACLs: each tenant owns a name prefix, and ACLs are written against the prefix rather than against individual topics. The decision rule is simple—use prefixed for a namespace a team will keep adding to (their topics, their groups, their transactional IDs); use literal only for a specific shared resource that should never expand by name.

Consider a cluster shared by orders, payments, and analytics. Each team owns topics under its prefix. ACLs for the orders team:

# Full access to the orders team's own topic namespace
kafka-acls --bootstrap-server localhost:9092 --command-config admin.conf \
  --add --allow-principal User:orders-service \
  --operation ALL --topic orders. --resource-pattern-type prefixed

# Read-only access for analytics to the orders namespace
kafka-acls --bootstrap-server localhost:9092 --command-config admin.conf \
  --add --allow-principal User:analytics-service \
  --operation READ --operation DESCRIBE --topic orders. --resource-pattern-type prefixed

--resource-pattern-type prefixed matches any topic whose name starts with orders., so orders-service automatically covers new topics like orders.checkout or orders.fulfillment with no further ACL changes. (The default pattern type is literal; match is a list/remove-time mode that returns the literal, wildcard, and prefixed rules affecting a given name.) Note that a consumer also needs DESCRIBE to resolve topic metadata—granting READ alone is a frequent cause of confusing failures, which is why the analytics grant above includes both.

Consumer groups follow the same convention. orders-service needs group access under its prefix:

kafka-acls --bootstrap-server localhost:9092 --command-config admin.conf \
  --add --allow-principal User:orders-service \
  --operation READ --group orders. --resource-pattern-type prefixed

It helps to keep a per-role checklist of the minimum ACLs each client type needs, so reviewers can spot an under- or over-grant at a glance:

For implementing this at scale—automation, shared topics, and naming governance—see Setting Up ACLs for Multi-Tenant Kafka Clusters.

Transactional producers (exactly-once semantics) need TransactionalId ACLs in addition to topic ACLs. Grant WRITE and DESCRIBE on the transactional-id prefix:

kafka-acls --bootstrap-server localhost:9092 --command-config admin.conf \
  --add --allow-principal User:orders-service \
  --operation WRITE --operation DESCRIBE \
  --transactional-id orders. --resource-pattern-type prefixed

On modern brokers a transactional producer is granted idempotent write implicitly once it is authorized for its transactional ID, so a separate IDEMPOTENT_WRITE grant on the cluster is generally unnecessary. A non-transactional idempotent producer relies on the default cluster-level idempotent-write behavior; only grant IDEMPOTENT_WRITE on Cluster explicitly if you have an unusual setup that needs it.

Automating ACL Management

Running kafka-acls by hand is error-prone and does not scale. In production, ACLs should be managed as code.

The kafka-acls tool has no built-in idempotency or reconciliation: there is no --force or upsert flag, and re-running the same --add is a no-op only because the authorizer deduplicates identical bindings. Crucially, drift in the other direction is invisible to --add—a binding that exists on the cluster but was deleted from your file will not be removed by re-applying the file. A practical sync script therefore queries existing ACLs and applies the diff, adding what is missing:

#!/bin/bash
# acl-sync.sh: add ACLs from a JSON file if they are not already present
set -euo pipefail

BOOTSTRAP_SERVER="localhost:9092"
ADMIN_CONF="admin.conf"
ACL_FILE="acls.json"

jq -c '.[]' "$ACL_FILE" | while read -r entry; do
  principal=$(jq -r '.principal'     <<<"$entry")
  operation=$(jq -r '.operation'     <<<"$entry")
  resource_type=$(jq -r '.resource_type' <<<"$entry")   # topic | group | transactional-id | cluster
  resource_name=$(jq -r '.resource_name' <<<"$entry")
  pattern_type=$(jq -r '.pattern_type'   <<<"$entry")    # literal | prefixed

  existing=$(kafka-acls --bootstrap-server "$BOOTSTRAP_SERVER" \
    --command-config "$ADMIN_CONF" --list \
    --principal "$principal" \
    --"$resource_type" "$resource_name" \
    --resource-pattern-type "$pattern_type" 2>/dev/null \
    | grep -c "operation=${operation}" || true)

  if [ "$existing" -eq 0 ]; then
    echo "Adding: $principal -> $operation on $resource_type:$resource_name ($pattern_type)"
    kafka-acls --bootstrap-server "$BOOTSTRAP_SERVER" \
      --command-config "$ADMIN_CONF" --add \
      --allow-principal "$principal" --operation "$operation" \
      --"$resource_type" "$resource_name" \
      --resource-pattern-type "$pattern_type"
  fi
done

Two correctness notes on the script: --list accepts --principal (the filter form), whereas --add requires --allow-principal or --deny-principal—they are not interchangeable, and mixing them up is a common first-run error. And the snippet only adds; full reconciliation that also removes stale ACLs requires a second pass that diffs the cluster’s current ACLs against the desired set and issues --remove for the extras.

For larger estates, a dedicated tool gives you a declarative source of truth, a reconcile loop, and audit logging—for example the Conduktor Kafka Security Manager, which reconciles a CSV/YAML of ACLs against the cluster. The mature setup keeps that desired-state file in Git, gates every change behind code review, validates it in CI, and applies it from the pipeline—giving you an audit trail and shutting the door on out-of-band changes that drift away from the file.

Monitoring and Auditing ACLs

A misconfigured ACL fails closed and silently: a producer or consumer starts getting denied, and unless you are watching for it, you find out when a downstream business metric degrades. Watch the authorization-failure signal directly.

Brokers expose per-request-type, per-error-code counts through the request-metrics MBean. The object name is kafka.network (not kafka.server), the metric is ErrorsPerSec, and the error key is set to a Kafka protocol error name:

• kafka.network:type=RequestMetrics,name=ErrorsPerSec,request=Produce,error=TOPIC_AUTHORIZATION_FAILED
• kafka.network:type=RequestMetrics,name=ErrorsPerSec,request=Fetch,error=TOPIC_AUTHORIZATION_FAILED
• kafka.network:type=RequestMetrics,name=ErrorsPerSec,request=JoinGroup,error=GROUP_AUTHORIZATION_FAILED
• kafka.network:type=RequestMetrics,name=ErrorsPerSec,request=InitProducerId,error=TRANSACTIONAL_ID_AUTHORIZATION_FAILED

The error label uses the protocol error name (TOPIC_AUTHORIZATION_FAILED = code 29, GROUP_AUTHORIZATION_FAILED = 30, CLUSTER_AUTHORIZATION_FAILED = 31, TRANSACTIONAL_ID_AUTHORIZATION_FAILED = 53). Reading the request label tells you which grant is missing: a spike on Produce/Fetch with TOPIC_AUTHORIZATION_FAILED points at a topic ACL, while JoinGroup with GROUP_AUTHORIZATION_FAILED points at a missing group ACL even though the topic grant is fine. See Monitoring Kafka with JMX, Prometheus, and Grafana for the scrape-and-alert setup.

A Prometheus alert, using the metric/label names the JMX exporter produces from that MBean (lowercased object name, MBean keys become labels):

groups:
  - name: kafka_authorization
    rules:
      - alert: KafkaAuthorizationFailureRate
        expr: |
          sum by (request) (
            rate(kafka_network_requestmetrics_errorspersec_total{
              error=~".*AUTHORIZATION_FAILED"
            }[5m])
          ) > 1
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Kafka authorization failures elevated"
          description: >
            {{ $value | printf "%.2f" }}/s authorization failures on
            {{ $labels.request }} requests — likely a missing ACL or an
            unauthorized client. Check broker authorizer logs.

The exact exported series name (_total suffix, count vs rate) depends on your exporter version and rules file, so confirm the series in Prometheus before shipping the alert. A > 1/s threshold catches a broken client without firing on the occasional probe; tune it to your normal baseline, which for a healthy cluster should sit at zero.

For per-decision detail, the authorizer logs every allow/deny under the kafka.authorizer.logger logger at DEBUG. Enabling it globally is far too noisy; instead raise it on one broker for the duration of an investigation using dynamic broker logging (KIP-412), applied without a restart:

# Temporarily raise authorizer logging to DEBUG on broker 0
kafka-configs --bootstrap-server localhost:9092 --command-config admin.conf \
  --entity-type broker-loggers --entity-name 0 \
  --alter --add-config "kafka.authorizer.logger=DEBUG"

Set it back to INFO the same way when you are done; broker-loggers changes persist across restarts, so a forgotten DEBUG will keep flooding the log. The official Kafka authorization documentation is the authoritative reference for resource types, operations, and version-specific behavior.

Common Pitfalls and Troubleshooting

The “no ACL found” trap. When allow.everyone.if.no.acl.found=true, a resource with zero ACLs is open to every authenticated principal. Create a topic, forget the ACLs, and it is world-readable and world-writable until you add them. Setting the property to false closes the gap but obliges you to grant access to every resource, including internal topics such as __consumer_offsets and __transaction_state.

READ without DESCRIBE. A consumer with READ on a topic but no DESCRIBE may still fail to resolve metadata. Grant both—or grant ALL—when a principal genuinely needs to consume. The symptom is a client that connects but never receives partition assignments.

Consumer-group naming mismatch. A consumer with topic READ still fails its JoinGroup with GROUP_AUTHORIZATION_FAILED if it has no READ on the group resource. Cover topics and groups with the same prefix convention so they stay in lockstep; the request-label split in the metrics above is what distinguishes this from a topic-ACL problem.

Wildcard over-permissioning. READ on * for User:* effectively disables read authorization. Audit for broad grants regularly:

# Surface wildcard principals and wildcard/prefixed resource patterns
kafka-acls --bootstrap-server localhost:9092 --command-config admin.conf \
  --list | grep -E 'principal=User:\*|patternType=PREFIXED|name=\*'

Transactional-ID omissions. EOS applications need WRITE and DESCRIBE on the TransactionalId resource in addition to topic ACLs; the giveaway is TransactionalIdAuthorizationException (protocol error TRANSACTIONAL_ID_AUTHORIZATION_FAILED, surfacing on InitProducerId). Grant on the transactional-id prefix.

Delegation-token gaps. In clusters using delegation tokens, the token is itself a resource. A principal that mints a token for another user needs CreateTokens authorization (a User-resource operation introduced in KIP-373), and DESCRIBE on the DelegationToken resource to inspect tokens. This is easy to miss during initial setup.

Lifecycle Management and Cluster Evolution

ACLs are not static. As teams onboard, applications evolve, and topics are deprecated, the ACL set drifts. A quarterly audit is a healthy cadence; it should answer three questions:

1. Orphaned ACLs? Bindings for deleted topics or retired principals clutter the store and confuse incident response.
2. Over-permissive rules? User:*, * resources, and ALL on broad prefixes should be justified or tightened.
3. Ownership drift? As teams reorganize, prefixes change; ACLs should reflect current ownership.

Automate the audit by diffing the live ACL set against the live set of topics, groups, and principals. On KRaft clusters the ACLs live in the __cluster_metadata log and are reachable through the AdminClient (describeAcls), which is the right interface for audit tooling—prefer it over scraping CLI text for anything you intend to run on a schedule, since the text format is not a stable contract.

During rolling upgrades, authorization stays in effect throughout. StandardAuthorizer is compatible across versions, but the set of supported operations and resource types can grow (for example, the User-resource token operations). Read the release notes when upgrading and update your ACL tooling to recognize any new operations. The Operations, Security & Observability pillar covers planning and executing upgrades without downtime.

Finally, ACLs are one layer. Quotas operate independently: a principal can hold WRITE on a topic and still be throttled by a produce quota. ACLs govern what a principal may do; quotas govern how much. Together with authentication and encryption, they form the defense-in-depth posture that protects the cluster from both external threats and internal misbehavior.