Quota Management and Throttling in Multi-Tenant Clusters

In a shared Apache Kafka cluster, the line between a stable platform and a noisy-neighbor incident comes down to how rigorously you bound resource consumption. When several teams or applications share brokers for cost and operational efficiency, the absence of quotas all but guarantees that one misconfigured producer or an unexpected spike will degrade latency for every other tenant. Quotas let you explicitly allocate and enforce throughput, bandwidth, and request-handling capacity, turning a contended shared resource into a governed one.

This page sits within the broader Operations, Security & Observability domain. Authentication establishes who can connect—see Securing Kafka with mTLS and SASL/SCRAM—and authorization, via Kafka ACLs and Authorization Best Practices, defines what they can access. Quotas answer a third question: how much can they consume? A client can be fully authenticated and authorized to produce to a topic and still need a hard cap on its byte rate so it cannot saturate links shared with other tenants. Quotas operate at a layer orthogonal to authentication and authorization.

The Quota Model: Client IDs, Users, and Precedence

Kafka’s quota framework operates on two entity types: client IDs and users (principals). A client ID is a logical application identifier that producers and consumers set via the client.id property. A user corresponds to the authenticated principal—typically the distinguished name from an X.509 certificate or the username from a SASL exchange. Quotas can be configured on a client ID, a user, or a (user, client-id) pair.

Precedence is the most commonly misunderstood part of the model, so be precise about it: Kafka applies the most specific matching quota, not the most restrictive one. For a given connection the broker walks a fixed hierarchy and stops at the first match:

1. /config/users/<user>/clients/<client-id>
2. /config/users/<user>/clients/<default>
3. /config/users/<user>
4. /config/users/<default>/clients/<client-id>
5. /config/users/<default>/clients/<default>
6. /config/users/<default>
7. /config/clients/<client-id>
8. /config/clients/<default>

A (user, client-id) quota wins over a user quota, which wins over a client-id quota—regardless of which number is larger. This matters in practice: if you set a generous per-user quota and a tighter per-client-id quota, the per-client-id value is not automatically the one that applies. To enforce intra-tenant fairness you must configure the more specific (user, client-id) level, because a plain user-level quota will shadow a less-specific client-id quota entirely.

Quota entities are stored as dynamic configuration: under /config/users/<user> and /config/clients/<client-id> in ZooKeeper-based clusters, or in the metadata log for KRaft clusters. Manage them through kafka-configs.sh or the AdminClient API—never by editing ZooKeeper directly. Changes propagate to all brokers within seconds without a rolling restart, which makes quotas suitable for both planned capacity allocation and emergency throttling.

# Set a producer byte-rate quota of 10 MB/s for client ID "order-service"
kafka-configs.sh --bootstrap-server kafka1:9092 \
  --entity-type clients --entity-name order-service \
  --alter --add-config 'producer_byte_rate=10485760'

# Set a consumer byte-rate quota of 5 MB/s for user principal "User:data-pipeline"
kafka-configs.sh --bootstrap-server kafka1:9092 \
  --entity-type users --entity-name 'User:data-pipeline' \
  --alter --add-config 'consumer_byte_rate=5242880'

To configure a (user, client-id) quota, supply both entity selectors in the same command:

# Per-(user, client-id) quota — the most specific level
kafka-configs.sh --bootstrap-server kafka1:9092 \
  --entity-type users --entity-name 'User:data-pipeline' \
  --entity-type clients --entity-name order-service \
  --alter --add-config 'producer_byte_rate=2097152'

Quota Types: Network Bandwidth and Request Rate

Kafka exposes four quota keys across two resource dimensions: network bandwidth and request rate.

Bandwidth Quotas

producer_byte_rate and consumer_byte_rate limit the bytes per second a client can send or receive, measured per broker. These are the most commonly deployed quotas because they directly protect the network interfaces and disk I/O paths shared across tenants.

Request Rate Quotas

request_percentage (introduced in KIP-124) limits the percentage of broker thread time a client may consume on the request-handler (I/O) and network threads within a quota window. It is effectively a CPU throttle: a client sending millions of tiny requests can starve other tenants of processing capacity without ever approaching a byte-rate limit. Crucially, the value is not capped at 100. Total broker capacity is (num.io.threads + num.network.threads) * 100%, so a broker with 8 I/O and 5 network threads has 1300% to allocate, and an individual client may legitimately be granted more than 100%.

controller_mutation_rate (KIP-599) caps the rate at which a client can perform partition-mutating admin operations: creating topics, creating partitions, and deleting topics. It is measured as partitions mutated per second, not requests per second, and it deliberately does not cover partition reassignment or arbitrary config changes. In multi-tenant environments where tenants self-serve topic creation, this quota protects the controller from a storm of create/delete operations. Its sampling window is governed by separate broker configs, controller.quota.window.size.seconds and controller.quota.window.num.

# Limit client "analytics-producer" to 5% of broker thread time
kafka-configs.sh --bootstrap-server kafka1:9092 \
  --entity-type clients --entity-name analytics-producer \
  --alter --add-config 'request_percentage=5'

# Cap user "User:platform-admin" to 10 partition mutations/sec
kafka-configs.sh --bootstrap-server kafka1:9092 \
  --entity-type users --entity-name 'User:platform-admin' \
  --alter --add-config 'controller_mutation_rate=10'

Throttling Mechanics and Client Behavior

When a client exceeds its quota the broker does not reject the request or drop the connection. Instead it computes the delay needed to bring the client back under its limit, returns a response immediately (a fetch response carries no data in this case), and then mutes the client’s channel until the delay elapses, so no further requests from that client are processed in the interim. This back-pressures the client’s send and receive buffers and slows it to the configured rate.

The client perceives throttling as elevated latency. The Java clients are not quota-aware in any active sense; they rely entirely on the broker-imposed delay, which keeps client logic simple and ensures enforcement is server-side and uncircumventable. The practical consequence is that client timeouts must accommodate the maximum expected throttle delay—request.timeout.ms and delivery.timeout.ms for producers, request.timeout.ms for consumers. Setting these too aggressively causes spurious timeout exceptions during legitimate throttling. The brokers surface the imposed delay directly via the client metrics produce-throttle-time-avg and fetch-throttle-time-avg.

The rate is tracked over a sliding set of fixed-size windows rather than a single instantaneous measurement. quota.window.size.seconds (default 1 second) sets the window width and num.quota.samples (default 11) sets how many windows are retained, so the broker averages the rate over roughly the last 10 seconds. This smoothing means a brief burst is tolerated, but sustained overage is throttled; it also bounds how a fully exhausted client is delayed rather than allowing an unbounded stall.

# Describe current quotas for client "order-service"
kafka-configs.sh --bootstrap-server kafka1:9092 \
  --entity-type clients --entity-name order-service --describe

# Sample output:
# Quota configs for client-id 'order-service' are producer_byte_rate=10485760.0

Designing Multi-Tenant Quota Policies

A coherent quota policy starts with capacity planning. Determine the cluster’s sustainable throughput—bounded by network bandwidth, disk I/O, and CPU headroom—then partition that capacity across tenants by priority and isolation requirements. Capacity changes as brokers are added or removed, so manage quotas through infrastructure-as-code that recalculates limits when cluster topology changes rather than hand-editing them.

A layered approach works well:

1. Default quotas — Define defaults with --entity-default so any client ID or user without an explicit configuration inherits a safety net instead of running unbounded.
2. Per-tenant quotas — Assign user-level quotas reflecting each tenant’s allocated share of cluster capacity.
3. Per-instance quotas — Use (user, client-id) quotas (not bare client-id quotas, which a user quota would shadow) to enforce intra-tenant fairness so one application instance cannot starve its siblings.

# Default producer byte-rate quota of 1 MB/s for all unconfigured clients
kafka-configs.sh --bootstrap-server kafka1:9092 \
  --entity-type clients --entity-default \
  --alter --add-config 'producer_byte_rate=1048576'

# Default consumer byte-rate quota of 2 MB/s for all unconfigured users
kafka-configs.sh --bootstrap-server kafka1:9092 \
  --entity-type users --entity-default \
  --alter --add-config 'consumer_byte_rate=2097152'

Quotas and Partition Placement

Quotas are enforced per broker, not cluster-wide. If a tenant’s partitions are concentrated on a subset of brokers, those brokers enforce the full configured rate locally and may throttle the client even when its cluster-wide aggregate throughput is below the limit. Quota policy and partition balancing must therefore be designed together; automated rebalancing becomes important at scale so that a configured rate translates into predictable, uniform throughput.

Monitoring, Alerting, and Observability

Kafka exposes throttle metrics through JMX, broken out by resource dimension rather than under a single “Quota” object. The relevant broker MBeans are:

• kafka.server:type=Produce,user=...,client-id=... and kafka.server:type=Fetch,user=...,client-id=... — bandwidth-quota throttling, with a throttle-time attribute (milliseconds the client was throttled).
• kafka.server:type=Request,user=...,client-id=... — request-rate (request_percentage) throttling, also with a throttle-time attribute, plus request-time (percentage of broker thread time the client consumed).

Omit the user attribute for a per-client-id quota or the client-id attribute for a per-user quota. A sustained non-zero throttle-time indicates active throttling.

Scrape these via the JMX exporter and alert on throttle time. The exact Prometheus metric names depend on your exporter’s naming rules, so confirm them against /metrics before deploying; with the standard JMX exporter the produce-side throttle typically surfaces as a kafka_server_produce_throttle_time family keyed by client_id and user:

# Example Prometheus alerting rule for sustained throttling
# (verify the metric/label names against your JMX exporter output)
groups:
- name: kafka_quotas
  rules:
  - alert: KafkaClientThrottling
    expr: avg by (client_id) (kafka_server_produce_throttle_time{client_id!=""}) > 0
    for: 10m
    labels:
      severity: warning
    annotations:
      summary: "Client {{ $labels.client_id }} is being throttled"
      description: "Sustained produce throttle time for client {{ $labels.client_id }} over 10 minutes."

Throttle time is a leading indicator of tenant dissatisfaction—it tells you a client is being constrained before a ticket is filed. Pair it with broker-level saturation signals (network throughput against link capacity, CPU, disk I/O wait) to distinguish quotas working as designed from quotas masking a genuine capacity shortfall.

Client-side observability completes the picture. The dedicated indicators are produce-throttle-time-avg on the producer and fetch-throttle-time-avg on the consumer, which report the broker-imposed delay directly. General latency and utilization metrics—record-queue-time-avg, request-latency-avg, and io-ratio on producers; fetch-latency-avg and io-ratio on consumers—help confirm the downstream effect on application performance. In a mature platform, expose per-tenant dashboards of current quota utilization and throttle events so tenants can self-diagnose.

Overrides, Exemptions, and Emergency Procedures

A rigid policy needs an escape hatch. Overrides use the same dynamic-configuration mechanism to temporarily raise or remove limits for a specific entity without touching the baseline—useful for backfills, DR drills, or a negotiated temporary increase. The risk is obvious: a temporary override that is forgotten becomes a permanent loophole. Require every override to be time-bounded and tracked, recording reason, requester, and expiry, and have an automated process revert it. Implement this with the AdminClient API plus a scheduled sweep for expired overrides, or with a GitOps workflow where quota config lives in version control and a pipeline enforces review and expiration.

# Temporarily raise producer byte rate to 100 MB/s for a backfill
# (a separate process must revert this after the backfill window)
kafka-configs.sh --bootstrap-server kafka1:9092 \
  --entity-type clients --entity-name backfill-job \
  --alter --add-config 'producer_byte_rate=104857600'

# Remove the override to fall back to the default or a more-specific quota
kafka-configs.sh --bootstrap-server kafka1:9092 \
  --entity-type clients --entity-name backfill-job \
  --alter --delete-config 'producer_byte_rate'

When a client saturates the cluster despite quotas—because the limit was set too high, or it is exploiting a request pattern your existing keys don’t cover—you can drop its limits to near-zero within seconds, effectively quarantining it until you diagnose the root cause. Document this in incident runbooks and rehearse it during game days so the response is reflexive during a real incident.

Quotas in a Broader Operational Context

Quotas intersect with the rest of the Operations, Security & Observability pillar. They act on the principal identities that authentication establishes via mTLS and SASL/SCRAM, covered in Securing Kafka with mTLS and SASL/SCRAM; if principals are shared or spoofable, user-level quotas are easy to evade by switching identities. Authorization, via Kafka ACLs and Authorization Best Practices, controls access but not consumption: a principal with produce access can fill a broker’s disk and trigger cascading failures unless a quota bounds its impact.

Quotas also slot into capacity planning and cost models. Because they define the unit of resource allocation per tenant, they make a natural basis for chargeback or showback—by tracking bytes produced and consumed, request percentage, and throttle time per tenant, you can attribute cluster cost to the teams that drive it.

For the authoritative reference, see the Apache Kafka documentation on Quotas, and the design details in KIP-124 (request rate quotas) and KIP-599 (controller mutation quotas).