Resetting Consumer Offsets with kafka-consumer-groups CLI

Repositioning a consumer group’s committed offsets is a routine but high-blast-radius operation for platform and SRE engineers running Apache Kafka. You reach for it when recovering from a poison-pill record, replaying history after a processing-logic fix, or abandoning a backlog to catch up on fresh data. The kafka-consumer-groups.sh tool that ships with Kafka is the canonical way to do this. This guide covers the reset mechanics, the prerequisites that the tool enforces, every reset strategy it supports, and the production patterns and failure modes you will actually hit.

How an offset reset works

A consumer group’s progress is stored as committed offsets in the internal __consumer_offsets topic, keyed by (group, topic, partition). The CLI never touches the partition log files and never rewrites data. It computes a target offset per partition according to the strategy you choose (--to-earliest, --to-offset, --shift-by, and so on) and then commits those offsets to the group coordinator, which writes them to __consumer_offsets like any other commit. The new position becomes durable once the commit message is replicated to the in-sync replicas of that internal topic.

Because the reset is just a commit routed through the coordinator, the target group must be empty — no active members — while you run it. If a consumer were still attached, its automatic commits would race your manual one and the tool would have no stable assignment to write against. This is why the CLI rejects a reset against a live group, as described below. For how commits relate to delivery guarantees more broadly, see Producers, Consumers & Delivery Semantics.

Prerequisites and safety validation

A reset against a group with running instances is rejected. The --execute action fails unless the group is in the Empty state; if any member is connected the group is Stable (or transiently PreparingRebalance / CompletingRebalance) and the command refuses to proceed. So the first operational step is always to stop every consumer in the target group, then confirm the state with --describe:

# Check current state and committed offsets for the group
kafka-consumer-groups.sh \
  --bootstrap-server kafka-broker-1:9092,kafka-broker-2:9092 \
  --group payment-processor-v3 \
  --describe --state

You must see Empty before continuing. After you stop the application instances, the coordinator only declares the group empty once the last member’s session expires; that lag is bounded by the group’s session.timeout.ms (default 45000 ms / 45 s since Kafka 3.0). If the state stays Stable, an instance is still running.

On a cluster with authorization enabled, the principal running the reset needs:

• Describe on the Group and Describe on the Topic to run --describe.
• Read on the Group and Read on the Topic to commit the new offsets. Offset reset uses the same authorization as a normal offset commit (the OFFSET_COMMIT API), which requires Read on both the group and the topic. There is no separate “Write on group” operation — Read on the group is what authorizes the offset change.

The complete operation-to-resource mapping is in the Kafka authorization and ACLs documentation.

Always dry-run first

Run the reset with --dry-run before --execute. The dry run prints the assignment the tool would commit — each partition with its current offset and proposed new offset — without writing anything. It is the last check between you and accidentally rewinding a group to the earliest offset when you meant to skip three messages.

# Preview a reset-to-earliest without committing
kafka-consumer-groups.sh \
  --bootstrap-server kafka-broker-1:9092 \
  --group payment-processor-v3 \
  --topic transactions \
  --reset-offsets --to-earliest \
  --dry-run

If you pass neither --dry-run nor --execute, the tool defaults to dry-run behavior and only prints the plan. Read that output carefully before adding --execute. For how strategy choice affects reliability, see Offset Management Strategies for Reliable Processing.

Reset strategies and syntax

--reset-offsets requires exactly one strategy flag. The supported strategies are:

• --to-earliest — the earliest retained offset in each partition. Use it to replay a topic from the start of what is still retained (not necessarily offset 0, since older segments may have been deleted by retention).
• --to-latest — the log-end offset, skipping everything currently in the partition.
• --to-offset <N> — a specific absolute offset. If N is below the earliest or above the latest offset, the tool clamps it to the partition’s earliest/latest bound rather than failing.
• --shift-by <N> — shifts relative to the group’s current committed offset. Positive values move forward (skip); negative values move backward (replay).
• --to-datetime <YYYY-MM-DDTHH:mm:SS.sss> — the first offset whose record timestamp is greater than or equal to the given time. An offset (UTC) suffix is accepted, e.g. 2025-01-15T14:30:00.000+00:00.
• --by-duration <PnYnMnDTnHnMnS> — like --to-datetime but relative to now, using ISO-8601 duration syntax, e.g. PT5M for five minutes ago.
• --to-current — the group’s current committed offset (a no-op target, useful as a base when combined with file output).
• --from-file <csv> — per-partition offsets from a CSV file. Pair it with --dry-run --export to dump the current plan to a file, edit it, then replay with --from-file.

Choosing among them is mostly a question of what you know about the target position. The table maps the common operational intents to the right flag:

When you use --export, the tool prints (or writes) one CSV line per partition in topic,partition,offset form, for example:

transactions,0,487292
transactions,1,490115
transactions,2,488003

--from-file reads that same format back, so the export-edit-replay loop lets you reset different partitions to different offsets in a single, reviewable command — something none of the single-target strategies can express.

General execute form:

kafka-consumer-groups.sh \
  --bootstrap-server <broker-list> \
  --group <group-id> \
  --topic <topic-name> \
  --reset-offsets \
  <strategy-flag> \
  --execute

Scope the reset to specific partitions by appending them to the topic, e.g. --topic transactions:0,2,4. To target several topics use repeated --topic flags, or --all-topics to cover every topic the group has committed offsets for. Without a partition or topic filter, the reset applies to all partitions the group has committed offsets on.

Production execution patterns

Replaying after a logic change

When you ship a consumer that fixes a processing bug, you usually want to reprocess records from just before the bug shipped. --to-datetime rewinds to the first record at or after a wall-clock time, which is more robust than guessing raw offsets:

# Rewind to records with timestamp >= the given time
kafka-consumer-groups.sh \
  --bootstrap-server kafka-broker-1:9092,kafka-broker-2:9092 \
  --group fraud-detection-v2 \
  --topic transactions \
  --reset-offsets \
  --to-datetime 2025-01-15T14:30:00.000 \
  --execute

Note that --to-datetime matches on the record’s timestamp, which is either the producer’s create time or the broker’s log-append time depending on the topic’s message.timestamp.type. If producers backfill or clocks skew, the offset you land on may not be what you expect — confirm with the dry run. After execution, re---describe, restart consumers, and watch lag for the first few minutes.

Skipping a poison-pill record

A poison pill is a record that deterministically crashes the consumer, stalling its partition. To skip it, advance the committed offset past it. Get the offending offset and partition from your consumer logs, then set the offset to the value immediately after the bad record:

# Skip past offset 487291 on partition 3 (commit 487292)
kafka-consumer-groups.sh \
  --bootstrap-server kafka-broker-1:9092 \
  --group order-fulfillment \
  --topic orders:3 \
  --reset-offsets \
  --to-offset 487292 \
  --execute

Capture the skipped record somewhere — a dead-letter topic or your error store — before you move past it, so the data is not simply lost. Scope this to the single affected partition (orders:3), not the whole topic: a bare --to-offset 487292 would force every partition to that absolute offset, which is almost never what you want.

Dropping a backlog

After a downstream outage, a group can accumulate a large backlog. Once downstream recovers and you would rather process fresh data than grind through the backlog, jump to the end with --to-latest:

# Skip the backlog and resume from the log-end offset
kafka-consumer-groups.sh \
  --bootstrap-server kafka-broker-1:9092,kafka-broker-2:9092 \
  --group analytics-aggregator \
  --topic clickstream \
  --reset-offsets \
  --to-latest \
  --execute

This discards the unprocessed records for that group — confirm your SLAs tolerate the gap. If you only want to drop part of the backlog, --by-duration (e.g. PT10M) or a negative --shift-by lets you keep a recent window instead.

Validating and monitoring after a reset

Re-run --describe immediately after --execute and compare the committed offsets against your dry-run plan:

# Confirm the new committed offsets and lag
kafka-consumer-groups.sh \
  --bootstrap-server kafka-broker-1:9092 \
  --group payment-processor-v3 \
  --describe

The --describe output is itself your most reliable lag source: the LAG column is log-end-offset − committed-offset per partition. After you restart consumers, lag will spike (expected after a rewind) and should fall steadily; flat lag means consumers are stuck or never rejoined.

For dashboards, the consumer-side JMX metric is kafka.consumer:type=consumer-fetch-manager-metrics,client-id=* with attributes records-lag-max (and per-partition records-lag under ...,topic=*,partition=*). Note this reflects fetch lag of a running client, so it only exists while consumers are attached — it is not a group-level committed-lag metric and reports nothing during the reset. For group-level committed lag in Prometheus, scrape --describe output or run an exporter such as kafka-lag-exporter or Burrow, which expose series like kafka_consumergroup_lag. See the Kafka monitoring documentation for the metric catalog. Also alert on group state: a group that stays Empty long after a planned reset means consumers never rejoined.

Troubleshooting common failures

• “Assignments can only be reset if the group is inactive” (or similar rejection): a member is still connected. Stop every instance and wait for session.timeout.ms to elapse so the group goes Empty; re-check with --describe --state.
• “Topic not found”: the group references a topic that was deleted or never existed. List the group’s committed topics with --describe and confirm with kafka-topics.sh --list.
• Authorization failures: grant Describe on the group/topic for --describe, and Read on both the group and topic for the commit that --execute performs. (Read implies Describe, so Read on both is sufficient for the whole flow.) See the ACL documentation.
• Querying valid offset ranges: before an absolute --to-offset, inspect each partition’s earliest/latest offsets with kafka-get-offsets.sh (Kafka 3.4+) or the older kafka-run-class.sh kafka.tools.GetOffsetShell. Note the reset tool clamps out-of-range targets to the partition bounds rather than erroring, so verify the dry-run plan landed where you intended.

A word on re-running: absolute strategies (--to-earliest, --to-latest, --to-offset, --to-datetime, --by-duration) are idempotent — applying them twice yields the same committed offset. --shift-by is not idempotent: it moves relative to the current committed offset, so a second run shifts again. If a reset is interrupted, re-run an absolute strategy freely, but recompute before re-applying a --shift-by.

Offset resets are powerful and unforgiving. Stop the group, dry-run first, read the per-partition plan, execute, then verify with --describe and watch lag drain. Folded into your wider offset management strategy, that discipline keeps a sharp tool safe in production.