Offset Management Strategies for Reliable Processing

Offset management determines whether Kafka consumers deliver data consistently, recover from failures, and uphold the processing guarantees your application depends on. When a consumer commits an offset, it declares that everything before that point has been processed and everything after remains to be handled. The strategies you choose shape resilience to crashes, your ability to reprocess during incidents, and your end-to-end delivery semantics.

This article examines offset management from a production-operations perspective for platform engineers, SREs, and backend developers running Kafka at scale. We cover automatic versus manual commit strategies, procedures for resetting offsets during incident recovery, architectural patterns that prevent data loss and duplication, and monitoring practices for offset behavior. Throughout, we ground the discussion in Producers, Consumers & Delivery Semantics, where producer configuration and consumer offset management together determine pipeline reliability.

Understanding offset management requires a clear mental model of the consumer group protocol. Each consumer in a group is assigned a set of partitions. For each partition, the consumer maintains its current position—the offset of the next record it will read. The committed offset is separate: the position the consumer reports back to the broker, stored in the internal __consumer_offsets topic. When a consumer restarts or a Consumer Group Rebalancing: Protocols and Tuning event occurs, the consumer that takes over a partition resumes from its committed offset. The gap between current position and committed offset represents records consumed but not yet acknowledged—and it is within this gap that the most consequential operational decisions reside.

The Offset Commit Lifecycle and Its Failure Modes

To design reliable offset management, you must understand what happens during a commit and how it can fail. When a consumer calls commitSync() or commitAsync(), it sends an OffsetCommitRequest to the group coordinator. The request carries the consumer group ID, the generation ID (which identifies the current rebalance generation), the member ID, and a map of topic-partitions to offsets. The coordinator validates the generation ID against the current group generation; if a rebalance has occurred since the consumer last joined, the commit is rejected. The client surfaces this as a CommitFailedException (the rebalance already completed and partitions were reassigned) or a RebalanceInProgressException (a rebalance is in flight). This rejection is a safety mechanism that prevents a consumer from committing offsets for partitions it no longer owns.

The coordinator writes the commit to the __consumer_offsets topic, a compacted log. Because this topic is replicated and durable like any other Kafka topic, a commit is durable only once it has been replicated to the offsets-topic partition’s in-sync replicas. The broker delays the response until replication completes or the offsets.commit.timeout.ms deadline is reached. (Older releases exposed a offsets.commit.required.acks broker config to tune this; it was deprecated and removed in Apache Kafka 4.0 via KAFKA-16736, and offset commits now always wait for all in-sync replicas.) If the coordinator fails before replication completes, the commit can be lost. This is why commitSync() blocks until the coordinator acknowledges the write, whereas commitAsync() invokes a callback but does not guarantee persistence before returning control.

Several failure modes emerge from this lifecycle:

• A consumer processes a batch successfully but fails to commit—due to a network partition or a rebalance—so those records are reprocessed after the next assignment.
• A consumer commits offsets for records it has not fully processed—for example, under auto-commit with a fixed interval—causing data loss if it crashes before processing completes.
• The interaction between commit timing and processing logic is where most production incidents originate.

Consider a consumer processing financial transactions, where each record triggers a database write. If it commits the offset immediately after reading but before the write completes, a crash loses the transaction permanently. If it commits after the write, a crash between the write and the commit processes the transaction twice. Neither is acceptable without additional coordination, which is why transactional patterns and idempotency are necessary for exactly-once processing. The Idempotent Producers and Exactly-Once Semantics article explores these patterns; the offset management layer must be aligned with them for end-to-end guarantees.

Diagnosing Commit Failures in Production

When commits fail, the symptoms are subtle. Consumers may appear healthy but silently fall behind, or they may repeatedly rejoin the group as failed commits trigger rebalances. Start by inspecting consumer group state with the kafka-consumer-groups CLI:

kafka-consumer-groups --bootstrap-server broker1:9092,broker2:9092 \
  --group order-processing-service \
  --describe

This outputs the current offset, log-end offset, lag, and the consumer/client ID assigned to each partition. A non-zero lag that is not decreasing, or a partition with no assigned consumer, points to stalled progress. Because Kafka does not expose a dedicated failed-commit JMX metric, commit rejections are diagnosed primarily from consumer logs: search for CommitFailedException and RebalanceInProgressException. Common causes:

• Slow or overloaded offsets topic: If the __consumer_offsets topic is under-replicated or the coordinator is overloaded, commit requests can time out. The consumer’s request.timeout.ms defaults to 30 seconds; persistent timeouts usually point to coordinator or offsets-topic health rather than a value that needs raising.
• Generation mismatch: If processing a batch takes longer than max.poll.interval.ms, the consumer is removed from the group and its next commit fails. This signals that processing is too slow or max.poll.interval.ms is set too low.
• Coordinator unavailable: If the group coordinator broker is down and no replacement has been elected, commits fail until a new coordinator takes over. This is uncommon in healthy clusters but appears during rolling restarts or network partitions.

To investigate coordinator issues, look up the group’s coordinator and state:

kafka-consumer-groups --bootstrap-server broker1:9092 \
  --group order-processing-service \
  --describe --state

The --state output includes the coordinator’s broker ID and the group’s current state. Cross-reference that broker’s logs for GC pauses, disk I/O saturation, or network issues. If a small number of brokers host coordinators for many groups, consider increasing the __consumer_offsets partition count (default 50) so coordinator responsibility spreads across more brokers.

Automatic vs. Manual Offset Commits: Operational Trade-offs

The consumer offers two commit modes: automatic (enable.auto.commit=true, the default) and manual (commitSync() or commitAsync()). The choice follows directly from your processing semantics and failure tolerance. Auto-commit is simpler but opens a window between the commit and processing completion. Manual commits give precise control but require careful error handling to avoid loss or duplication.

Automatic Commit: The Hidden Risks

With enable.auto.commit=true, the consumer periodically commits the offsets of records returned by poll(), regardless of whether the application has finished processing them. The commit is performed during poll() once auto.commit.interval.ms (default 5 seconds) has elapsed. In effect, roughly every interval the consumer tells the broker “everything up to here is processed,” even if the application is still working through the batch.

The danger is acute when processing has side effects that cannot be rolled back. Imagine a consumer that sends an email per record. If it crashes shortly after an auto-commit, records since that commit are reprocessed and duplicate emails go out. If it crashes after processing some records but before the next auto-commit, those records’ offsets are still advanced on the next successful auto-commit only if they were returned by a prior poll—but records processed and then lost before any commit are skipped on restart. Either way, the interval creates a window of uncertainty that is fundamentally incompatible with strict at-least-once or exactly-once guarantees.

Despite this, auto-commit is acceptable where occasional duplication or loss is tolerable: analytics pipelines computing approximate aggregates, or log shipping where duplicates are filtered downstream. If you use auto-commit, also set auto.offset.reset to control behavior when no committed offset exists. The default latest means a new group reads only new messages; earliest reads from the start of the topic. Services that must process historical data often prefer earliest, paired with idempotent processing to absorb the resulting duplicates.

A minimal auto-commit configuration for low-risk, high-throughput pipelines:

Properties props = new Properties();
props.put("bootstrap.servers", "broker1:9092,broker2:9092");
props.put("group.id", "analytics-pipeline");
props.put("enable.auto.commit", "true");
props.put("auto.commit.interval.ms", "10000");
props.put("auto.offset.reset", "earliest");
props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");

A longer commit interval reduces commit frequency and coordinator load but widens the window of potential reprocessing. Tune it against your tolerance for duplication and the volume flowing through the pipeline.

Manual Commit: Synchronous vs. Asynchronous Control

Manual commits let you commit only after processing succeeds. commitSync() blocks until the commit is acknowledged or fails; commitAsync() returns immediately and invokes a callback with the result. The choice depends on whether you can tolerate blocking and how you want to handle failures.

commitSync() is the simpler, safer default. It commits in order and surfaces any failure directly to the calling thread, retrying internally on transient errors until it succeeds or hits an unrecoverable error. Its cost is latency: the consumer waits for the coordinator before polling again. Commit synchronously after each batch, handling failure explicitly:

while (true) {
    ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(1000));
    for (ConsumerRecord<String, String> record : records) {
        processRecord(record);
    }
    try {
        consumer.commitSync();
    } catch (CommitFailedException e) {
        logger.error("Commit failed for group {}", groupId, e);
        // A rebalance reassigned these partitions; the new owner will
        // reprocess from the last committed offset. Do not blindly retry.
    }
}

commitAsync() does not block the poll loop, so the consumer keeps fetching while a commit is in flight, improving throughput. The trade-off is that async commits can complete out of order, so a failed commit should generally not be retried blindly (a later commit may already have advanced the offset). A robust pattern uses commitAsync() with a logging callback during steady state, plus a final commitSync() on shutdown:

try {
    while (true) {
        ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(1000));
        for (ConsumerRecord<String, String> record : records) {
            processRecord(record);
        }
        consumer.commitAsync((offsets, exception) -> {
            if (exception != null) {
                logger.error("Async commit failed for offsets {}", offsets, exception);
                // Avoid unconditional retries: a subsequent commit may
                // already cover these offsets, or partitions may be revoked.
            }
        });
    }
} finally {
    try {
        consumer.commitSync(); // block to persist final offsets on exit
    } finally {
        consumer.close();
    }
}

Async commits during normal operation with a sync commit on shutdown balance performance and safety. This still does not provide exactly-once: for that, integrate offset commits with your output using Kafka transactions, covered below.

Offset Reset Operations: Incident Recovery Procedures

When a consumer group hits a poison-pill message, a bug that corrupts state, or a downstream outage that requires reprocessing, you reset offsets to a specific point in the stream. This is one of the most common Kafka SRE tasks and must be done precisely to avoid loss or unintended duplication. The Resetting Consumer Offsets with kafka-consumer-groups CLI article walks through the tooling in detail; here we focus on the decisions and safety procedures around resets.

Choosing the Reset Target

kafka-consumer-groups --reset-offsets supports several target strategies, each suited to a different scenario:

• --to-earliest: Resets to the earliest retained offset of each partition. Use this to reprocess all retained data, such as after a schema change that requires re-deriving downstream state. It can generate enormous reprocessing volume and overwhelm downstream systems if not throttled.
• --to-latest: Resets to the log-end offset, discarding all current lag. Appropriate when the lag is stale and the data is no longer needed, or when seeding a new group that should only process new data.
• --to-offset <n>: Resets to a specific offset. The most precise option, but it requires knowing the exact offset per partition.
• --to-datetime <timestamp>: Resets to the earliest offset whose message timestamp is greater than or equal to the given time. The timestamp uses ISO-8601 with milliseconds (for example, 2025-01-15T14:30:00.000); append an offset such as Z or +00:00 to pin the time zone, otherwise the tool uses the client’s local zone.
• --shift-by <n>: Shifts the current offset forward (positive) or backward (negative) by a number of records. Useful for skipping a known count of poison-pill messages or rewinding a recent window.

Before any reset, stop all consumers in the group. The command fails while the group is active—a safety feature that prevents races between the reset and ongoing commits. In production, drain the group by scaling it to zero instances or by gracefully shutting down all consumer pods/processes through your orchestrator.

Executing a Safe Offset Reset

The following demonstrates a datetime reset, the most common incident-recovery scenario. Suppose a bug was deployed at 2025-01-15T14:30:00 UTC and you must reprocess everything from that point. First, capture the current state:

kafka-consumer-groups --bootstrap-server broker1:9092 \
  --group order-processing-service \
  --describe

Note the current offsets and lag. Then, always dry-run first to validate the proposed offsets. The reset is a preview unless --execute is supplied; --dry-run is the explicit, default-equivalent preview mode:

kafka-consumer-groups --bootstrap-server broker1:9092 \
  --group order-processing-service \
  --topic orders \
  --reset-offsets \
  --to-datetime 2025-01-15T14:30:00.000Z \
  --dry-run

The output shows the proposed new offset for each partition. Because each partition has its own offset sequence and message timestamps are not perfectly aligned across partitions, the per-partition offsets will differ—this is expected. Once verified, run the same command with --execute:

kafka-consumer-groups --bootstrap-server broker1:9092 \
  --group order-processing-service \
  --topic orders \
  --reset-offsets \
  --to-datetime 2025-01-15T14:30:00.000Z \
  --execute

After the reset, restart your consumers. They resume from the new offsets and begin reprocessing. Watch consumer lag to confirm progress and that downstream systems are not overwhelmed; if the reprocessing volume is large, rate-limit within the consumer.

Handling Offset Reset Failures

Resets fail for a few predictable reasons. The most common is an active group: the command reports Error: Assignments can only be reset if the group is inactive, but the current state is Stable. Ensure all consumers have stopped and the coordinator has registered their departure (this can take up to session.timeout.ms, or until a clean LeaveGroup is processed). A second case: a datetime that predates the earliest retained message. Kafka enforces topic retention, so if the target falls outside the retention window, --to-datetime resolves to the earliest available offset—any data older than that is already gone, making some loss unavoidable. This is why topic retention must be aligned with your recovery objectives.

Architectural Patterns for Preventing Data Loss and Duplication

Offset management must be integrated with application architecture to achieve the guarantees you want. Several patterns combine offset management with idempotency, transactional boundaries, and external state to prevent loss and minimize duplication.

The Idempotent Consumer Pattern

An idempotent consumer can process the same record multiple times without changing the final result. It is the most robust defense against the at-least-once behavior that manual commit-after-processing produces. If processing is idempotent, you can commit after processing without fear of duplicates from reprocessing.

Idempotency typically means recording a unique identifier per record—a message key or a dedicated ID field—in the output system, then making the side effect conditional on it. A consumer writing to a relational database can use an upsert that is idempotent by design:

INSERT INTO processed_orders (order_id, amount, processed_at)
VALUES ('ord-12345', 99.99, NOW())
ON CONFLICT (order_id) DO NOTHING;

A duplicate order_id is silently ignored. This works when the output system supports unique constraints or conditional writes. For systems that do not, maintain a separate dedup store—a Redis set or a compacted Kafka topic—keyed by record ID, checked before and updated after processing. That store must itself be fault-tolerant and consistent with your offset commits; otherwise losing the dedup state reintroduces duplicates.

Transactional Offset Management

For exactly-once processing—each record reflected in the output exactly once—Kafka transactions atomically commit both the consumer offsets and the producer outputs. This is the basis of Kafka’s exactly-once guarantees, covered in Idempotent Producers and Exactly-Once Semantics. Operationally, the key detail is that offsets are committed as part of the same transaction as the outputs, so either both commit or neither does.

A transactional consume-transform-produce loop sets the producer’s transactional.id, disables consumer auto-commit, calls initTransactions() once, and sends offsets through the producer via sendOffsetsToTransaction(). Downstream consumers of the output topic should set isolation.level=read_committed so they never see records from aborted or in-flight transactions:

producer.initTransactions();
while (true) {
    ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(1000));
    producer.beginTransaction();
    for (ConsumerRecord<String, String> record : records) {
        ProducerRecord<String, String> outputRecord = transform(record);
        producer.send(outputRecord);
    }
    Map<TopicPartition, OffsetAndMetadata> offsets = offsetsToCommit(records);
    producer.sendOffsetsToTransaction(offsets, consumer.groupMetadata());
    producer.commitTransaction();
}

Transactions are not free. Each one adds round trips to the transaction coordinator, and read_committed consumers cannot read past an open transaction’s records, which raises end-to-end latency. The producer’s transaction.timeout.ms (default 60 seconds, capped by the broker’s transaction.max.timeout.ms) bounds how long a transaction may stay open; if processing exceeds it, the transaction aborts and its records are reprocessed. Tuning this timeout and watching abort behavior—via the producer’s transaction metrics under kafka.producer:type=producer-metrics (for example txn-abort-time-ns-total)—are essential for transactional pipelines.

The Outbox Pattern for Dual Writes

A recurring challenge is the dual-write problem: a consumer must update a database and publish an event to another topic atomically. A crash between the two leaves the system inconsistent. The outbox pattern writes the outgoing event into an “outbox” table within the same database transaction as the business update. A separate process—a Debezium connector or a custom change-data-capture (CDC) worker—tails the outbox table and publishes the events to Kafka.

From the offset perspective, the consumer’s commit is decoupled from event publishing. The consumer commits after the database transaction (business update plus outbox insert) succeeds. If it crashes after the commit but before the CDC worker publishes, the event is still eventually published because it is durably stored in the outbox. This yields at-least-once publishing and is widely used in microservices. The operational cost is running the CDC pipeline and purging successfully published rows so the outbox table does not grow unbounded.

Monitoring Offset Behavior and Consumer Lag

Visibility into commit behavior is essential for catching problems before they cause loss or delay. Consumer lag—the difference between a partition’s log-end offset and the consumer’s committed offset—is the headline metric, but lag alone is incomplete. Also track commit rate, commit latency, and rebalance frequency.

Key Metrics to Monitor

These JMX metrics, exposed by the consumer client, give a rounded view of offset health:

• kafka.consumer:type=consumer-fetch-manager-metrics,client-id=* — records-lag-max: the maximum lag across this consumer’s assigned partitions. A steady climb means the consumer cannot keep up with the produce rate; a sudden spike suggests a processing stall.
• kafka.consumer:type=consumer-coordinator-metrics,client-id=* — commit-rate (commits per second) and commit-latency-avg / commit-latency-max (time per commit request). A falling commit rate alongside rising commit latency points to coordinator overload or offsets-topic issues.
• kafka.consumer:type=consumer-coordinator-metrics,client-id=* — rebalance-rate-per-hour and failed-rebalance-rate-per-hour: how often the group rebalances and how often a rebalance fails. Frequent or failing rebalances disrupt processing through repeated partition reassignment.

Kafka does not expose a dedicated failed-commit-rate metric. Commit rejections are detected indirectly—through a stalled commit-rate, elevated failed-rebalance-rate-per-hour, and CommitFailedException / RebalanceInProgressException entries in consumer logs—so route consumer logs into your observability stack and alert on those exceptions.

Scrape these into Prometheus and chart them; the Monitoring Consumer Lag with JMX and Prometheus article details that pipeline. For a quick point-in-time check, the CLI prints per-partition lag (the LAG column is the 6th field of --describe output):

kafka-consumer-groups --bootstrap-server broker1:9092 \
  --group order-processing-service \
  --describe

Read the PARTITION, CURRENT-OFFSET, LOG-END-OFFSET, and LAG columns directly rather than post-processing with brittle column math, since the column layout has changed across releases.

Alerting on Offset Anomalies

Effective alerting distinguishes transient spikes from sustained problems. Alerting on absolute lag generates false positives during normal traffic swings. Instead, alert on the rate of change: for example, if lag grows more than 50% over a 5-minute window and stays elevated for another 5 minutes, fire. This filters out brief producer bursts the consumer quickly absorbs.

Because there is no failed-commit metric, build the commit-health alert from real signals: a sustained drop in commit-rate to zero while the consumer is assigned partitions, a rising commit-latency-avg, and a non-zero failed-rebalance-rate-per-hour, corroborated by CommitFailedException log volume. Any of these sustained over a 1-minute window warrants paging—when offsets stop committing, the next rebalance or restart resumes from the last successful commit, potentially far in the past, triggering a reprocessing spike.

For transactional pipelines, watch the producer’s transaction metrics and your own abort counter. Rising aborts mean records are being reprocessed, which produces duplicates unless the consumer is idempotent, and often indicate transaction.timeout.ms is too low for current processing latency.

Tuning Offset Management for Performance and Resilience

The default consumer configuration targets out-of-the-box correctness, not production throughput. Tuning balances throughput, latency, and durability against your workload.

Commit Frequency and Batching

Commit frequency trades durability of progress against coordinator load. Committing after every record floods the coordinator and becomes a bottleneck; committing too rarely increases reprocessing after a failure. The right cadence depends on your processing rate and recovery time objective (RTO).

A good starting point is to commit after each polled batch, sized by max.poll.records (default 500), so each commit covers meaningful work without being chatty. If processing is light and you handle thousands of records per second, commit less often—say every 10,000 records. Control the poll batch size:

props.put("max.poll.records", "1000");

Then track processed records and commit when a threshold is reached:

int processedCount = 0;
while (true) {
    ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(1000));
    for (ConsumerRecord<String, String> record : records) {
        processRecord(record);
        processedCount++;
    }
    if (processedCount >= 10000) {
        consumer.commitSync();
        processedCount = 0;
    }
}

This gives explicit control over commit cadence and aligns it with batch boundaries.

Session Timeout and Heartbeat Configuration

The group protocol uses heartbeats to detect failures. With the classic group protocol, session.timeout.ms (default 45000) sets how long the coordinator waits without a heartbeat before declaring a member dead and triggering a rebalance, and heartbeat.interval.ms (default 3000) sets the heartbeat cadence; the chosen session.timeout.ms must fall within the broker’s group.min.session.timeout.ms and group.max.session.timeout.ms. With the newer KIP-848 consumer rebalance protocol (group.protocol=consumer), the session timeout is set by the broker via group.consumer.session.timeout.ms and the client-side heartbeat/session timeout settings no longer apply.

Liveness during processing is governed separately by max.poll.interval.ms (default 300000, i.e. 5 minutes): if the application does not call poll() within this interval, the consumer proactively leaves the group, which is what protects you from a heartbeat thread that keeps beating while processing is stuck. Set it above your worst-case single-batch processing time. A balanced classic-protocol configuration for a moderately heavy workload:

props.put("session.timeout.ms", "30000");
props.put("heartbeat.interval.ms", "10000");
props.put("max.poll.interval.ms", "600000"); // 10 minutes

Keep heartbeat.interval.ms no higher than roughly one-third of session.timeout.ms so several heartbeats can be missed before the session expires, and give max.poll.interval.ms a comfortable margin over your maximum processing time. A very large max.poll.interval.ms is a real trade-off: a genuinely stuck consumer takes correspondingly longer to be evicted.

Coordinator Load and the __consumer_offsets Topic

The __consumer_offsets topic underpins offset management, and its health directly affects commit latency and reliability. It defaults to 50 partitions, which determines how many group coordinators can be spread across the cluster. With thousands of consumer groups, coordinator load can concentrate on a few brokers and produce commit timeouts and rebalance storms.

Set offsets.topic.num.partitions (default 50) before the topic is first created to size coordinator capacity. If the topic already exists, you can grow it with kafka-topics --alter, but do so during a maintenance window: changing a topic’s partition count changes key-to-partition routing, and for __consumer_offsets that means existing groups can be reassigned to different coordinator partitions, disrupting commits.

kafka-topics --bootstrap-server broker1:9092 \
  --alter --topic __consumer_offsets \
  --partitions 100

After the change, coordinator responsibility redistributes across more brokers; compare commit-latency-avg before and after to confirm improvement. Keep offsets.topic.replication.factor at 3 for production (this too is fixed at topic-creation time), and size segments appropriately—offsets.topic.segment.bytes defaults to 104857600 (100 MiB). Very high commit volume may justify a larger segment size to reduce segment rolling and the disk I/O spikes that accompany it.

Operational Runbooks for Offset-Related Incidents

When offset management fails in production, the response must be fast and precise. These runbooks cover the most common incidents.

Runbook 1: Consumer Group Stuck with High Lag

Symptoms: Lag rises monotonically, consumers are not crashing, and no rebalances occur. records-lag-max is above threshold.

Diagnosis:

1. Confirm the consumers are actually processing by checking records-consumed-rate (under consumer-fetch-manager-metrics). Near-zero means they are stuck in processing or blocked on an external resource.
2. Inspect consumer logs for exceptions or timeouts on external calls.
3. Run kafka-consumer-groups --describe to confirm all partitions are assigned and consumer/client IDs are stable.
4. Check fetch-rate and fetch-latency-avg (under consumer-fetch-manager-metrics) to confirm consumers can fetch from brokers; high fetch latency points to broker-side issues.

Recovery:

• If stuck on a specific record (poison pill), skip it by shifting the offset forward by 1 on the affected partition with --shift-by 1 scoped to that partition.
• If blocked on a downed external resource, restart with a circuit breaker that fails fast and routes the failing record to a dead-letter topic, then commits, so the pipeline does not stall.
• If the cause is sustained producer growth, scale the consumer group out. Ensure the topic has enough partitions for the added instances; if not, increasing partition count is a more involved operation.

Runbook 2: Frequent Consumer Group Rebalances

Symptoms: rebalance-rate-per-hour (and possibly failed-rebalance-rate-per-hour) is elevated, consumers churn in and out of the group, and throughput is unstable.

Diagnosis:

1. Inspect consumer logs for CommitFailedException and RebalanceInProgressException, which indicate commits arriving after a rebalance has begun.
2. Check max.poll.interval.ms: if processing one poll’s batch exceeds it, the consumer leaves the group and forces a rebalance.
3. Verify session.timeout.ms and heartbeat.interval.ms (classic protocol): a heartbeat interval too close to the session timeout lets transient network delays expire the session.
4. Confirm rebalance-latency-avg is not climbing, which would indicate the rebalance itself is slow (large group, expensive partition assignment).

Recovery:

• Raise max.poll.interval.ms to cover the maximum observed processing time, or lower max.poll.records to process smaller batches more frequently when processing time is unpredictable.
• Adjust session.timeout.ms and heartbeat.interval.ms for more network-jitter tolerance; 30 seconds with a 10-second heartbeat is a reasonable classic-protocol starting point. Consider migrating to the KIP-848 consumer protocol, which removes client-driven session timeouts.
• If slow or failing commits drive the churn, investigate coordinator load and consider increasing the __consumer_offsets partition count or reducing commit frequency.

Runbook 3: Data Loss After Consumer Restart

Symptoms: After a restart, records that should have been processed are missing from the output, indicating they were skipped rather than reprocessed.

Diagnosis:

1. Compare the group’s committed offsets (kafka-consumer-groups --describe) against log-end offsets to see whether the consumer skipped ahead.
2. Determine whether auto-commit was in use: it can advance offsets past records that were not fully processed, so a restart skips them.
3. Check auto.offset.reset: if set to latest and the committed offsets were lost (e.g., offset expiration), the consumer starts at the end of the topic and skips all existing data.

Recovery:

• If the loss is recent and the records are still within retention, reset the group to a point before the loss with --to-datetime to reprocess them.
• To prevent recurrence, switch to manual commits and commit only after processing completes. If auto-commit is unavoidable, set auto.offset.reset=earliest and make processing idempotent to absorb the duplicates a from-beginning restart produces.
• Review offsets.retention.minutes (default 10080, i.e. 7 days): a group inactive longer than this loses its committed offsets. For groups intentionally paused for long periods, raise this retention or keep offsets in an external store.

Conclusion

Offset management is the linchpin of reliable stream processing on Apache Kafka. Your choices—automatic versus manual commits, synchronous versus asynchronous acknowledgment, transactional boundaries versus idempotent consumers—determine how the system behaves under failure and how quickly you recover. There is no single right answer; it depends on your processing semantics, your tolerance for duplication and loss, and your team’s operational maturity.

Resetting offsets, monitoring lag, and tuning commit behavior are ongoing practices, not one-time setup. As deployments scale, coordinator load and __consumer_offsets performance become real bottlenecks that need proactive management. Treat offset management as a first-class operational concern and your pipelines will deliver reliably.

These concerns are intertwined with the rest of the platform. Delivery semantics emerge from both producer configuration and consumer offset management, as explored in Producers, Consumers & Delivery Semantics. Rebalance protocols govern group membership and directly shape commit behavior, and when incidents strike, precise offset resets via the CLI are a skill every operator should have. Build a working understanding of these connected systems and you can operate Kafka with confidence.