[jira] [Created] (KAFKA-21249) Pre-rebalance auto-commit retries may commit offsets beyond unprocessed records

["sanghyeok An (Jira)" <ji...@apache.org>]

sanghyeok An created KAFKA-21249:
------------------------------------

             Summary: Pre-rebalance auto-commit retries may commit offsets beyond unprocessed records
                 Key: KAFKA-21249
                 URL: https://issues.apache.org/jira/browse/KAFKA-21249
             Project: Kafka
          Issue Type: Bug
            Reporter: sanghyeok An
            Assignee: sanghyeok An


When using the Java {{AsyncKafkaConsumer}} with {{group.protocol=consumer}} and {{{}enable.auto.commit=true{}}}, a pre-rebalance auto-commit retry may commit an offset beyond records that the application has not yet processed.

When{{ CommitRequestManager.autoCommitSyncBeforeRebalanceWithRetries()}} receives a retriable error, it calls {{subscriptions.allConsumed()}} again and replaces the existing request’s offset map. These offsets reflect consumer positions advanced while collecting records for {{{}poll(){}}}, rather than the application’s processing progress.

During a rebalance, record collection is blocked for partitions pending revocation, but records can still be returned from retained partitions. Although the revocation and assignment sequence waits for the auto-commit to complete, the application thread does not wait for the entire commit retry sequence. Consequently, the following sequence is possible:
 # After the application finishes processing records through offset 99 in partition P, a pre-rebalance auto-commit starts with offset 100.
 # While the initial commit request is pending, {{poll()}} returns records 100–149 from retained partition P. The internal position advances to 150, and the application starts processing the batch.
 # The initial commit request fails with a retriable error.
 # The retry logic reads the current position again, changing the offset to commit from 100 to 150.
 # If offset 150 is committed and the process exits before processing the entire batch, the restarted consumer may resume from 150 and skip the remaining unprocessed records.

This can occur even when a single application thread follows the sequence {{{}poll → process all returned records → next poll{}}}. It does not require the periodic auto-commit interval to expire.

An auto-commit retry should not commit beyond the application’s processing progress. However, this retry path can include positions advanced by records returned after the initial request was created. This may cause records to remain unprocessed by the consumer group after a failure.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)