[jira] [Created] (KAFKA-17480) New consumer commit all consumed should retrieve offsets in background thread

["Lianet Magrans (Jira)" <ji...@apache.org>]

Lianet Magrans created KAFKA-17480:
--------------------------------------

             Summary: New consumer commit all consumed should retrieve offsets in background thread
                 Key: KAFKA-17480
                 URL: https://issues.apache.org/jira/browse/KAFKA-17480
             Project: Kafka
          Issue Type: Bug
          Components: clients, consumer
            Reporter: Lianet Magrans


When committing all consumed offsets (sync, async, or on close), the new consumer retrieves the offsets from subscriptionState.allConsumed() in the app thread (and generates events containing those offsets, that are sent to the background thread to be processed). We should review this to consider retrieving the allConsumed in the background when processing the events, to avoid inconsistencies given that the subscription state could be modified in the background thread since the moment the allConsumed was retrieved in the app thread.

These are the places in the app thread that currently retrieve the allConsumed to pass it into the events. 
 * [https://github.com/apache/kafka/blob/59f5d91d8f1412c50c74a776a624879101402284/clients/src/main/java/org/apache/kafka/clients/consumer/internals/AsyncKafkaConsumer.java#L735]
 * [https://github.com/apache/kafka/blob/59f5d91d8f1412c50c74a776a624879101402284/clients/src/main/java/org/apache/kafka/clients/consumer/internals/AsyncKafkaConsumer.java#L1299]
 * [https://github.com/apache/kafka/blob/59f5d91d8f1412c50c74a776a624879101402284/clients/src/main/java/org/apache/kafka/clients/consumer/internals/AsyncKafkaConsumer.java#L1323]

We could consider to keep the same CommitSync and CommitAsync events, but with the option of not taking offsets as input (maybe a flag for commitAllConsumed). We could also create new events for the allConsumed alternatives (I lean towards keeping just CommitSync and CommitAsync but open to whatever seems clearer)



--
This message was sent by Atlassian Jira
(v8.20.10#820010)