[jira] [Commented] (KAFKA-18641) AsyncKafkaConsumer could lose records with auto offset commit

["TengYao Chi (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-18641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17916869#comment-17916869 ] 

TengYao Chi commented on KAFKA-18641:
-------------------------------------

Hi [~junrao] 
I will investigate this issue, feel free to reassign if you have another plan for this issue.

Thank you :)

> AsyncKafkaConsumer could lose records with auto offset commit
> -------------------------------------------------------------
>
>                 Key: KAFKA-18641
>                 URL: https://issues.apache.org/jira/browse/KAFKA-18641
>             Project: Kafka
>          Issue Type: Bug
>          Components: clients, consumer
>    Affects Versions: 4.0.0
>            Reporter: Jun Rao
>            Assignee: TengYao Chi
>            Priority: Blocker
>
> In the new AsyncKafkaConsumer, the application thread will keep updating the auto commit timer through PollEvent. In the consumer network thread, once the timer has expired, it generates an offset commit request with the current offset position in subscriptions. However, at this point, the records before that offset could just be polled from FetchBuffer, but not actually consumed by the application. If the application dies immediately, those records may never be consumed by the application since the offset could have been committed.
> The ClassicKafkaConsumer doesn't seem to have this problem. In each poll() call, before fetching new records, it first calls ConsumerCoordinator.poll(), which generates an OffsetCommitRequest with the current offset position in subscriptions. Since this is done in the same application thread, it guarantees that all records returned in the previous poll() have been processed. The problem exists in AsyncKafkaConsumer because the polling of new records and committing offsets are done in separate threads.
>  
> This problem exists for async offset commit and auto offset commit during rebalance in AsyncKafkaConsumer too.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)