You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "Kirk True (Jira)" <ji...@apache.org> on 2025/01/24 17:16:00 UTC

[jira] [Updated] (KAFKA-18641) AsyncKafkaConsumer could lose records with auto offset commit

     [ https://issues.apache.org/jira/browse/KAFKA-18641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Kirk True updated KAFKA-18641:
------------------------------
    Component/s: clients

> AsyncKafkaConsumer could lose records with auto offset commit
> -------------------------------------------------------------
>
>                 Key: KAFKA-18641
>                 URL: https://issues.apache.org/jira/browse/KAFKA-18641
>             Project: Kafka
>          Issue Type: Bug
>          Components: clients, consumer
>    Affects Versions: 4.0.0
>            Reporter: Jun Rao
>            Priority: Blocker
>
> In the new AsyncKafkaConsumer, the application thread will keep updating the auto commit timer through PollEvent. In the consumer network thread, once the timer has expired, it generates an offset commit request with the current offset position in subscriptions. However, at this point, the records before that offset could just be polled from FetchBuffer, but not actually consumed by the application. If the application dies immediately, those records may never be consumed by the application since the offset could have been committed.
> The ClassicKafkaConsumer doesn't seem to have this problem. In each poll() call, before fetching new records, it first calls ConsumerCoordinator.poll(), which generates an OffsetCommitRequest with the current offset position in subscriptions. Since this is done in the same application thread, it guarantees that all records returned in the previous poll() have been processed. The problem exists in AsyncKafkaConsumer because the polling of new records and committing offsets are done in separate threads.
>  
> This problem exists for async offset commit and auto offset commit during rebalance in AsyncKafkaConsumer too.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)