[jira] [Commented] (KAFKA-13472) Connect can lose track of last committed offsets for topic partitions after partial consumer revocation

["Chris Egerton (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-13472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17448053#comment-17448053 ] 

Chris Egerton commented on KAFKA-13472:
---------------------------------------

Approved as a 3.1 blocker on the [3.1 release thread|https://mail-archives.apache.org/mod_mbox/kafka-dev/202111.mbox/%3CCAHn4u3tPba%2BOQuyawfg%2BmrfyAMnZYFBOrmzruZNGsdJy%2BrBJXg%40mail.gmail.com%3E].

> Connect can lose track of last committed offsets for topic partitions after partial consumer revocation
> -------------------------------------------------------------------------------------------------------
>
>                 Key: KAFKA-13472
>                 URL: https://issues.apache.org/jira/browse/KAFKA-13472
>             Project: Kafka
>          Issue Type: Bug
>          Components: KafkaConnect
>    Affects Versions: 3.1.0, 3.0.1
>            Reporter: Chris Egerton
>            Assignee: Chris Egerton
>            Priority: Blocker
>             Fix For: 3.1.0, 3.0.1
>
>
> The Connect framework tracks the last successfully-committed offsets for each topic partition that is currently assigned to the consumer of each sink task. If a sink task throws an exception from {{{}SinkTask::preCommit{}}}, the consumer is "rewound" by seeking to those last successfully-committed offsets for each topic partition, so that the same records can be redelivered to the task again.
> With the changes from KAFKA-12487, we failed to correctly update the logic for tracking these last-committed offsets which can cause topic partitions to be missing from them after partial revocation of topic partitions from the consumer. Specifically, we make the assumption that, whenever an offset commit succeeds, the offsets that were successfully committed constitute the entirely of the last-committed offsets for the task; when a partial consumer revocation takes place, we only commit offsets for some of the topic partitions assigned to the task's producer, and this assumption fails.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)