[jira] [Resolved] (KAFKA-9982) [kafka-connect] Source connector does not guarantee at least once delivery

["Chris Egerton (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/KAFKA-9982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Chris Egerton resolved KAFKA-9982.
----------------------------------
    Resolution: Not A Bug

> [kafka-connect] Source connector does not guarantee at least once delivery
> --------------------------------------------------------------------------
>
>                 Key: KAFKA-9982
>                 URL: https://issues.apache.org/jira/browse/KAFKA-9982
>             Project: Kafka
>          Issue Type: Bug
>          Components: KafkaConnect
>    Affects Versions: 2.5.0
>            Reporter: Qinghui Xu
>            Priority: Major
>
> In kafka-connect runtime, the WorkerSourceTask is responsible for sending records to the destination topics and managing the source offset commit. Committed offsets are then used later for recovery of tasks during rebalance or restart.
> But there are two concerns when looking into the WorkerSourceTask implementation:
>  * When producer fail to send records, there's no retry but just skipping offset commit and then execute next loop (poll for new records)
>  * The offset commit and effectively sending records over network are in fact asynchronous, which means the offset commit could happen before records are received by brokers, and a rebalance/restart in this gap could lead to message loss.
> The conclusion is thus that the source connector does not support at least once semantics by default (without the plugin implementation making extra effort itself). I consider this as a bug.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)