[jira] [Created] (KAFKA-9982) [kafka-connect] Source connector does not guarantee at least once delivery

["Qinghui Xu (Jira)" <ji...@apache.org>]

Qinghui Xu created KAFKA-9982:
---------------------------------

             Summary: [kafka-connect] Source connector does not guarantee at least once delivery
                 Key: KAFKA-9982
                 URL: https://issues.apache.org/jira/browse/KAFKA-9982
             Project: Kafka
          Issue Type: Bug
          Components: KafkaConnect
    Affects Versions: 2.5.0
            Reporter: Qinghui Xu


In kafka-connect runtime, the WorkerSourceTask is responsible for sending records to the destination topics and managing the source offset commit. Committed offsets are then used later for recovery of tasks during rebalance or restart.

But there are two concerns when looking into the WorkerSourceTask implementation:
 * When producer fail to send records, there's no retry but just skipping offset commit and then execute next loop (poll for new records)
 * The offset commit and effectively sending records over network are in fact asynchronous, which means the offset commit could happen before records are received by brokers, and a rebalance/restart in this gap could lead to message loss.

The conclusion is thus that the source connector does not support at least once semantics by default (without the plugin implementation making extra effort itself). I consider this as a bug.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)