[jira] [Created] (KAFKA-14727) Connect EOS mode should periodically call task commit

["Greg Harris (Jira)" <ji...@apache.org>]

Greg Harris created KAFKA-14727:
-----------------------------------

             Summary: Connect EOS mode should periodically call task commit
                 Key: KAFKA-14727
                 URL: https://issues.apache.org/jira/browse/KAFKA-14727
             Project: Kafka
          Issue Type: Bug
          Components: KafkaConnect
    Affects Versions: 3.4.0
            Reporter: Greg Harris
            Assignee: Greg Harris


In non-EOS mode, there is a background thread which periodically commits offsets for a task. If this thread does not have resources to flush on the framework side (records, or offsets) it still calls the task's commit() method to update the internal state of the task.

In EOS mode, there is no background thread, and all offset commits are performed on the main task thread in response to sending records to Kafka. This has the effect of only triggering the task's commit() method when there are records to send to Kafka, which is different than non-EOS mode.

In order to bring the two modes into better alignment, and allow tasks reliant on the non-EOS empty commit() behavior to work in EOS mode out-of-the-box, EOS mode should provide offset commits periodically for tasks which do not produce records.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)