[jira] [Created] (KAFKA-3820) Provide utilities for tracking source offsets

["Ewen Cheslack-Postava (JIRA)" <ji...@apache.org>]

Ewen Cheslack-Postava created KAFKA-3820:
--------------------------------------------

             Summary: Provide utilities for tracking source offsets
                 Key: KAFKA-3820
                 URL: https://issues.apache.org/jira/browse/KAFKA-3820
             Project: Kafka
          Issue Type: New Feature
          Components: KafkaConnect
            Reporter: Ewen Cheslack-Postava
            Assignee: Ewen Cheslack-Postava


OffsetStorageReader does not (and is not expected to) be immediately updated when a SourceRecord is returned from poll(). However, this can be a bit confusing to connector developers as they may return that data, then expect a subsequent read from OffsetStorageReader should match that. In other words, rather than tracking which offset they are at themselves in variables maintained by the task implementation, the connector developer expected OffsetStorageReader to do this for them.

Part of the confusion comes from the fact that data is sent asynchronously after returned from poll(), which explains the semantics we have. However, it does also mean many connectors have similarly structured code where they keep track of the current offset themselves. It might be nice to provide some utilities, probably through the Context object, to get the last returned offset for each source partition being processed by a task.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)