[jira] [Commented] (KAFKA-3821) Allow Kafka Connect source tasks to produce offset without writing to topics

["Randall Hauch (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-3821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16476237#comment-16476237 ] 

Randall Hauch commented on KAFKA-3821:
--------------------------------------

Hi, Gunnar. Trying to get back to this to see if we can make progress. After thinking about this more, I feel like we need to avoid changing how Connect obtains records from a {{SourceTask.poll()}}. As mentioned earlier in the discussion, a simpler approach more closely aligned with the current API is to introduce subtypes of SourceRecord. For this change, a {{OffsetSourceRecord}} would work well, would fit into the existing design, and would allow a source task to very easily maintain the order of these records w/r/t other normal records. Further, this same approach would work really well with the transaction-related subtypes for KAFKA-6080.

It's a luxury to be able to change the API more to find a pattern that may fit better, but we prefer to keep API stable and backward compatible, and wherever possible keeping our existing abstractions. Is there something about this simpler approach that won't work? I do think that writing out a KIP (which really is the next step) will help highlight how much smaller a change the simpler approach is w/r/t the public API.

> Allow Kafka Connect source tasks to produce offset without writing to topics
> ----------------------------------------------------------------------------
>
>                 Key: KAFKA-3821
>                 URL: https://issues.apache.org/jira/browse/KAFKA-3821
>             Project: Kafka
>          Issue Type: Improvement
>          Components: KafkaConnect
>    Affects Versions: 0.9.0.1
>            Reporter: Randall Hauch
>            Priority: Major
>              Labels: needs-kip
>             Fix For: 2.0.0
>
>
> Provide a way for a {{SourceTask}} implementation to record a new offset for a given partition without necessarily writing a source record to a topic.
> Consider a connector task that uses the same offset when producing an unknown number of {{SourceRecord}} objects (e.g., it is taking a snapshot of a database). Once the task completes those records, the connector wants to update the offsets (e.g., the snapshot is complete) but has no more records to be written to a topic. With this change, the task could simply supply an updated offset.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)