[jira] [Commented] (KAFKA-3821) Allow Kafka Connect source tasks to produce offset without writing to topics

["Randall Hauch (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-3821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15323429#comment-15323429 ] 

Randall Hauch commented on KAFKA-3821:
--------------------------------------

Source tasks produce records by returning them from their {{poll()}} method, and the records in the resulting list are ordered, and the lists of records are ordered based upon the sequence of calls to {{poll()}}. Therefore, a straightforward way to update the offset explicitly while maintaining this order is to request it via inclusion of a special {{SourceRecord}} within the list returned by {{poll()}}. Some ideas include:

* a {{SourceRecord}} object with a null (or special) topic, or a null key and value; or
* a {{SourceRecord}} object with a null key and value; or
* a subclass of {{SourceRecord}} that denotes this particular kind of request.

Neither is clean or elegant. The first two are very implicit, while the second requires more processing overhead and logic for every {{SourceRecord}} (though there are ways around this that may be acceptable).

Any other thoughts?

> Allow Kafka Connect source tasks to produce offset without writing to topics
> ----------------------------------------------------------------------------
>
>                 Key: KAFKA-3821
>                 URL: https://issues.apache.org/jira/browse/KAFKA-3821
>             Project: Kafka
>          Issue Type: Improvement
>          Components: KafkaConnect
>    Affects Versions: 0.9.0.1
>            Reporter: Randall Hauch
>            Assignee: Ewen Cheslack-Postava
>
> Provide a way for a {{SourceTask}} implementation to record a new offset for a given partition without necessarily writing a source record to a topic.
> Consider a connector task that uses the same offset when producing an unknown number of {{SourceRecord}} objects (e.g., it is taking a snapshot of a database). Once the task completes those records, the connector wants to update the offsets (e.g., the snapshot is complete) but has no more records to be written to a topic. With this change, the task could simply supply an updated offset.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)