[jira] [Comment Edited] (KAFKA-3821) Allow Kafka Connect source tasks to produce offset without writing to topics

["Randall Hauch (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-3821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15973506#comment-15973506 ] 

Randall Hauch edited comment on KAFKA-3821 at 4/18/17 9:17 PM:
---------------------------------------------------------------

[~criccomini] logged KAFKA-5084, but that is really another situation where the connector would like to provide new offsets without having to produce {{SourceRecord}}. As he says [here|https://confluentcommunity.slack.com/archives/C49L0V3L7/p1492547735471505], the connector sometimes need to tell Kafka Connect:

{quote}
i made some progress in my source, but i don’t want to send any of the data
{quote}

This becomes an issue when stopping and restarting the connector. In cases when it's been a long time since the connector produced source records with an updated offset, the connector upon startup will receive this old (but still the latest) offset. But the source has long purged that portion of the log, so the connector is unable to start up. In reality, the connector had processed much farther than was described by the last offset, and if it had been able to submit that offset to Kafka Connect, upon restart it would have been able to successfully start at a point in the log that still exists in the source.


was (Author: rhauch):
[~criccomini] logged KAFKA-5084, but that is really another situation where the connector would like to provide new offsets without having to produce {{SourceRecord}}. As he says [here|https://confluentcommunity.slack.com/archives/C49L0V3L7/p1492547735471505], the connector sometimes need to tell Kafka Connect:

{quote}
i made some progress in my source, but i don’t want to send any of the data
{quote}

> Allow Kafka Connect source tasks to produce offset without writing to topics
> ----------------------------------------------------------------------------
>
>                 Key: KAFKA-3821
>                 URL: https://issues.apache.org/jira/browse/KAFKA-3821
>             Project: Kafka
>          Issue Type: Improvement
>          Components: KafkaConnect
>    Affects Versions: 0.9.0.1
>            Reporter: Randall Hauch
>              Labels: needs-kip
>
> Provide a way for a {{SourceTask}} implementation to record a new offset for a given partition without necessarily writing a source record to a topic.
> Consider a connector task that uses the same offset when producing an unknown number of {{SourceRecord}} objects (e.g., it is taking a snapshot of a database). Once the task completes those records, the connector wants to update the offsets (e.g., the snapshot is complete) but has no more records to be written to a topic. With this change, the task could simply supply an updated offset.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)