You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2017/07/07 05:17:00 UTC
[jira] [Commented] (KAFKA-5567) With transformations that mutate
the topic-partition committing offsets should to refer to the original
topic-partition
[ https://issues.apache.org/jira/browse/KAFKA-5567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16077603#comment-16077603 ]
ASF GitHub Bot commented on KAFKA-5567:
---------------------------------------
GitHub user kkonstantine opened a pull request:
https://github.com/apache/kafka/pull/3499
KAFKA-5567: Connect sink worker should commit offsets of original topic partitions
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/kkonstantine/kafka KAFKA-5567-With-transformations-that-mutate-the-topic-partition-committing-offsets-should-to-refer-to-the-original-topic-partition
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/kafka/pull/3499.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3499
----
commit b9eeeb12812f0b237b4356717dfc5bf0e252e8e6
Author: Konstantine Karantasis <ko...@confluent.io>
Date: 2017-07-07T05:13:50Z
KAFKA-5567: Connect sink worker should commit offsets of original topic partitions.
----
> With transformations that mutate the topic-partition committing offsets should to refer to the original topic-partition
> -----------------------------------------------------------------------------------------------------------------------
>
> Key: KAFKA-5567
> URL: https://issues.apache.org/jira/browse/KAFKA-5567
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect
> Affects Versions: 0.10.2.0, 0.10.2.1, 0.11.0.0
> Reporter: Konstantine Karantasis
> Assignee: Konstantine Karantasis
> Fix For: 0.11.0.1
>
>
> When a chain of transformations (SMTs) that mutate a record's topic-partition is applied, then Connect is unable to map the transformed record to its original topic-partition. This affects committing offsets.
> Currently, in order to reproduce the issue one could use the {{TimestampRouter}} transformation with a sink connector such as the {{FileStreamSinkConnector}}.
> In this ticket we'll address the issue for connectors that don't manage/commit their offsets themselves. For the connectors that do such management, broader API changes are required to supply the connectors with the necessary information that will allow them to map a transformed record to the original.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)