You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by "Chris Egerton (Jira)" <ji...@apache.org> on 2023/07/21 16:07:00 UTC

[jira] [Resolved] (KAFKA-13431) Sink Connectors: Support topic-mutating SMTs for async connectors (preCommit users)

     [ https://issues.apache.org/jira/browse/KAFKA-13431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Chris Egerton resolved KAFKA-13431.
-----------------------------------
    Fix Version/s: 3.6.0
       Resolution: Done

> Sink Connectors: Support topic-mutating SMTs for async connectors (preCommit users)
> -----------------------------------------------------------------------------------
>
>                 Key: KAFKA-13431
>                 URL: https://issues.apache.org/jira/browse/KAFKA-13431
>             Project: Kafka
>          Issue Type: Improvement
>          Components: KafkaConnect
>            Reporter: Diego Erdody
>            Assignee: Yash Mayya
>            Priority: Major
>              Labels: needs-kip
>             Fix For: 3.6.0
>
>
> There's currently an incompatibility between Sink connectors overriding the {{SinkTask.preCommit}} method (for asynchronous processing) and SMTs that mutate the topic field.
> The problem was present since the {{preCommit}} method inception and is rooted in a mismatch between the topic/partition that is passed to {{open/preCommit}} (the original topic and partition before applying any transformations) and the topic partition that is present in the SinkRecord that the {{SinkTask.put}} method receives (after transformations are applied). Since that's all the information the connector has to implement any kind of internal offset tracking, the topic/partitions it can return in preCommit will correspond to the transformed topic, when the framework actually expects it to be the original topic.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)