You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "Abdelhamide EL ARIB (JIRA)" <ji...@apache.org> on 2018/07/20 13:07:00 UTC
[jira] [Updated] (KAFKA-7189) Add a Merge Transformer for Kafka
Connect
[ https://issues.apache.org/jira/browse/KAFKA-7189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Abdelhamide EL ARIB updated KAFKA-7189:
---------------------------------------
Description:
Like the [flatten|https://docs.confluent.io/current/connect/transforms/flatten.html#flatten], there is the need too for a merge transformer.
Example transformation :
We want to add the offset and the partition for each record, and after that merge them into one field _metadata :
{code:java}
"transforms":"AddOffset, AddPartition, MergeFields",
"transforms.AddOffset.type":"org.apache.kafka.connect.transforms.InsertField$Value",
"transforms.AddOffset.offset.field":"offset!",
"transforms.AddPartition.type":"org.apache.kafka.connect.transforms.InsertField$Value",
"transforms.AddPartition.partition.field":"partition!",
"transforms.MergeFields.type":"org.apache.kafka.connect.transforms.Merge$Value",
"transforms.MergeFields.field.list":"offset,partition",
"transforms.MergeFields.field.root":"_metadata"
{code}
This is our MR for this transformer
was:
Like the [flatten|[https://docs.confluent.io/current/connect/transforms/flatten.html#flatten],] there is the need too for a merge transformer.
Example transformation :
We want to add the offset and the partition for each record, and after that merge them into one field _metadata :
{code:java}
"transforms":"AddOffset, AddPartition, MergeFields",
"transforms.AddOffset.type":"org.apache.kafka.connect.transforms.InsertField$Value",
"transforms.AddOffset.offset.field":"offset!",
"transforms.AddPartition.type":"org.apache.kafka.connect.transforms.InsertField$Value",
"transforms.AddPartition.partition.field":"partition!",
"transforms.MergeFields.type":"org.apache.kafka.connect.transforms.Merge$Value",
"transforms.MergeFields.field.list":"offset,partition",
"transforms.MergeFields.field.root":"_metadata"
{code}
This is our MR for this transformer
> Add a Merge Transformer for Kafka Connect
> -----------------------------------------
>
> Key: KAFKA-7189
> URL: https://issues.apache.org/jira/browse/KAFKA-7189
> Project: Kafka
> Issue Type: Improvement
> Components: KafkaConnect
> Reporter: Abdelhamide EL ARIB
> Priority: Minor
>
> Like the [flatten|https://docs.confluent.io/current/connect/transforms/flatten.html#flatten], there is the need too for a merge transformer.
> Example transformation :
> We want to add the offset and the partition for each record, and after that merge them into one field _metadata :
>
> {code:java}
> "transforms":"AddOffset, AddPartition, MergeFields",
> "transforms.AddOffset.type":"org.apache.kafka.connect.transforms.InsertField$Value",
> "transforms.AddOffset.offset.field":"offset!",
> "transforms.AddPartition.type":"org.apache.kafka.connect.transforms.InsertField$Value",
> "transforms.AddPartition.partition.field":"partition!",
> "transforms.MergeFields.type":"org.apache.kafka.connect.transforms.Merge$Value",
> "transforms.MergeFields.field.list":"offset,partition",
> "transforms.MergeFields.field.root":"_metadata"
> {code}
>
> This is our MR for this transformer
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)