[jira] [Commented] (KAFKA-3209) Support single message transforms in Kafka Connect

["Michael Graff (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-3209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15268959#comment-15268959 ] 

Michael Graff commented on KAFKA-3209:
--------------------------------------

We have a use case which is in effect "take a single Kafka input message and produce 1 or more output items" -- I understand people refer to this as a "flatMap" or something like that.

Specifically, our use case is to get a chunk formatted like this:

Header-1: value1
Header-2: value2

record1
record2
record3
...
records

Our transformation is to take some header fields and merge them into a record, then emit that record for all (or some) records in the message.


> Support single message transforms in Kafka Connect
> --------------------------------------------------
>
>                 Key: KAFKA-3209
>                 URL: https://issues.apache.org/jira/browse/KAFKA-3209
>             Project: Kafka
>          Issue Type: Improvement
>          Components: KafkaConnect
>            Reporter: Neha Narkhede
>
> Users should be able to perform light transformations on messages between a connector and Kafka. This is needed because some transformations must be performed before the data hits Kafka (e.g. filtering certain types of events or PII filtering). It's also useful for very light, single-message modifications that are easier to perform inline with the data import/export.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)