[jira] [Created] (NIFI-7953) Allow ConsumeKafka_2_* / ConsumeKafkaRecord_2_* to separate Records based on Message Key

["Mark Payne (Jira)" <ji...@apache.org>]

Mark Payne created NIFI-7953:
--------------------------------

             Summary: Allow ConsumeKafka_2_* / ConsumeKafkaRecord_2_* to separate Records based on Message Key
                 Key: NIFI-7953
                 URL: https://issues.apache.org/jira/browse/NIFI-7953
             Project: Apache NiFi
          Issue Type: Improvement
          Components: Extensions
            Reporter: Mark Payne
            Assignee: Mark Payne


The Kafka consumer processors ignore the Message Key when using a demarcator or when using the Record variant, because adding a FlowFile attribute for the message key doesn't make sense, since the key is likely different per record.

However, a fairly common use case is to use the Key only for partitioning and many Kafka Messages may exist for the same key. For example, this is commonly used for CDC type of use cases where the key may just be a table name.

For such cases, it would be helpful to allow the consumer processors to avoid adding two messages to the same FlowFile if their keys are different. I.e., all records in a given FlowFile will have the same key. The key can then be added as a FlowFile attribute.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)