[jira] [Created] (NIFI-4133) PublishKafkaRecord_0_10 should allow publishing all messages from a flow file to the same partition

["Bryan Bende (JIRA)" <ji...@apache.org>]

Bryan Bende created NIFI-4133:
---------------------------------

             Summary: PublishKafkaRecord_0_10 should allow publishing all messages from a flow file to the same partition
                 Key: NIFI-4133
                 URL: https://issues.apache.org/jira/browse/NIFI-4133
             Project: Apache NiFi
          Issue Type: Improvement
    Affects Versions: 1.3.0, 1.2.0
            Reporter: Bryan Bende
            Priority: Minor


In some use cases it is required to publish all of the messages from a given flow file to the same partition so that they can later be consumer in the same order. 

Currently the processor provides an option to choose between the default partitioner and a round-robin partitioner, and also allows specifying the name of a field in each record to use as a message key.

The default partitioner has the following behavior:
1)  If a partition is specified in the record, use it
 2) If no partition is specified but a key is present choose a partition based on a hash of the key
 3) If no partition or key is present choose a partition in a round-robin fashion

Currently we never pass in a partition to the Kafka record that is created, so we always fall into #2 or #3, and the message key is really meant to be unique per-event so we shouldn't be relying on every message using the same message key.

We should add an option to the processor like "Partition per FlowFile" which can be used with the default partitioner, and the NiFi side will pass in the same partition for each message created from the same flow file.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)