[jira] [Commented] (NIFI-4133) PublishKafkaRecord_0_10 should allow publishing all messages from a flow file to the same partition

["Sandish Kumar HN (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/NIFI-4133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16824746#comment-16824746 ] 

Sandish Kumar HN commented on NIFI-4133:
----------------------------------------

[~bende] Please correct me If I'm wrong here.
We should have partition as an option for users to enter Kafka topic partition number? so all the messages would go to the same partition? and add EL support so that based FlowFlow attributed (set kafka.partition = 1), partition number would be decided and used in PublicKafka Processor partition option? do we need to add this feature for all versions of Kafka? 

> PublishKafkaRecord_0_10 should allow publishing all messages from a flow file to the same partition
> ---------------------------------------------------------------------------------------------------
>
>                 Key: NIFI-4133
>                 URL: https://issues.apache.org/jira/browse/NIFI-4133
>             Project: Apache NiFi
>          Issue Type: Improvement
>    Affects Versions: 1.2.0, 1.3.0
>            Reporter: Bryan Bende
>            Assignee: Sandish Kumar HN
>            Priority: Minor
>
> In some use cases it is required to publish all of the messages from a given flow file to the same partition so that they can later be consumer in the same order. 
> Currently the processor provides an option to choose between the default partitioner and a round-robin partitioner, and also allows specifying the name of a field in each record to use as a message key.
> The default partitioner has the following behavior:
> 1)  If a partition is specified in the record, use it
>  2) If no partition is specified but a key is present choose a partition based on a hash of the key
>  3) If no partition or key is present choose a partition in a round-robin fashion
> Currently we never pass in a partition to the Kafka record that is created, so we always fall into #2 or #3, and the message key is really meant to be unique per-event so we shouldn't be relying on every message using the same message key.
> We should add an option to the processor like "Partition per FlowFile" which can be used with the default partitioner, and the NiFi side will pass in the same partition for each message created from the same flow file.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)