[jira] [Commented] (NIFI-3108) PublishKafka Should Support Custom Partitioners

["Koji Kawamura (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/NIFI-3108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15838881#comment-15838881 ] 

Koji Kawamura commented on NIFI-3108:
-------------------------------------

I agree we should provide a way to support custom partitioning with PublishKafka processors. I think specifying custom Partitioner implementation by "Partitioner JARs" is very useful. In addition to that, I know some users are using 'Partition' property with PutKafka processor and Kafka 0.8, so it'd be great if we can support both "Partitioner JARs" and "Partition" properties.

For clarification, those properties should work as follows:

- Both "Partitioner Class" and "Partition" can be set, in this case, "Partitioner Class" is only used when "Partition" does NOT return a value.
- "Partition" : A literal partition ID, or NiFi Attribute EL can be used to specify partition ID directly.
- "Partitioner Class": A name of the class that implements 'org.apache.kafka.clients.producer.Partitioner', default is 'org.apache.kafka.clients.producer.internals.DefaultPartitioner'.
- "Partitioner JARs": Specify one or more JARs to add to the classpath of the processor.

I will work on this.

> PublishKafka Should Support Custom Partitioners
> -----------------------------------------------
>
>                 Key: NIFI-3108
>                 URL: https://issues.apache.org/jira/browse/NIFI-3108
>             Project: Apache NiFi
>          Issue Type: Improvement
>    Affects Versions: 1.0.0, 1.1.0
>            Reporter: Bryan Bende
>            Priority: Minor
>
> Currently PublishKafka/PublishKafka_0_10 have a property for choosing the partitioner which equates to setting the 'partitioner.class' property on the Kafka client, but the property on the processor only allows selecting from "Default Partitioner" and "Round-Robin" partitioner which are provided by the Kafka client.
> If someone wants to implement their own partitioner, there currently isn't an easy way to add it to NiFi's classpath, and there is no way to specify it in the processor.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)