[jira] [Comment Edited] (FLINK-6288) FlinkKafkaProducer's custom Partitioner is always invoked with number of partitions of default topic

["Fang Yong (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-6288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15980636#comment-15980636 ] 

Fang Yong edited comment on FLINK-6288 at 4/24/17 2:18 AM:
-----------------------------------------------------------

    Thanks for providing the mailing list for this issue, I think this is very helpful for me. As discussed in the mailing list, I'm agree to add a extra API to support the partitioner for each topic. I think a field named Map<String, PartitionerInfo> topicPartitionerMap should be added in FlinkKafkaProducerBase, where the PartitionerInfo projo contains fields such as topic/partitions. 

    User can use an extra api naned addTopicPartitioner in FlinkKafkaProducerBase to add his special topic and partitioner, all the topic and partitioner in the topicPartitionerMap will be intialed in the open of FlinkKafkaProducerBase. When new data arrives to te sink and the target topic is in the topicPartitionerMap, the topic's special Partitioner whill be used to slice the data.


was (Author: zjureel):
Thanks for providing the mailing list for this issue, I think this is very helpful for me. As discussed in the mailing list, I'm agree to add a extra API to support the partitioner for each topic. I think a field named Map<String, PartitionerInfo> topicPartitionerMap should be added in FlinkKafkaProducerBase, where the PartitionerInfo projo contains fields such as topic/partitions. 
User can use an extra api naned addTopicPartitioner in FlinkKafkaProducerBase to add his special topic and partitioner, all the topic and partitioner in the topicPartitionerMap will be intialed in the open of FlinkKafkaProducerBase. When new data arrives to te sink and the target topic is in the topicPartitionerMap, the topic's special Partitioner whill be used to slice the data.

> FlinkKafkaProducer's custom Partitioner is always invoked with number of partitions of default topic
> ----------------------------------------------------------------------------------------------------
>
>                 Key: FLINK-6288
>                 URL: https://issues.apache.org/jira/browse/FLINK-6288
>             Project: Flink
>          Issue Type: Improvement
>          Components: Kafka Connector
>            Reporter: Tzu-Li (Gordon) Tai
>            Assignee: Fang Yong
>
> The {{FlinkKafkaProducerBase}} supports routing records to topics besides the default topic, but the custom {{Partitioner}} interface does not follow this semantic.
> The partitioner is always invoked the {{partition}} method with the number of partitions in the default topic, and not the number of partitions of the current {{targetTopic}}.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)