You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by "Jun Yao (JIRA)" <ji...@apache.org> on 2016/11/13 21:45:58 UTC
[jira] [Updated] (KAFKA-4402) Kafka Producer's DefaultPartitioner
is actually not round robin as said in the code comments "If no partition
or key is present choose a partition in a round-robin fashion"
[ https://issues.apache.org/jira/browse/KAFKA-4402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jun Yao updated KAFKA-4402:
---------------------------
Description:
From this code comments, it is said that Kafka client Producer's DefaultPartitioner will do round robin if "no partition or key is present",
https://github.com/apache/kafka/blob/41e676d29587042994a72baa5000a8861a075c8c/clients/src/main/java/org/apache/kafka/clients/producer/internals/DefaultPartitioner.java#L34
from the code it looks trying to do round robin as well, as it maintained a counter and try to increase it every time and then will decide which partition to go to;
However the issue here is the counter is a global counter that is shared by all the topics, so it is actually not round robin per topic and sometimes caused unbalanced routing among different partitions.
Although we can pass a custom implementation of interface "org.apache.kafka.clients.producer.Partitioner", it might be still good to make the default implementation true round robin as comment.
> Kafka Producer's DefaultPartitioner is actually not round robin as said in the code comments "If no partition or key is present choose a partition in a round-robin fashion"
> ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: KAFKA-4402
> URL: https://issues.apache.org/jira/browse/KAFKA-4402
> Project: Kafka
> Issue Type: Improvement
> Reporter: Jun Yao
> Priority: Minor
>
> From this code comments, it is said that Kafka client Producer's DefaultPartitioner will do round robin if "no partition or key is present",
> https://github.com/apache/kafka/blob/41e676d29587042994a72baa5000a8861a075c8c/clients/src/main/java/org/apache/kafka/clients/producer/internals/DefaultPartitioner.java#L34
> from the code it looks trying to do round robin as well, as it maintained a counter and try to increase it every time and then will decide which partition to go to;
> However the issue here is the counter is a global counter that is shared by all the topics, so it is actually not round robin per topic and sometimes caused unbalanced routing among different partitions.
> Although we can pass a custom implementation of interface "org.apache.kafka.clients.producer.Partitioner", it might be still good to make the default implementation true round robin as comment.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)