Consumers are not load balanced against partitions

[Gomathivinayagam Muthuvinayagam <sa...@gmail.com>]

I have started multiple consumers with some time delay. Even after long
period of time, later joined consumers are not getting any distribution of
partitions. Only one consumer is loaded with all the partitions. I dont see
any configuration parameter to change this behavior.

Did anyone face similar problem? I started all the consumers with same
number of consumer threads.  Note that all the consumers are started within
the same host. Also I see consumer ids, on zookeeper.

If you guys can share any to fix this problem that will be great.

Thanks & Regards,

Re: Consumers are not load balanced against partitions

[Jiangjie Qin <jq...@linkedin.com.INVALID>]

The default range partition assignment algorithm will assign partition on
per topic basis. If you have more consumer threads than number of
partitions in a topic, some threads will not be assigned any partition.
If you are consuming from multiple topics, You might want to set the
partition.assignment.strategy to ³roundrobin². This will assign partitions
based on all the topics the consumer group is consuming from. But remember
there are two restrictions for this algorithm:
1. All consumers in the same consumer group must have same number of
consumer threads.
2. All consumers in the same consumer group must consume from the same set
of topics.

Thanks.

Jiangjie (Becket) Qin

On 5/14/15, 3:28 PM, "Gomathivinayagam Muthuvinayagam"
<sa...@gmail.com> wrote:

>I have started multiple consumers with some time delay. Even after long
>period of time, later joined consumers are not getting any distribution of
>partitions. Only one consumer is loaded with all the partitions. I dont
>see
>any configuration parameter to change this behavior.
>
>Did anyone face similar problem? I started all the consumers with same
>number of consumer threads.  Note that all the consumers are started
>within
>the same host. Also I see consumer ids, on zookeeper.
>
>If you guys can share any to fix this problem that will be great.
>
>Thanks & Regards,