You are viewing a plain text version of this content. The canonical link for it is here.

Posted to users@kafka.apache.org by Tobias Adamson <to...@stargazer.com.sg> on 2017/01/12 14:32:53 UTC

Consumer Balancing multi topic, single partition

Hi
We have a scenario where we have a lot of single partition topics for ordering purposes.
We then want to use multiple consumer processes listening to many topics. 
During testing it seems like one consumer process will always end up with all topics/partitions assigned to it and there is no real distribution going on.

Does the assignment process take into account the total number of partitions assigned to each consumer process or does it only
check assignment per topic?

Regards
Toby

Re: Consumer Balancing multi topic, single partition

Posted by Tobias Adamson <to...@stargazer.com.sg>.

Thanks, works well
For someone searching for this..
Example:

props.put(ConsumerConfig.PARTITION_ASSIGNMENT_STRATEGY_CONFIG, "org.apache.kafka.clients.consumer.RoundRobinAssignor");


> On 12 Jan 2017, at 11:37 PM, tao xiao <xi...@gmail.com> wrote:
> 
> The default partition assignor is range assignor which assigns works on a
> per-topic basis. If you topics with one partition only they will be
> assigned to the same consumer. You can change the assignor to
> org.apache.kafka.clients.consumer.RoundRobinAssignor
> 
> On Thu, 12 Jan 2017 at 22:33 Tobias Adamson <to...@stargazer.com.sg> wrote:
> 
> Hi
> We have a scenario where we have a lot of single partition topics for
> ordering purposes.
> We then want to use multiple consumer processes listening to many topics.
> During testing it seems like one consumer process will always end up with
> all topics/partitions assigned to it and there is no real distribution
> going on.
> 
> Does the assignment process take into account the total number of
> partitions assigned to each consumer process or does it only
> check assignment per topic?
> 
> Regards
> Toby

Re: Consumer Balancing multi topic, single partition

Posted by tao xiao <xi...@gmail.com>.

The default partition assignor is range assignor which assigns works on a
per-topic basis. If you topics with one partition only they will be
assigned to the same consumer. You can change the assignor to
org.apache.kafka.clients.consumer.RoundRobinAssignor

On Thu, 12 Jan 2017 at 22:33 Tobias Adamson <to...@stargazer.com.sg> wrote:

Hi
We have a scenario where we have a lot of single partition topics for
ordering purposes.
We then want to use multiple consumer processes listening to many topics.
During testing it seems like one consumer process will always end up with
all topics/partitions assigned to it and there is no real distribution
going on.

Does the assignment process take into account the total number of
partitions assigned to each consumer process or does it only
check assignment per topic?

Regards
Toby