You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by su...@gmail.com, su...@gmail.com on 2019/06/13 06:35:57 UTC

Consumer rebalance not triggered

I have 100 Kafka topics with one partition each.
I have started 10 instance of consumer and they use the same consumer group id and i'm using subscribe by pattern.
I noticed that all the topic partition is assigned to the same consumer and all other instances starve.
I expected that rebalance would be triggered and all instance would get some topic partition.
Can someone confirm if this is expected behavior.

Thanks,
Sunil

Re: Consumer rebalance not triggered

Posted by Harper Henn <ha...@datto.com>.
I just realized you wrote that your topics have only one partition each. My
response is not relevant, but the replies you received in a similar email
thread were.

On Sun, Jun 16, 2019, 1:46 PM Harper Henn <ha...@datto.com> wrote:

> Is there anything in Kafka server logs for your consumer group/group
> coordinator when new members join the group?
>
> I would expect partitions to be divided among members of the same consumer
> group.
>
> Does each consumer in the group have a client ID? This appears to be
> optional from what I've read, but just wondering.
>
> On Thu, Jun 13, 2019, 7:53 AM sunilverm@gmail.com <su...@gmail.com>
> wrote:
>
>> I have 100 Kafka topics with one partition each.
>> I have started 10 instance of consumer and they use the same consumer
>> group id and i'm using subscribe by pattern.
>> I noticed that all the topic partition is assigned to the same consumer
>> and all other instances starve.
>> I expected that rebalance would be triggered and all instance would get
>> some topic partition.
>> Can someone confirm if this is expected behavior.
>>
>> Thanks,
>> Sunil
>>
>>

Re: Consumer rebalance not triggered

Posted by Harper Henn <ha...@datto.com>.
Is there anything in Kafka server logs for your consumer group/group
coordinator when new members join the group?

I would expect partitions to be divided among members of the same consumer
group.

Does each consumer in the group have a client ID? This appears to be
optional from what I've read, but just wondering.

On Thu, Jun 13, 2019, 7:53 AM sunilverm@gmail.com <su...@gmail.com>
wrote:

> I have 100 Kafka topics with one partition each.
> I have started 10 instance of consumer and they use the same consumer
> group id and i'm using subscribe by pattern.
> I noticed that all the topic partition is assigned to the same consumer
> and all other instances starve.
> I expected that rebalance would be triggered and all instance would get
> some topic partition.
> Can someone confirm if this is expected behavior.
>
> Thanks,
> Sunil
>
>