You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by on 2019/06/13 06:27:21 UTC
All topic are assigned to same consumer in a consumer group
I have 100 topics with one partition each.
I have 10 instance of consumer with the Same consumer group id and I'm using subscribe using pattern.
I observe that all the topic partitions are assigned to the same consumer and others are starving. It is not distributed.
I was expecting that rebalance would kick-in and the different topic partitions would be assigned to different consumer instances.
Can anyone confirm if this is expected behavior.
Thanks,
Sunil
Re: All topic are assigned to same consumer in a consumer group
Posted by "Matthias J. Sax" <ma...@confluent.io>.
What `PartitionAssignor` did you configure?
Are you by any change using Kafka Streams? For this case, it's expected,
because Kafka Streams co-partitions during the assignment.
-Matthias
On 6/12/19 11:27 PM, sunilverm@gmail.com wrote:
> I have 100 topics with one partition each.
> I have 10 instance of consumer with the Same consumer group id and I'm using subscribe using pattern.
> I observe that all the topic partitions are assigned to the same consumer and others are starving. It is not distributed.
> I was expecting that rebalance would kick-in and the different topic partitions would be assigned to different consumer instances.
> Can anyone confirm if this is expected behavior.
>
> Thanks,
> Sunil
>
Re: All topic are assigned to same consumer in a consumer group
Posted by Babu Turlapati <ba...@confluent.io>.
Set partition.assignment.strategy to Round Robin, it will resolve your
problem.
On Thu, Jun 13, 2019 at 11:10 AM Pavel Molchanov <
pavel.molchanov@infodesk.com> wrote:
> One consumer reads from one or more partitions. If you have 1 partition per
> topic, only one consumer in the consumer group will read from that
> partition, others will be in standby.
>
> To process data in parallel in the consumer group, you need to increase the
> number of partitions per topic. For example, 2 partitions will allow 2
> consumers to work in parallel, others will be standby.
>
>
> *Pavel Molchanov*
> Director of Software Development
> InfoDesk
> www.infodesk.com
>
> 1 Bridge Street | Suite 105 | Irvington | New York | 10551
> <
> https://maps.google.com/?q=660+White+Plains+Road+%7C+Suite+300+%7C+Tarrytown+%7C+New+York+%7C+10591&entry=gmail&source=g
> >|
> Office: +1 (914) 332-5940
> Change Privacy Settings
> <https://www.infodesk.com/unsubscription-options> | Contact
> Privacy Team <un...@infodesk.com> | Privacy Policy
> <https://www.infodesk.com/privacy-policy>
>
> This e-mail message may contain confidential or legally privileged
> information and is intended only for the use of the intended recipient(s).
> Any unauthorized disclosure, dissemination, distribution, copying or the
> taking of any action in reliance on the information herein is prohibited.
>
>
> On Thu, Jun 13, 2019 at 10:53 AM sunilverm@gmail.com <su...@gmail.com>
> wrote:
>
> > I have 100 topics with one partition each.
> > I have 10 instance of consumer with the Same consumer group id and I'm
> > using subscribe using pattern.
> > I observe that all the topic partitions are assigned to the same consumer
> > and others are starving. It is not distributed.
> > I was expecting that rebalance would kick-in and the different topic
> > partitions would be assigned to different consumer instances.
> > Can anyone confirm if this is expected behavior.
> >
> > Thanks,
> > Sunil
> >
> >
>
Re: All topic are assigned to same consumer in a consumer group
Posted by Pavel Molchanov <pa...@infodesk.com>.
One consumer reads from one or more partitions. If you have 1 partition per
topic, only one consumer in the consumer group will read from that
partition, others will be in standby.
To process data in parallel in the consumer group, you need to increase the
number of partitions per topic. For example, 2 partitions will allow 2
consumers to work in parallel, others will be standby.
*Pavel Molchanov*
Director of Software Development
InfoDesk
www.infodesk.com
1 Bridge Street | Suite 105 | Irvington | New York | 10551
<https://maps.google.com/?q=660+White+Plains+Road+%7C+Suite+300+%7C+Tarrytown+%7C+New+York+%7C+10591&entry=gmail&source=g>|
Office: +1 (914) 332-5940
Change Privacy Settings
<https://www.infodesk.com/unsubscription-options> | Contact
Privacy Team <un...@infodesk.com> | Privacy Policy
<https://www.infodesk.com/privacy-policy>
This e-mail message may contain confidential or legally privileged
information and is intended only for the use of the intended recipient(s).
Any unauthorized disclosure, dissemination, distribution, copying or the
taking of any action in reliance on the information herein is prohibited.
On Thu, Jun 13, 2019 at 10:53 AM sunilverm@gmail.com <su...@gmail.com>
wrote:
> I have 100 topics with one partition each.
> I have 10 instance of consumer with the Same consumer group id and I'm
> using subscribe using pattern.
> I observe that all the topic partitions are assigned to the same consumer
> and others are starving. It is not distributed.
> I was expecting that rebalance would kick-in and the different topic
> partitions would be assigned to different consumer instances.
> Can anyone confirm if this is expected behavior.
>
> Thanks,
> Sunil
>
>