Not able to run multiple consumer instances

[paresh <pa...@glam.com>]

Hi Team,

Can anybody help with issue I am facing with running multiple consumer 
instances on single topic having single partition?

Kafka is broadcasting messages to all the consumers though all consumers 
are running in same group
I am using kafka-php client lib to connect with kafka server and using 
socket connect (Encoder/Decoder) for connecting consumer and producer.

Regards,
Paresh Behede

Re: Not able to run multiple consumer instances

[R Krishna <kr...@gmail.com>]

Not sure about your config, but I read somewhere (also a newbie) that if
number of consumers is more than number of partitions on the topic, some
will not get any messages. Search consumer parallelism.

On Fri, Apr 29, 2016 at 4:11 AM, Marko Bonaći <ma...@sematext.com>
wrote:

> Yes, a partition can be accessed by only a single thread at a time, so that
> second consumer of yours, since it's in the same group, has nothing to do.
> If you add more partitions your consumers would help each other consume all
> messages out of that topic (roughly 50% of messages each).
> If you were to assign the second consumer with a different group, each
> consumer would consume all messages (independently of one another).
>
> BTW, Kafka is not broadcasting anything, your consumers are pulling
> messages out of Kafka :)
>
> Marko Bonaći
> Monitoring | Alerting | Anomaly Detection | Centralized Log Management
> Solr & Elasticsearch Support
> Sematext <http://sematext.com/> | Contact
> <http://sematext.com/about/contact.html>
>
> On Fri, Apr 29, 2016 at 12:18 PM, paresh <pa...@glam.com> wrote:
>
> > Hi Team,
> >
> > Can anybody help with issue I am facing with running multiple consumer
> > instances on single topic having single partition?
> >
> > Kafka is broadcasting messages to all the consumers though all consumers
> > are running in same group
> > I am using kafka-php client lib to connect with kafka server and using
> > socket connect (Encoder/Decoder) for connecting consumer and producer.
> >
> > Regards,
> > Paresh Behede
> >
>



-- 
Radha Krishna, Proddaturi
253-234-5657

Re: Not able to run multiple consumer instances

[Marko Bonaći <ma...@sematext.com>]

Yes, a partition can be accessed by only a single thread at a time, so that
second consumer of yours, since it's in the same group, has nothing to do.
If you add more partitions your consumers would help each other consume all
messages out of that topic (roughly 50% of messages each).
If you were to assign the second consumer with a different group, each
consumer would consume all messages (independently of one another).

BTW, Kafka is not broadcasting anything, your consumers are pulling
messages out of Kafka :)

Marko Bonaći
Monitoring | Alerting | Anomaly Detection | Centralized Log Management
Solr & Elasticsearch Support
Sematext <http://sematext.com/> | Contact
<http://sematext.com/about/contact.html>

On Fri, Apr 29, 2016 at 12:18 PM, paresh <pa...@glam.com> wrote:

> Hi Team,
>
> Can anybody help with issue I am facing with running multiple consumer
> instances on single topic having single partition?
>
> Kafka is broadcasting messages to all the consumers though all consumers
> are running in same group
> I am using kafka-php client lib to connect with kafka server and using
> socket connect (Encoder/Decoder) for connecting consumer and producer.
>
> Regards,
> Paresh Behede
>