Regarding an issue with message distributions in kafka.

[Puneet Dubey <pu...@epictenet.com>]

Hi.

In our app we have integrated kafka to publish and subscribe to the events.
Based on the documentation and distribution policy we designed our
application where we are facing some contradictory behavior. As per the
documentation at a given time there can be a maximum of one consumer per
partition. Our application is written in spring boot and we are using
org.springframework.kafka dependency to manage kafka and kafka clusters are
hosted in aws ec2 instances.

While producing messages we are writing them in a specific partition let's
say 0 and we have a group of 3 consumers all reading same partition and
belong to the same group. Our expectation was that at a time only one
active consumer rest of the two will be idle. But what's happening is all
of them are receiving and processing the same message. Due to which desired
action is being performed thrice. Please suggest a solution.

-- 

Thanks & Regards,

Puneet Dubey
*Technical Consultant*

[image: Epictenet] <https://www.epictenet.com/>

*M.* +91 8269285876
*P.* +0755 4030158
*W.* www.epictenet.com
*A.* 3 & 11B, Essargee New Fort, BHEL, Bhopal (M.P.) 462022, India
*HO. *Level 10, 47 York Street, Sydney, NSW 2000, Australia

Re: Regarding an issue with message distributions in kafka.

[Philip Nee <ph...@gmail.com>]

Hello Puneet,

Why do you need multiple consumers from the same group consuming from the
same partition? I am trying to think of a compiling reason for such a use
case...  It makes more sense to me to have a thread pool to process the
records fetched from a single consumer.  This is a great article to read:
https://www.confluent.io/blog/kafka-consumer-multi-threaded-messaging/

P

On Wed, Nov 23, 2022 at 8:00 AM Puneet Dubey <pu...@epictenet.com>
wrote:

> Hi.
>
> In our app we have integrated kafka to publish and subscribe to the
> events. Based on the documentation and distribution policy we designed our
> application where we are facing some contradictory behavior. As per the
> documentation at a given time there can be a maximum of one consumer per
> partition. Our application is written in spring boot and we are using
> org.springframework.kafka dependency to manage kafka and kafka clusters are
> hosted in aws ec2 instances.
>
> While producing messages we are writing them in a specific partition let's
> say 0 and we have a group of 3 consumers all reading same partition and
> belong to the same group. Our expectation was that at a time only one
> active consumer rest of the two will be idle. But what's happening is all
> of them are receiving and processing the same message. Due to which desired
> action is being performed thrice. Please suggest a solution.
>
> --
>
> Thanks & Regards,
>
> Puneet Dubey
> *Technical Consultant*
>
> [image: Epictenet] <https://www.epictenet.com/>
>
> *M.* +91 8269285876
> *P.* +0755 4030158
> *W.* www.epictenet.com
> *A.* 3 & 11B, Essargee New Fort, BHEL, Bhopal (M.P.) 462022, India
> *HO. *Level 10, 47 York Street, Sydney, NSW 2000, Australia
>

Re: Regarding an issue with message distributions in kafka.

[John Roesler <vv...@apache.org>]

Hi Puneet,

Thanks for the question. It sounds like either the three consumers have different group.id configurations or they are using “assign” instead of “subscribe”.

You are correct, if the consumers are all in the same consumer group and subscribed to the same topic, Kafka will only assign partition 0 to one of them at a time. 

I hope this helps!
-John

On Wed, Nov 23, 2022, at 05:14, Puneet Dubey wrote:
> Hi.
>
> In our app we have integrated kafka to publish and subscribe to the 
> events. Based on the documentation and distribution policy we designed 
> our application where we are facing some contradictory behavior. As per 
> the documentation at a given time there can be a maximum of one 
> consumer per partition. Our application is written in spring boot and 
> we are using org.springframework.kafka dependency to manage kafka and 
> kafka clusters are hosted in aws ec2 instances. 
>
> While producing messages we are writing them in a specific partition 
> let's say 0 and we have a group of 3 consumers all reading same 
> partition and belong to the same group. Our expectation was that at a 
> time only one active consumer rest of the two will be idle. But what's 
> happening is all of them are receiving and processing the same message. 
> Due to which desired action is being performed thrice. Please suggest a 
> solution.
>
> -- 
> Thanks & Regards,____
> Puneet Dubey
> *Technical Consultant*__
> Epictenet <https://www.epictenet.com/>____
> *M.* +91 8269285876
> *P.* +0755 4030158 
> *W.* www.epictenet.com
> *A.* 3 & 11B, Essargee New Fort, BHEL, Bhopal (M.P.) 462022, India
> *HO. *Level 10, 47 York Street, Sydney, NSW 2000, Australia