Second consumer within the same thread gets stuck during poll call

[Sasa Trifunovic <sa...@gmail.com>]

 I have two Kafka consumers, subscribed to different topics and belonging
to the same consumer group, having different consumer ids, running in the
same thread. They are executing poll sequentially but after the first is
done second seems to be stuck in poll. I tried using proper Kafka broker
and embedded one for testing and nothing changes.I tried associating them
with different consumer groups and that seems to be working but
unfortunately, that is not a viable solution for me.

I found this in "Kafka: The Definitive Guide":

"You can’t have multiple consumers that belong to the same group in one
thread and you can’t have multiple threads safely use the same consumer.
One consumer per thread is the rule."

That quote directs me towards some form of thread cooperation due to the
specific order of message processing I need to do.

Can someone provide an explanation of why is it necessary to run different
consumers belonging to the same consumer group, subscribed to different
topics in separate threads?

Test code with embedded kafka:

@Test
public void pollTest() {
    kafkaAdmin.createTopics(topics1);
    kafkaAdmin.createTopics(topics2);

    Consumer<String, String> consumer1 = createConsumer(topics1);
    Consumer<String, String> consumer2 = createConsumer(topics2);

    consumer1.poll(200);
    consumer2.poll(200);
}

Version of kafka - 2.3.0

Thank you.

Re: Second consumer within the same thread gets stuck during poll call

[Sasa Trifunovic <sa...@gmail.com>]

Thank you for the explanation.
I tried using them in the same thread because there is a need for message
prioritization,
thus I tried polling from high priority and then from lower priority topic.



On Wed, Dec 11, 2019 at 8:24 AM Matthias J. Sax <ma...@confluent.io>
wrote:

> First of all, from a performance point of view, it does not make sense
> to have two consumers of the same group in one thread. They will share
> the same network and same CPU resource (but are not able to utilize them
> more efficiently as both are on the same thread). Each consumer within a
> group will get some partitions assigned, thus, instead of having two
> consumers (with partition set A and B assigned), just use one consumer
> that gets A+B assigned and it will perform equally efficient.
>
> Why does it block? Well, after the first consumer calls poll(), it will
> join the group and the group will have 1 member. When the second
> consumer calls poll() it will try to join the group. The broker side
> group coordinator knows that was already one consumer in the group and
> it will wait until the first consumer say "I am still here and still
> part of the group" -- however, this will never happen, because the first
> consumer would do this via calling poll(), but it can't, because the
> second consumer is stuck in its own poll() call to actually join the
> group. Eventually the first consumer would time out and be remove from
> the group and the second consumer unblocks (the group has still one
> member only). Afterward the first consumer will retry to join the
> group... etc. etc. This ping pong game will continue forever...
>
> -Matthias
>
> On 12/10/19 5:52 AM, Sasa Trifunovic wrote:
> >  I have two Kafka consumers, subscribed to different topics and belonging
> > to the same consumer group, having different consumer ids, running in the
> > same thread. They are executing poll sequentially but after the first is
> > done second seems to be stuck in poll. I tried using proper Kafka broker
> > and embedded one for testing and nothing changes.I tried associating them
> > with different consumer groups and that seems to be working but
> > unfortunately, that is not a viable solution for me.
> >
> > I found this in "Kafka: The Definitive Guide":
> >
> > "You can’t have multiple consumers that belong to the same group in one
> > thread and you can’t have multiple threads safely use the same consumer.
> > One consumer per thread is the rule."
> >
> > That quote directs me towards some form of thread cooperation due to the
> > specific order of message processing I need to do.
> >
> > Can someone provide an explanation of why is it necessary to run
> different
> > consumers belonging to the same consumer group, subscribed to different
> > topics in separate threads?
> >
> > Test code with embedded kafka:
> >
> > @Test
> > public void pollTest() {
> >     kafkaAdmin.createTopics(topics1);
> >     kafkaAdmin.createTopics(topics2);
> >
> >     Consumer<String, String> consumer1 = createConsumer(topics1);
> >     Consumer<String, String> consumer2 = createConsumer(topics2);
> >
> >     consumer1.poll(200);
> >     consumer2.poll(200);
> > }
> >
> > Version of kafka - 2.3.0
> >
> > Thank you.
> >
>
>

-- 
www.fractalcastle.com

Re: Second consumer within the same thread gets stuck during poll call

["Matthias J. Sax" <ma...@confluent.io>]

First of all, from a performance point of view, it does not make sense
to have two consumers of the same group in one thread. They will share
the same network and same CPU resource (but are not able to utilize them
more efficiently as both are on the same thread). Each consumer within a
group will get some partitions assigned, thus, instead of having two
consumers (with partition set A and B assigned), just use one consumer
that gets A+B assigned and it will perform equally efficient.

Why does it block? Well, after the first consumer calls poll(), it will
join the group and the group will have 1 member. When the second
consumer calls poll() it will try to join the group. The broker side
group coordinator knows that was already one consumer in the group and
it will wait until the first consumer say "I am still here and still
part of the group" -- however, this will never happen, because the first
consumer would do this via calling poll(), but it can't, because the
second consumer is stuck in its own poll() call to actually join the
group. Eventually the first consumer would time out and be remove from
the group and the second consumer unblocks (the group has still one
member only). Afterward the first consumer will retry to join the
group... etc. etc. This ping pong game will continue forever...

-Matthias

On 12/10/19 5:52 AM, Sasa Trifunovic wrote:
>  I have two Kafka consumers, subscribed to different topics and belonging
> to the same consumer group, having different consumer ids, running in the
> same thread. They are executing poll sequentially but after the first is
> done second seems to be stuck in poll. I tried using proper Kafka broker
> and embedded one for testing and nothing changes.I tried associating them
> with different consumer groups and that seems to be working but
> unfortunately, that is not a viable solution for me.
> 
> I found this in "Kafka: The Definitive Guide":
> 
> "You can’t have multiple consumers that belong to the same group in one
> thread and you can’t have multiple threads safely use the same consumer.
> One consumer per thread is the rule."
> 
> That quote directs me towards some form of thread cooperation due to the
> specific order of message processing I need to do.
> 
> Can someone provide an explanation of why is it necessary to run different
> consumers belonging to the same consumer group, subscribed to different
> topics in separate threads?
> 
> Test code with embedded kafka:
> 
> @Test
> public void pollTest() {
>     kafkaAdmin.createTopics(topics1);
>     kafkaAdmin.createTopics(topics2);
> 
>     Consumer<String, String> consumer1 = createConsumer(topics1);
>     Consumer<String, String> consumer2 = createConsumer(topics2);
> 
>     consumer1.poll(200);
>     consumer2.poll(200);
> }
> 
> Version of kafka - 2.3.0
> 
> Thank you.
>