Looking for help with a question on the consumer API

["Moiz Raja (moraja)" <mo...@cisco.com.INVALID>]

Hi All,

I have an issue with the consumer getting kicked out of the group possibly due to other issues going on in the system. The issue is detailed here https://stackoverflow.com/questions/51754794/how-to-reinstate-a-kafka-consumer-which-has-been-kicked-out-of-the-group

Any help with this issue would be appreciated.

Regards,
-Moiz

Re: Looking for help with a question on the consumer API

[Moiz Raja <mo...@viptela.com>]

Actually there is very minimal work done by the thread doing the polling. However it is possible that the thread doing the polling may not get schedule due to other things happening on the system - like a long GC pause.

From what you all are saying it sounds like a poll will do a heartbeat which will cause the consumer to be added back to the consumer group. If that is the case then I think I either have a problem with session timeout being hit or an issue with the thread doing the poll dying.

> On Aug 9, 2018, at 12:06 PM, Manoj Khangaonkar <kh...@gmail.com> wrote:
> 
> Hi,
> 
> Yes , if you don'nt call poll within the configured timeouts, the broker
> thinks the consumer is gone.
> 
> But increasing the timeout is not a sustainable design.
> 
> In general the code in the consumer poll loop should be very fast and do
> minimal work.
> 
> Any heavy duty work should be done by handing of to a thread pool.
> 
> What is happening is that the partitions for this consumer are assigned to
> another.
> 
> If reading too many messages too fast becomes an issue, You can consider
> using the pause and resume API to pause the consumer.
> 
> In some special cases, you might consider manually assigning partitions to
> the consumer.
> 
> 
> regards
> 
> 
> On Wed, Aug 8, 2018 at 1:30 PM, Moiz Raja (moraja) <moraja@cisco.com.invalid
>> wrote:
> 
>> Hi All,
>> 
>> I have an issue with the consumer getting kicked out of the group possibly
>> due to other issues going on in the system. The issue is detailed here
>> https://stackoverflow.com/questions/51754794/how-to-
>> reinstate-a-kafka-consumer-which-has-been-kicked-out-of-the-group
>> 
>> Any help with this issue would be appreciated.
>> 
>> Regards,
>> -Moiz
>> 
> 
> 
> 
> -- 
> http://khangaonkar.blogspot.com/

Re: Looking for help with a question on the consumer API

[Manoj Khangaonkar <kh...@gmail.com>]

Hi,

Yes , if you don'nt call poll within the configured timeouts, the broker
thinks the consumer is gone.

But increasing the timeout is not a sustainable design.

In general the code in the consumer poll loop should be very fast and do
minimal work.

Any heavy duty work should be done by handing of to a thread pool.

What is happening is that the partitions for this consumer are assigned to
another.

If reading too many messages too fast becomes an issue, You can consider
using the pause and resume API to pause the consumer.

In some special cases, you might consider manually assigning partitions to
the consumer.


regards


On Wed, Aug 8, 2018 at 1:30 PM, Moiz Raja (moraja) <moraja@cisco.com.invalid
> wrote:

> Hi All,
>
> I have an issue with the consumer getting kicked out of the group possibly
> due to other issues going on in the system. The issue is detailed here
> https://stackoverflow.com/questions/51754794/how-to-
> reinstate-a-kafka-consumer-which-has-been-kicked-out-of-the-group
>
> Any help with this issue would be appreciated.
>
> Regards,
> -Moiz
>



-- 
http://khangaonkar.blogspot.com/

Re: Looking for help with a question on the consumer API

[Shantanu Deshmukh <sh...@gmail.com>]

Consumer gets kicked out if it fails to send heart beat in designated time
period. Every call to poll sends one heart beat to consumer group
coordinator.

You need to look at *how much time is it taking to process your single
record*. *Maybe it is exceeding session.timeout.ms
<http://session.timeout.ms> value* which you have set as 30s. Try
increasing that. Also *keep max.poll.records to a lower value*. This
setting determines how many records are fetched after the call to poll
method. If you fetch too many records then even if you keep
session.timeout.ms to a large value your consumer might still get kicked
out and group will enter rebalancing stage.

On Thu, Aug 9, 2018 at 2:00 AM Moiz Raja (moraja) <mo...@cisco.com.invalid>
wrote:

> Hi All,
>
> I have an issue with the consumer getting kicked out of the group possibly
> due to other issues going on in the system. The issue is detailed here
> https://stackoverflow.com/questions/51754794/how-to-reinstate-a-kafka-consumer-which-has-been-kicked-out-of-the-group
>
> Any help with this issue would be appreciated.
>
> Regards,
> -Moiz
>