Kafka consumer session timeouts

[Terry Cumaranatunge <cu...@gmail.com>]

Hi
I would like to get some feedback on design choices with kafka consumers.
We have an application that a consumer reads a message and the thread does
a number of things, including database accesses before a message is
produced to another topic. The time between consuming and producing the
message on the thread can take several minutes. Once message is produced to
new topic, a commit is done to indicate we are done with work on the
consumer queue message. Auto commit is disabled for this reason.

I'm using the high level consumer and what I'm noticing is that zookeeper
and kafka sessions timeout because it is taking too long before we do
anything on consumer queue so kafka ends up rebalancing every time the
thread goes back to read more from consumer queue and it starts to take a
long time before a consumer reads a new message after a while.

I can set zookeeper session timeout very high to not make that a problem
but then i have to adjust the rebalance parameters accordingly and kafka
won't pickup a new consumer for a while among other side effects.

What are my options to solve this problem? Is there a way to heartbeat to
kafka and zookeeper to keep both happy? Do i still have these same issues
if i were to use a simple consumer?

Thanks

Re: Kafka consumer session timeouts

[Neha Narkhede <ne...@confluent.io>]

Terry,

The zookeeper client used by the high level Kafka consumer has a separate
thread that does the heartbeat in the background. So even if it takes long
to process the message, it should not make the consumer's session to time
out or make the consumer rebalance. You may be running into long GC pauses
on your consumer that might be causing the session timeouts.

Thanks,
Neha

On Fri, Dec 19, 2014 at 12:51 PM, Terry Cumaranatunge <cu...@gmail.com>
wrote:

> Hi
> I would like to get some feedback on design choices with kafka consumers.
> We have an application that a consumer reads a message and the thread does
> a number of things, including database accesses before a message is
> produced to another topic. The time between consuming and producing the
> message on the thread can take several minutes. Once message is produced to
> new topic, a commit is done to indicate we are done with work on the
> consumer queue message. Auto commit is disabled for this reason.
>
> I'm using the high level consumer and what I'm noticing is that zookeeper
> and kafka sessions timeout because it is taking too long before we do
> anything on consumer queue so kafka ends up rebalancing every time the
> thread goes back to read more from consumer queue and it starts to take a
> long time before a consumer reads a new message after a while.
>
> I can set zookeeper session timeout very high to not make that a problem
> but then i have to adjust the rebalance parameters accordingly and kafka
> won't pickup a new consumer for a while among other side effects.
>
> What are my options to solve this problem? Is there a way to heartbeat to
> kafka and zookeeper to keep both happy? Do i still have these same issues
> if i were to use a simple consumer?
>
> Thanks
>



-- 
Thanks,
Neha