Problems encountered during the consumer shutdown.

[Paresh Shah <pa...@ecofactor.com>]

We have a use case where we would want to to shutdown the consumers if they
do not have anything to process. The process which these consumers are a
part is still alive.

Once the consumer shutdown we see that the Zookeeper connections are not
closed/cleaned. Since these consumer's will come back up again periodically
and the process repeats. At some point we see that the rebalance starts
failing since the old zookeeper client's are still in connected state.

Wanted to know if anyone else has seen this problem.

thanks
Paresh

Re: Problems encountered during the consumer shutdown.

[Neha Narkhede <ne...@gmail.com>]

You can configure consumer.timeout.ms that basically notifies your consumer
iterator when there is no data for consumer.timeout.ms milliseconds. You
can then catch it and invoke shutdown() on the consumer. Once the shutdown
returns, the consumer will no longer maintain its connection to zookeeper.
Is that what you are looking for?

Thanks,
Neha


On Tue, Jan 28, 2014 at 3:21 PM, Paresh Shah <pa...@ecofactor.com>wrote:

> We have a use case where we would want to to shutdown the consumers if they
> do not have anything to process. The process which these consumers are a
> part is still alive.
>
> Once the consumer shutdown we see that the Zookeeper connections are not
> closed/cleaned. Since these consumer's will come back up again periodically
> and the process repeats. At some point we see that the rebalance starts
> failing since the old zookeeper client's are still in connected state.
>
> Wanted to know if anyone else has seen this problem.
>
> thanks
> Paresh
>