Zookeeper offset

[Srividhya Shanmugam <Sr...@fico.com>]

Kafka Team,

In the integration environment Kafka and zookeeper are running under supervision. Once in a while when zookeeper and kafka are shut-down and started back again, the consumers are not able to read the data from the topic. I am not seeing any exceptions in the log. The consumer offset checker utility does show a lag for the consumer group.

Does that mean when kafka/zookeeper are shut-down abruptly, it's possible that the zookeeper data directories are not committed with proper offset or got corrupted? I tried with new consumer groups too and with simple consumers. After that point, I am not able to retrieve the data from the topic. How do I recover the data?

This is a critical problem and any help is really appreciated.

Thanks!

This email and any files transmitted with it are confidential, proprietary and intended solely for the individual or entity to whom they are addressed. If you have received this email in error please delete it immediately.

Re: Zookeeper offset

[Joel Koshy <jj...@gmail.com>]

Do you have _any_ logs? i.e., does the consumer log say that it is
rebalancing? If not, what is your log4j level setting?

Joel

On Thu, Jul 31, 2014 at 08:40:45PM +0000, Srividhya Shanmugam wrote:
> Kafka Team,
> 
> In the integration environment Kafka and zookeeper are running under supervision. Once in a while when zookeeper and kafka are shut-down and started back again, the consumers are not able to read the data from the topic. I am not seeing any exceptions in the log. The consumer offset checker utility does show a lag for the consumer group.
> 
> Does that mean when kafka/zookeeper are shut-down abruptly, it's possible that the zookeeper data directories are not committed with proper offset or got corrupted? I tried with new consumer groups too and with simple consumers. After that point, I am not able to retrieve the data from the topic. How do I recover the data?
> 
> This is a critical problem and any help is really appreciated.
> 
> Thanks!
> 
> This email and any files transmitted with it are confidential, proprietary and intended solely for the individual or entity to whom they are addressed. If you have received this email in error please delete it immediately.