Zookeeper fails to see all the brokers at once

[vivek thakre <vi...@gmail.com>]

Hello All,

I have a Kafka Cluster deployed on AWS.
I am noticing this issue randomly when none of the brokers are registered
with zookeeper ( I have set up a monitor on this by using zk-shell util)

During this issue, the cluster continues to operate i.e events can be
produced and consumed. But the topics can not be created or listed

When I query for brokers using Zk Shell (ls /brokers/ids) , its all empty.

I have noticed this issue in both following setup:
1. 3 Zookeepers and 12 Brokers are deployed on Single AWS Zone
2. Multizone deployment, where 3 Zookeepers one on each AWS zone and 12
brokers, 4 on each AWS Zone.

Not sure if this is related to AWS or Kafka.
I looked for logs , but do not see any errors (currently log level is set
to INFO)

Kafka version is 0.9.0.1. (I have seen this issue happened with 0.8.2.1 too)

Has any one encountered this issue?
I am looking for some pointers as to what could be wrong in my setup and
how should I go about debugging

Summary of broker's server.properties

log.index.interval.bytes=4096
log.index.size.max.bytes=10485760
log.flush.scheduler.interval.ms=2000
log.flush.interval.ms=10000
log.flush.interval.messages=20000
log.cleanup.interval.mins=30
log.segment.bytes=500000000
log.retention.check.interval.ms=180000
message.max.bytes=9000000
num.network.threads=3
num.io.threads=8
num.replica.fetchers=4
num.partitions=2
num.recovery.threads.per.data.dir=1
queued.max.requests=500
replica.fetch.max.bytes=1000012
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
zookeeper.connection.timeout.ms=60000
zookeeper.session.timeout.ms=10000

Thanks,
Vivek