Recommended way of handling brokers coming up / down in the SimpleConsumer

[Rajiv Kurian <ra...@signalfuse.com>]

I am using the SimpleConsumer to consume specific partitions on specific
processes. The workflow is kind of like this:

i) An external arbiter assigns partitions to a specific processes. It
provides the guarantees of:
   a) All partitions are consumed by the cluster.
   b) A single partition is only consumed by a single host.
ii) Each host when assigned partitions figures out the leaders for each of
those partitions. It then starts a thread each for each kafka leader that
it needs to connect to and consumes from these partitions.
iii)The offset management is also done external to Kafka.
iv) f any of these requests lead to responses with errors which signify
that the partitions have moved to new leaders (UnknownTopicOrPartition,
LeaderNotAvailable, NotLeaderForPartitionCode etc etc), we refresh the
broker -> partition mapping for the partitions we own and then start/stop
threads based on which leaders we need to talk to and rinse and repeat.

I refresh the broker -> partition mapping by querying kafka brokers
serially with TopicMetadataRequest for the partitions that my host owns. As
soon as I get leaders for all the partitions I own I break out i.e. I don't
ask all of my seed brokers the same question. The TopicMetadataRequest just
seems to be a proxy to some data stored in ZK. I am not sure whether this
request will always give me all the partitions I am querying for. For
example what do I get if one of the partitions I am making a
TopicMetadataRequest for is in the middle of moving to a new leader?

If I lose a broker, the thread for that broker starts complaining about not
being able to connect to it and I refresh the broker -> partition mapping
and the partitions should now be assigned to another broker. Similarly if
one of the downed brokers comes back up, some partitions will be assigned
to it and  the per-broker consumer thread which owned the partitions will
complain and I refresh the broker->partition mapping and will know about
the new leader for those partitions. I am still worried about the
transition phase like I mentioned in the previous paragraph. I don't quite
know what to do if I query for TopicMetadataRequest and don't get results
for all the partitions I queried. Maybe just retry with an exponential
back-off till I get results? Again is this even possible or is the ZK
partition->leader mapping always consistent?

Thanks!