Clients and replica leaders

[David Arthur <mu...@gmail.com>]

In writing a client for 0.8, I now have to keep state of which 
topic+partition is owned by what broker. This is inherently a pain to 
deal with and has the downside that I must wait for an error before I am 
notified about a change in the broker topology.

I would be nice if the clients didn't need to know so much about the 
brokers. In Apache Solr, which actually has a similar 
partition+replication strategy, each server (broker) can handle requests 
for any shard (partition) in the cluster. If the current server happens 
to be the leader then it will process the request; if not it will 
forward it to the correct server, wait for a response, then forward the 
response back to the client.

Dumb clients will pay the extra cost of the additional hop, but do not 
need to know anything about the brokers. Smart clients will work 
basically like they would now with the added benefit of not getting an 
error when leader changes.

Would a strategy like this work in 0.8? Do the brokers know about one 
another?

-David