Kafka Cluster management

[Cassa L <lc...@gmail.com>]

Hi,
 Is there documentation on how connection management, connection pooling
works in Kafka for multiple brokers? Do I need to take care of my own
connection management for cluster? e.g. while publishing messages, if one
broker stops responding, I need to switch to the other. Does Kafka client
takes care of this or do I need load balancer between brokers and client?

Thanks,
LCassa

Re: Kafka Cluster management

[Todd Palino <tp...@linkedin.com>]

Both the producer and consumer clients will take care of rebalancing if
there is a broker failure. The consumer handles this entirely in Zookeeper.

For the producer side, you do need some sort of front end to handle the
metadata requests, or else you would constantly have to change your
producer configuration if brokers fail. In our case, we use a load
balancer for this because it's the most robust solution. Only the metadata
request itself will go through the load balancer. Once the producer has
the metadata for the topic and partition, it reconnects directly to the
leader for the partition for producing.

-Todd

On 3/11/14 10:07 PM, "Cassa L" <lc...@gmail.com> wrote:

>Hi,
> Is there documentation on how connection management, connection pooling
>works in Kafka for multiple brokers? Do I need to take care of my own
>connection management for cluster? e.g. while publishing messages, if one
>broker stops responding, I need to switch to the other. Does Kafka client
>takes care of this or do I need load balancer between brokers and client?
>
>Thanks,
>LCassa