[Production Guideline] Is it good idea to provide limited number of brokers(3 brokers) in kafka client in large cluster

[Pritam Kadam <kp...@thoughtworks.com>]

Hi,

I have a use case where my kafka cluster will have 1000 brokers and I am
writing kafka client.
In order to write client, i need to provide brokers list.

Question is, what are the recommended guidelines to provide brokers list in
client?

*Is there any proxy like service available in kafka which we can give to
client?*
 - that proxy will know all the brokers in cluster and connect client to
appropriate broker.
 - like in redis world, we have twemproxy (nutcracker)

*Is it recommended to provide any specific number of brokers in client, for
example provide list of 3 brokers even though cluster has 1000 nodes?*
 - what if provided brokers gets crashed?
 - what if provided brokers restarts and there location/ip changes?

Thanks,
Pritam.

Re: [Production Guideline] Is it good idea to provide limited number of brokers(3 brokers) in kafka client in large cluster

[Jakub Scholz <ja...@scholz.cz>]

Hi Pritam,

It is basically about the probabilities, right? If you use 3 brokers as
bootstrap servers, what is the probability that all three will be
unavailable at the same time? To make it more reliable:
- the more brokers in the bootstrap list the lower the probability that all
of them will be down
- use brokers from different availability zones / racks etc.

If you plan to have many clients which are connecting and disconnecting
often, you might also want to make sure that not every client uses the same
3 brokers or that they do not have too much other work to do, so that you
don't overload them. Of course if your brokers do not have some permanent
address (IP or DNS) you will need to change the config from time to time.
So this would not be optimal.

I don't think there is any special proxy to help with this. But you
actually don't need one. You can for example use a regular loadbalancer /
proxy which simply keeps a list of living brokers and forwards the
connection to one of them. They do not need any special support for Kafka.

Jakub


On Wed, Jan 17, 2018 at 9:48 AM, Pritam Kadam <kp...@thoughtworks.com>
wrote:

> Hi,
>
> I have a use case where my kafka cluster will have 1000 brokers and I am
> writing kafka client.
> In order to write client, i need to provide brokers list.
>
> Question is, what are the recommended guidelines to provide brokers list in
> client?
>
> *Is there any proxy like service available in kafka which we can give to
> client?*
>  - that proxy will know all the brokers in cluster and connect client to
> appropriate broker.
>  - like in redis world, we have twemproxy (nutcracker)
>
> *Is it recommended to provide any specific number of brokers in client, for
> example provide list of 3 brokers even though cluster has 1000 nodes?*
>  - what if provided brokers gets crashed?
>  - what if provided brokers restarts and there location/ip changes?
>
> Thanks,
> Pritam.
>

Fwd: [Production Guideline] Is it good idea to provide limited number of brokers(3 brokers) in kafka client in large cluster

[Pritam Kadam <kp...@thoughtworks.com>]

Hi,

I have a use case where my kafka cluster will have 1000 brokers and I am
writing kafka client.
In order to write client, i need to provide brokers list.

Question is, what are the recommended guidelines to provide brokers list in
client?

*Is there any proxy like service available in kafka which we can give to
client?*
 - that proxy will know all the brokers in cluster and connect client to
appropriate broker.
 - like in redis world, we have twemproxy (nutcracker)

*Is it recommended to provide any specific number of brokers in client, for
example provide list of 3 brokers even though cluster has 1000 nodes?*
 - what if provided brokers gets crashed?
 - what if provided brokers restarts and there location/ip changes?

Thanks,
Pritam.