You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by cs user <ac...@gmail.com> on 2016/06/03 12:15:42 UTC
Kafka behind a load balancer
Hi All,
Does anyone have any experience of using kafka behind a load balancer?
Would this work? Are there any reasons why you would not want to do it?
Thanks!
Re: Kafka behind a load balancer
Posted by Todd Palino <tp...@gmail.com>.
Yep, what Ewen said. We have all of our Kafka clusters behind hardware load
balancers. Producers (and eventually consumers, once we switch to the new
consumer) get configured with those VIPs. It’s better than providing a list
of brokers for the cluster, because we often change the particular brokers
in the cluster.
Note that there is a difference in behavior between the older producer
library and the current one. The old producer would make all metadata
requests through the VIP (the broker list), and then connect directly to
the individual brokers for produce/fetch. The current producer uses the VIP
(bootstrap servers) once at startup to populate the list the list of
brokers for the cluster and then makes metadata and all other requests to
those brokers directly (not through the VIP).
As a side note here, has anyone validated the behavior of the clients when
the bootstrap.servers is a round robin DNS entry?
-Todd
On Sun, Jun 5, 2016 at 6:34 AM, Ewen Cheslack-Postava <ew...@confluent.io>
wrote:
> Note, however, that a load balancer can be useful for bootstrapping
> purposes, i.e. use it for the bootstrap.servers setting to have a single
> consistent value for the setting but allow the broker list to change over
> time. From there, as Tom says, it'll start using broker hostnames and
> automatically target the specific brokers it needs to communicate with.
>
> -Ewen
>
> On Fri, Jun 3, 2016 at 5:37 AM, cs user <ac...@gmail.com> wrote:
>
> > Hi Tom,
> >
> > That's great, I thought as much, thanks for taking the time to respond,
> > much appreciated!
> >
> > Cheers
> >
> > On Fri, Jun 3, 2016 at 1:18 PM, Tom Crayford <tc...@heroku.com>
> wrote:
> >
> > > Hi,
> > >
> > > Kafka is designed to distribute traffic between brokers itself. It's
> > > naturally distributed and does not need, and indeed will not work
> behind
> > a
> > > load balancer. I'd recommend reading the docs for more, but
> > > http://kafka.apache.org/documentation.html#design_loadbalancing is a
> > good
> > > start.
> > >
> > > Thanks
> > >
> > > Tom Crayford
> > > Heroku Kafka
> > >
> > > On Fri, Jun 3, 2016 at 1:15 PM, cs user <ac...@gmail.com> wrote:
> > >
> > > > Hi All,
> > > >
> > > > Does anyone have any experience of using kafka behind a load
> balancer?
> > > >
> > > > Would this work? Are there any reasons why you would not want to do
> it?
> > > >
> > > > Thanks!
> > > >
> > >
> >
>
>
>
> --
> Thanks,
> Ewen
>
--
*Todd Palino*
Staff Site Reliability Engineer
Data Infrastructure Streaming
linkedin.com/in/toddpalino
Re: Kafka behind a load balancer
Posted by Ewen Cheslack-Postava <ew...@confluent.io>.
Note, however, that a load balancer can be useful for bootstrapping
purposes, i.e. use it for the bootstrap.servers setting to have a single
consistent value for the setting but allow the broker list to change over
time. From there, as Tom says, it'll start using broker hostnames and
automatically target the specific brokers it needs to communicate with.
-Ewen
On Fri, Jun 3, 2016 at 5:37 AM, cs user <ac...@gmail.com> wrote:
> Hi Tom,
>
> That's great, I thought as much, thanks for taking the time to respond,
> much appreciated!
>
> Cheers
>
> On Fri, Jun 3, 2016 at 1:18 PM, Tom Crayford <tc...@heroku.com> wrote:
>
> > Hi,
> >
> > Kafka is designed to distribute traffic between brokers itself. It's
> > naturally distributed and does not need, and indeed will not work behind
> a
> > load balancer. I'd recommend reading the docs for more, but
> > http://kafka.apache.org/documentation.html#design_loadbalancing is a
> good
> > start.
> >
> > Thanks
> >
> > Tom Crayford
> > Heroku Kafka
> >
> > On Fri, Jun 3, 2016 at 1:15 PM, cs user <ac...@gmail.com> wrote:
> >
> > > Hi All,
> > >
> > > Does anyone have any experience of using kafka behind a load balancer?
> > >
> > > Would this work? Are there any reasons why you would not want to do it?
> > >
> > > Thanks!
> > >
> >
>
--
Thanks,
Ewen
Re: Kafka behind a load balancer
Posted by cs user <ac...@gmail.com>.
Hi Tom,
That's great, I thought as much, thanks for taking the time to respond,
much appreciated!
Cheers
On Fri, Jun 3, 2016 at 1:18 PM, Tom Crayford <tc...@heroku.com> wrote:
> Hi,
>
> Kafka is designed to distribute traffic between brokers itself. It's
> naturally distributed and does not need, and indeed will not work behind a
> load balancer. I'd recommend reading the docs for more, but
> http://kafka.apache.org/documentation.html#design_loadbalancing is a good
> start.
>
> Thanks
>
> Tom Crayford
> Heroku Kafka
>
> On Fri, Jun 3, 2016 at 1:15 PM, cs user <ac...@gmail.com> wrote:
>
> > Hi All,
> >
> > Does anyone have any experience of using kafka behind a load balancer?
> >
> > Would this work? Are there any reasons why you would not want to do it?
> >
> > Thanks!
> >
>
Re: Kafka behind a load balancer
Posted by Tom Crayford <tc...@heroku.com>.
Hi,
Kafka is designed to distribute traffic between brokers itself. It's
naturally distributed and does not need, and indeed will not work behind a
load balancer. I'd recommend reading the docs for more, but
http://kafka.apache.org/documentation.html#design_loadbalancing is a good
start.
Thanks
Tom Crayford
Heroku Kafka
On Fri, Jun 3, 2016 at 1:15 PM, cs user <ac...@gmail.com> wrote:
> Hi All,
>
> Does anyone have any experience of using kafka behind a load balancer?
>
> Would this work? Are there any reasons why you would not want to do it?
>
> Thanks!
>