You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Xuyen On <xo...@ancestry.com> on 2013/09/11 21:38:47 UTC
Hardware Loadbalancer vs zookeeper for producers
Can anyone from LinkedIn let me know if they are using Hardware Loadbalancer to distribute the producer messages or if they're using zookeeper?
I'm trying to find out what the pros and cons are of using a hardware loadbalancer vs. using zookeeper to distribute the load of producers sending messages to brokers. Does anyone have any experience using either/both?
Thanks,
Xuyen
RE: Hardware Loadbalancer vs zookeeper for producers
Posted by Xuyen On <xo...@ancestry.com>.
Yes, I was trying to find out how we can scale out with our Kafka cluster later if we wanted to add more topics. But as you say, it might be simpler just to use another Kafka node at some point.
Thanks for your response, it was very helpful!
Xuyen
-----Original Message-----
From: Neha Narkhede [mailto:neha.narkhede@gmail.com]
Sent: Thursday, September 12, 2013 8:34 PM
To: users@kafka.apache.org
Subject: Re: Hardware Loadbalancer vs zookeeper for producers
>> So my question is if we go with a hardware load balancer, do all the
broker nodes have to treated equally? Ie: All broker nodes will have the same topics and number of partitions for each topic?
All the brokers behind the same hardware load balancer or virtual IP will be treated equally.
>> Or is it possible to have brokers with different topics? If we go
>> with
this configuration how can the hardware load balancer direct the messages to the right broker for a given topic? We were thinking that we might be able to do this by having a different VIP's per topic. How do you guys do it?
It is tricky to selectively host topics on a subset of brokers in 0.7. You need to create the right directory structure for every topic/partition and bounce the broker. But I'm wondering why you would want to selectively host topics on certain brokers. Is the purpose being able to host more topics on the same cluster? In 0.7, an easier strategy is to logically categorize classes of topics and host topics from the same class on a separate cluster. But this strategy of course only works if there are fewer such categories of topics.
>> Lastly, why do you guys use a hardware load balancer vs. using
zookeeper? The only reason that I could think of was to offload traffic from the zookeeper cluster. Are there any other advantages to using a hardware load balancer?
We have several thousand producers that talk to various Kafka clusters. At that rate, zookeeper becomes the bottleneck. On the other hand, hardware load balancers are designed to load balance thousands of connections across machines, so that works better.
Thanks,
Neha
On Thu, Sep 12, 2013 at 1:03 PM, Xuyen On <xo...@ancestry.com> wrote:
> Hi Neha,
>
> First I should mention that we are initially using Kafka 0.7, so I
> don't think we'll have access to the API to get metadata info from the
> brokers since this is a 0.8 feature.
>
> So my question is if we go with a hardware load balancer, do all the
> broker nodes have to treated equally? Ie: All broker nodes will have
> the same topics and number of partitions for each topic?
>
> Or is it possible to have brokers with different topics? If we go with
> this configuration how can the hardware load balancer direct the
> messages to the right broker for a given topic? We were thinking that
> we might be able to do this by having a different VIP's per topic. How
> do you guys do it?
>
> Lastly, why do you guys use a hardware load balancer vs. using zookeeper?
> The only reason that I could think of was to offload traffic from the
> zookeeper cluster. Are there any other advantages to using a hardware
> load balancer?
>
> Thanks,
>
> Xuyen
>
> -----Original Message-----
> From: Neha Narkhede [mailto:neha.narkhede@gmail.com]
> Sent: Wednesday, September 11, 2013 5:43 PM
> To: users@kafka.apache.org
> Subject: Re: Hardware Loadbalancer vs zookeeper for producers
>
> Take a look at the "Producer Load balancing" section of the
> documentation
> - http://kafka.apache.org/08/documentation.html#theproducer
>
> Let us know how we can improve it to answer your question.
>
> Thanks,
> Neha
>
>
> On Wed, Sep 11, 2013 at 4:45 PM, Xuyen On <xo...@ancestry.com> wrote:
>
> > Hi Neha,
> >
> > How do you get metadata about the cluster from the hardware load
> balancer?
> > A load balancer should direct a message to brokers that are up but I
> > don't know how you would detect for things like adding/deleting
> > topics/partitions from the cluster through the hardware load balancer.
> > Can you help me understand how you guys do this?
> >
> > Thanks,
> >
> > Xuyen
> >
> > -----Original Message-----
> > From: Neha Narkhede [mailto:neha.narkhede@gmail.com]
> > Sent: Wednesday, September 11, 2013 2:39 PM
> > To: users@kafka.apache.org
> > Subject: Re: Hardware Loadbalancer vs zookeeper for producers
> >
> > Kafka 08 doesn't use zookeeper based load balancing on the producer side.
> > We rely on a hardware load balancer to be able to talk to any broker
> > and get metadata about the cluster. This happens on startup, for new
> > topics/partitions and if the producer runs into an error while
> > sending
> data.
> >
> > Thanks,
> > Neha
> >
> >
> > On Wed, Sep 11, 2013 at 12:38 PM, Xuyen On <xo...@ancestry.com> wrote:
> >
> > > Can anyone from LinkedIn let me know if they are using Hardware
> > > Loadbalancer to distribute the producer messages or if they're
> > > using zookeeper?
> > > I'm trying to find out what the pros and cons are of using a
> > > hardware loadbalancer vs. using zookeeper to distribute the load
> > > of producers sending messages to brokers. Does anyone have any
> > > experience using either/both?
> > >
> > > Thanks,
> > >
> > > Xuyen
> > >
> > >
> >
> >
> >
>
>
>
Re: Hardware Loadbalancer vs zookeeper for producers
Posted by Neha Narkhede <ne...@gmail.com>.
>> So my question is if we go with a hardware load balancer, do all the
broker nodes have to treated equally? Ie: All broker nodes will have the
same topics and number of partitions for each topic?
All the brokers behind the same hardware load balancer or virtual IP will
be treated equally.
>> Or is it possible to have brokers with different topics? If we go with
this configuration how can the hardware load balancer direct the messages
to the right broker for a given topic? We were thinking that we might be
able to do this by having a different VIP's per topic. How do you guys do
it?
It is tricky to selectively host topics on a subset of brokers in 0.7. You
need to create the right directory structure for every topic/partition and
bounce the broker. But I'm wondering why you would want to selectively host
topics on certain brokers. Is the purpose being able to host more topics on
the same cluster? In 0.7, an easier strategy is to logically categorize
classes of topics and host topics from the same class on a separate
cluster. But this strategy of course only works if there are fewer such
categories of topics.
>> Lastly, why do you guys use a hardware load balancer vs. using
zookeeper? The only reason that I could think of was to offload traffic
from the zookeeper cluster. Are there any other advantages to using a
hardware load balancer?
We have several thousand producers that talk to various Kafka clusters. At
that rate, zookeeper becomes the bottleneck. On the other hand, hardware
load balancers are designed to load balance thousands of connections across
machines, so that works better.
Thanks,
Neha
On Thu, Sep 12, 2013 at 1:03 PM, Xuyen On <xo...@ancestry.com> wrote:
> Hi Neha,
>
> First I should mention that we are initially using Kafka 0.7, so I don't
> think we'll have access to the API to get metadata info from the brokers
> since this is a 0.8 feature.
>
> So my question is if we go with a hardware load balancer, do all the
> broker nodes have to treated equally? Ie: All broker nodes will have the
> same topics and number of partitions for each topic?
>
> Or is it possible to have brokers with different topics? If we go with
> this configuration how can the hardware load balancer direct the messages
> to the right broker for a given topic? We were thinking that we might be
> able to do this by having a different VIP's per topic. How do you guys do
> it?
>
> Lastly, why do you guys use a hardware load balancer vs. using zookeeper?
> The only reason that I could think of was to offload traffic from the
> zookeeper cluster. Are there any other advantages to using a hardware load
> balancer?
>
> Thanks,
>
> Xuyen
>
> -----Original Message-----
> From: Neha Narkhede [mailto:neha.narkhede@gmail.com]
> Sent: Wednesday, September 11, 2013 5:43 PM
> To: users@kafka.apache.org
> Subject: Re: Hardware Loadbalancer vs zookeeper for producers
>
> Take a look at the "Producer Load balancing" section of the documentation
> - http://kafka.apache.org/08/documentation.html#theproducer
>
> Let us know how we can improve it to answer your question.
>
> Thanks,
> Neha
>
>
> On Wed, Sep 11, 2013 at 4:45 PM, Xuyen On <xo...@ancestry.com> wrote:
>
> > Hi Neha,
> >
> > How do you get metadata about the cluster from the hardware load
> balancer?
> > A load balancer should direct a message to brokers that are up but I
> > don't know how you would detect for things like adding/deleting
> > topics/partitions from the cluster through the hardware load balancer.
> > Can you help me understand how you guys do this?
> >
> > Thanks,
> >
> > Xuyen
> >
> > -----Original Message-----
> > From: Neha Narkhede [mailto:neha.narkhede@gmail.com]
> > Sent: Wednesday, September 11, 2013 2:39 PM
> > To: users@kafka.apache.org
> > Subject: Re: Hardware Loadbalancer vs zookeeper for producers
> >
> > Kafka 08 doesn't use zookeeper based load balancing on the producer side.
> > We rely on a hardware load balancer to be able to talk to any broker
> > and get metadata about the cluster. This happens on startup, for new
> > topics/partitions and if the producer runs into an error while sending
> data.
> >
> > Thanks,
> > Neha
> >
> >
> > On Wed, Sep 11, 2013 at 12:38 PM, Xuyen On <xo...@ancestry.com> wrote:
> >
> > > Can anyone from LinkedIn let me know if they are using Hardware
> > > Loadbalancer to distribute the producer messages or if they're using
> > > zookeeper?
> > > I'm trying to find out what the pros and cons are of using a
> > > hardware loadbalancer vs. using zookeeper to distribute the load of
> > > producers sending messages to brokers. Does anyone have any
> > > experience using either/both?
> > >
> > > Thanks,
> > >
> > > Xuyen
> > >
> > >
> >
> >
> >
>
>
>
RE: Hardware Loadbalancer vs zookeeper for producers
Posted by Xuyen On <xo...@ancestry.com>.
Hi Neha,
First I should mention that we are initially using Kafka 0.7, so I don't think we'll have access to the API to get metadata info from the brokers since this is a 0.8 feature.
So my question is if we go with a hardware load balancer, do all the broker nodes have to treated equally? Ie: All broker nodes will have the same topics and number of partitions for each topic?
Or is it possible to have brokers with different topics? If we go with this configuration how can the hardware load balancer direct the messages to the right broker for a given topic? We were thinking that we might be able to do this by having a different VIP's per topic. How do you guys do it?
Lastly, why do you guys use a hardware load balancer vs. using zookeeper? The only reason that I could think of was to offload traffic from the zookeeper cluster. Are there any other advantages to using a hardware load balancer?
Thanks,
Xuyen
-----Original Message-----
From: Neha Narkhede [mailto:neha.narkhede@gmail.com]
Sent: Wednesday, September 11, 2013 5:43 PM
To: users@kafka.apache.org
Subject: Re: Hardware Loadbalancer vs zookeeper for producers
Take a look at the "Producer Load balancing" section of the documentation - http://kafka.apache.org/08/documentation.html#theproducer
Let us know how we can improve it to answer your question.
Thanks,
Neha
On Wed, Sep 11, 2013 at 4:45 PM, Xuyen On <xo...@ancestry.com> wrote:
> Hi Neha,
>
> How do you get metadata about the cluster from the hardware load balancer?
> A load balancer should direct a message to brokers that are up but I
> don't know how you would detect for things like adding/deleting
> topics/partitions from the cluster through the hardware load balancer.
> Can you help me understand how you guys do this?
>
> Thanks,
>
> Xuyen
>
> -----Original Message-----
> From: Neha Narkhede [mailto:neha.narkhede@gmail.com]
> Sent: Wednesday, September 11, 2013 2:39 PM
> To: users@kafka.apache.org
> Subject: Re: Hardware Loadbalancer vs zookeeper for producers
>
> Kafka 08 doesn't use zookeeper based load balancing on the producer side.
> We rely on a hardware load balancer to be able to talk to any broker
> and get metadata about the cluster. This happens on startup, for new
> topics/partitions and if the producer runs into an error while sending data.
>
> Thanks,
> Neha
>
>
> On Wed, Sep 11, 2013 at 12:38 PM, Xuyen On <xo...@ancestry.com> wrote:
>
> > Can anyone from LinkedIn let me know if they are using Hardware
> > Loadbalancer to distribute the producer messages or if they're using
> > zookeeper?
> > I'm trying to find out what the pros and cons are of using a
> > hardware loadbalancer vs. using zookeeper to distribute the load of
> > producers sending messages to brokers. Does anyone have any
> > experience using either/both?
> >
> > Thanks,
> >
> > Xuyen
> >
> >
>
>
>
Re: Hardware Loadbalancer vs zookeeper for producers
Posted by Neha Narkhede <ne...@gmail.com>.
Take a look at the "Producer Load balancing" section of the documentation -
http://kafka.apache.org/08/documentation.html#theproducer
Let us know how we can improve it to answer your question.
Thanks,
Neha
On Wed, Sep 11, 2013 at 4:45 PM, Xuyen On <xo...@ancestry.com> wrote:
> Hi Neha,
>
> How do you get metadata about the cluster from the hardware load balancer?
> A load balancer should direct a message to brokers that are up but I don't
> know how you would detect for things like adding/deleting topics/partitions
> from the cluster through the hardware load balancer. Can you help me
> understand how you guys do this?
>
> Thanks,
>
> Xuyen
>
> -----Original Message-----
> From: Neha Narkhede [mailto:neha.narkhede@gmail.com]
> Sent: Wednesday, September 11, 2013 2:39 PM
> To: users@kafka.apache.org
> Subject: Re: Hardware Loadbalancer vs zookeeper for producers
>
> Kafka 08 doesn't use zookeeper based load balancing on the producer side.
> We rely on a hardware load balancer to be able to talk to any broker and
> get metadata about the cluster. This happens on startup, for new
> topics/partitions and if the producer runs into an error while sending data.
>
> Thanks,
> Neha
>
>
> On Wed, Sep 11, 2013 at 12:38 PM, Xuyen On <xo...@ancestry.com> wrote:
>
> > Can anyone from LinkedIn let me know if they are using Hardware
> > Loadbalancer to distribute the producer messages or if they're using
> > zookeeper?
> > I'm trying to find out what the pros and cons are of using a hardware
> > loadbalancer vs. using zookeeper to distribute the load of producers
> > sending messages to brokers. Does anyone have any experience using
> > either/both?
> >
> > Thanks,
> >
> > Xuyen
> >
> >
>
>
>
RE: Hardware Loadbalancer vs zookeeper for producers
Posted by Xuyen On <xo...@ancestry.com>.
Hi Neha,
How do you get metadata about the cluster from the hardware load balancer? A load balancer should direct a message to brokers that are up but I don't know how you would detect for things like adding/deleting topics/partitions from the cluster through the hardware load balancer. Can you help me understand how you guys do this?
Thanks,
Xuyen
-----Original Message-----
From: Neha Narkhede [mailto:neha.narkhede@gmail.com]
Sent: Wednesday, September 11, 2013 2:39 PM
To: users@kafka.apache.org
Subject: Re: Hardware Loadbalancer vs zookeeper for producers
Kafka 08 doesn't use zookeeper based load balancing on the producer side.
We rely on a hardware load balancer to be able to talk to any broker and get metadata about the cluster. This happens on startup, for new topics/partitions and if the producer runs into an error while sending data.
Thanks,
Neha
On Wed, Sep 11, 2013 at 12:38 PM, Xuyen On <xo...@ancestry.com> wrote:
> Can anyone from LinkedIn let me know if they are using Hardware
> Loadbalancer to distribute the producer messages or if they're using
> zookeeper?
> I'm trying to find out what the pros and cons are of using a hardware
> loadbalancer vs. using zookeeper to distribute the load of producers
> sending messages to brokers. Does anyone have any experience using
> either/both?
>
> Thanks,
>
> Xuyen
>
>
Re: Hardware Loadbalancer vs zookeeper for producers
Posted by Neha Narkhede <ne...@gmail.com>.
Kafka 08 doesn't use zookeeper based load balancing on the producer side.
We rely on a hardware load balancer to be able to talk to any broker and
get metadata about the cluster. This happens on startup, for new
topics/partitions and if the producer runs into an error while sending data.
Thanks,
Neha
On Wed, Sep 11, 2013 at 12:38 PM, Xuyen On <xo...@ancestry.com> wrote:
> Can anyone from LinkedIn let me know if they are using Hardware
> Loadbalancer to distribute the producer messages or if they're using
> zookeeper?
> I'm trying to find out what the pros and cons are of using a hardware
> loadbalancer vs. using zookeeper to distribute the load of producers
> sending messages to brokers. Does anyone have any experience using
> either/both?
>
> Thanks,
>
> Xuyen
>
>