You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Meghana Narasimhan <mn...@bandwidth.com> on 2017/06/02 18:56:20 UTC
Zookeeper on same server as Kafka
Hi,
What are the pros and cons of setting up Zookeeper on the same server as
the Kafka broker ? Earlier offsets were being written to zookeeper which
was a major overhead but with offsets being written to Kafka now, what
other requirements should be taken into consideration for setting up
Zookeeper on the same server as Kafka vs having a separate zookeeper
cluster ?
Thanks,
Meghana
Re: Zookeeper on same server as Kafka
Posted by Meghana Narasimhan <mn...@bandwidth.com>.
Thank you all for your inputs, so going by the opinions I guess its a lot
to do with use case and how the clusters will evolve over time. Will keep
these in mind. Thanks !
On Sun, Jun 4, 2017 at 10:13 AM, Michal Borowiecki <
michal.borowiecki@openbet.com> wrote:
> We are indeed running this setup in production, have been for almost 2
> years now, over a gradually increasing number of deployments.
>
> Let me clarify though:
>
> Our clusters don't exceed 5 nodes. We're not exercising Kafka nowhere near
> its limits, or bandwidth or disk I/O for that matter.
>
> When we reach the point we want to scale Kafka independently from ZK, we
> will. The current setup is not set in stone. But for now everything is
> running under modest load, so don't see the point of forgoing the
> simplicity of this until we actually have a need to scale further or put
> greater load on it. YMMV, not everyone is running at the same scale as
> Heroku ;-)
>
> Thanks,
>
> Michal
>
> On 04/06/17 11:34, Tom Crayford wrote:
>
> Hi,
>
> I would not recommend running this kind of set up in production. Busy
> Kafka brokers use up a lot of disk and network bandwidth, which zookeeper
> does not deal well with. This means that a burst of traffic to 1 node
> carries the risk of disrupting the ZK ensemble.
>
> Secondly, this will cause problems down the line, because you will want to
> scale Kafka independently from ZK. ZK gets slower as you add nodes, but
> Kafka can scale out for quite a while. For production clusters, I'd
> recommend to always have 5 ZK nodes, but for Kafka, you can scale well past
> that, or keep it small while you are starting out.
>
> Thanks,
>
> Tom Crayford
> Heroku Kafka
>
> On Sat, Jun 3, 2017 at 8:20 AM, Michal Borowiecki <
> michal.borowiecki@openbet.com> wrote:
>
>> I'm not an expert but I prefer keeping zookeepers on the same hosts as
>> kafka brokers and mimic each-others topology. The reason is to minimize the
>> chance of e.g. kafka brokers being able to talk to one another but
>> zookeepers not, or vice-versa. So, I'd say I *do* want my kafka broker
>> and the co-located zookeeper to go down together - for simplicity - prefer
>> that to some asymmetric failures to debug. This comes from past experience,
>> albeit using other technologies, when relying on 2 different clustering
>> mechanism made failures in one but not the other very difficult to debug.
>>
>> Also, I think I read this advice somewhere a long time ago (don't recall
>> where) and it made sense to me (given the prior experience) and we've never
>> tried a different arrangement.
>>
>> As to the overheads, I believe it's mostly disk IO and can hopefully be
>> addressed by separate disks for each but it's never been a bottleneck for
>> us, so can't really say.
>>
>> Thanks,
>>
>> Michał
>>
>> On 02/06/17 21:47, Mohammed Manna wrote:
>>
>> Usually, the overhead comes when you have kafka and zookeeper doing the
>> housekeeping (i.e. Disk IO) on the same server. ZK even suggests that you
>> should keep their logs on totally different physical machine for better
>> performance. Furthermore, if a mechanical failure occurs, you might not
>> want both zookeeper and broker going down together.
>>
>> Can anyone else chime in for some better points?
>>
>>
>> On 2 Jun 2017 7:57 pm, "Meghana Narasimhan" <mn...@bandwidth.com> <mn...@bandwidth.com>
>> wrote:
>>
>> Hi,
>> What are the pros and cons of setting up Zookeeper on the same server as
>> the Kafka broker ? Earlier offsets were being written to zookeeper which
>> was a major overhead but with offsets being written to Kafka now, what
>> other requirements should be taken into consideration for setting up
>> Zookeeper on the same server as Kafka vs having a separate zookeeper
>> cluster ?
>>
>> Thanks,
>> Meghana
>>
>>
>>
>> --
>> <http://www.openbet.com/> Michal Borowiecki
>> Senior Software Engineer L4
>> T: +44 208 742 1600 <+44%2020%208742%201600>
>>
>>
>> +44 203 249 8448 <+44%2020%203249%208448>
>>
>>
>>
>> E: michal.borowiecki@openbet.com
>> W: www.openbet.com
>> OpenBet Ltd
>>
>> Chiswick Park Building 9
>>
>> 566 Chiswick High Rd
>>
>> London
>>
>> W4 5XT
>>
>> UK
>> <https://www.openbet.com/email_promo>
>> This message is confidential and intended only for the addressee. If you
>> have received this message in error, please immediately notify the
>> postmaster@openbet.com and delete it from your system as well as any
>> copies. The content of e-mails as well as traffic data may be monitored by
>> OpenBet for employment and security purposes. To protect the environment
>> please do not print this e-mail unless necessary. OpenBet Ltd. Registered
>> Office: Chiswick Park Building 9, 566 Chiswick High Road, London, W4 5XT,
>> United Kingdom. A company registered in England and Wales. Registered no.
>> 3134634. VAT no. GB927523612
>>
>
>
> --
> <http://www.openbet.com/> Michal Borowiecki
> Senior Software Engineer L4
> T: +44 208 742 1600 <+44%2020%208742%201600>
>
>
> +44 203 249 8448 <+44%2020%203249%208448>
>
>
>
> E: michal.borowiecki@openbet.com
> W: www.openbet.com
> OpenBet Ltd
>
> Chiswick Park Building 9
>
> 566 Chiswick High Rd
>
> London
>
> W4 5XT
>
> UK
> <https://www.openbet.com/email_promo>
> This message is confidential and intended only for the addressee. If you
> have received this message in error, please immediately notify the
> postmaster@openbet.com and delete it from your system as well as any
> copies. The content of e-mails as well as traffic data may be monitored by
> OpenBet for employment and security purposes. To protect the environment
> please do not print this e-mail unless necessary. OpenBet Ltd. Registered
> Office: Chiswick Park Building 9, 566 Chiswick High Road, London, W4 5XT,
> United Kingdom. A company registered in England and Wales. Registered no.
> 3134634. VAT no. GB927523612
>
Re: Zookeeper on same server as Kafka
Posted by Michal Borowiecki <mi...@openbet.com>.
We are indeed running this setup in production, have been for almost 2
years now, over a gradually increasing number of deployments.
Let me clarify though:
Our clusters don't exceed 5 nodes. We're not exercising Kafka nowhere
near its limits, or bandwidth or disk I/O for that matter.
When we reach the point we want to scale Kafka independently from ZK, we
will. The current setup is not set in stone. But for now everything is
running under modest load, so don't see the point of forgoing the
simplicity of this until we actually have a need to scale further or put
greater load on it. YMMV, not everyone is running at the same scale as
Heroku ;-)
Thanks,
Michal
On 04/06/17 11:34, Tom Crayford wrote:
> Hi,
>
> I would not recommend running this kind of set up in production. Busy
> Kafka brokers use up a lot of disk and network bandwidth, which
> zookeeper does not deal well with. This means that a burst of traffic
> to 1 node carries the risk of disrupting the ZK ensemble.
>
> Secondly, this will cause problems down the line, because you will
> want to scale Kafka independently from ZK. ZK gets slower as you add
> nodes, but Kafka can scale out for quite a while. For production
> clusters, I'd recommend to always have 5 ZK nodes, but for Kafka, you
> can scale well past that, or keep it small while you are starting out.
>
> Thanks,
>
> Tom Crayford
> Heroku Kafka
>
> On Sat, Jun 3, 2017 at 8:20 AM, Michal Borowiecki
> <michal.borowiecki@openbet.com <ma...@openbet.com>>
> wrote:
>
> I'm not an expert but I prefer keeping zookeepers on the same
> hosts as kafka brokers and mimic each-others topology. The reason
> is to minimize the chance of e.g. kafka brokers being able to talk
> to one another but zookeepers not, or vice-versa. So, I'd say I
> /do/ want my kafka broker and the co-located zookeeper to go down
> together - for simplicity - prefer that to some asymmetric
> failures to debug. This comes from past experience, albeit using
> other technologies, when relying on 2 different clustering
> mechanism made failures in one but not the other very difficult to
> debug.
>
> Also, I think I read this advice somewhere a long time ago (don't
> recall where) and it made sense to me (given the prior experience)
> and we've never tried a different arrangement.
>
> As to the overheads, I believe it's mostly disk IO and can
> hopefully be addressed by separate disks for each but it's never
> been a bottleneck for us, so can't really say.
>
> Thanks,
>
> Michał
>
>
> On 02/06/17 21:47, Mohammed Manna wrote:
>> Usually, the overhead comes when you have kafka and zookeeper doing the
>> housekeeping (i.e. Disk IO) on the same server. ZK even suggests that you
>> should keep their logs on totally different physical machine for better
>> performance. Furthermore, if a mechanical failure occurs, you might not
>> want both zookeeper and broker going down together.
>>
>> Can anyone else chime in for some better points?
>>
>>
>> On 2 Jun 2017 7:57 pm, "Meghana Narasimhan"<mn...@bandwidth.com> <ma...@bandwidth.com>
>> wrote:
>>
>> Hi,
>> What are the pros and cons of setting up Zookeeper on the same server as
>> the Kafka broker ? Earlier offsets were being written to zookeeper which
>> was a major overhead but with offsets being written to Kafka now, what
>> other requirements should be taken into consideration for setting up
>> Zookeeper on the same server as Kafka vs having a separate zookeeper
>> cluster ?
>>
>> Thanks,
>> Meghana
>>
>
> --
> <http://www.openbet.com/> Michal Borowiecki
> Senior Software Engineer L4
> T: +44 208 742 1600 <tel:+44%2020%208742%201600>
>
>
> +44 203 249 8448 <tel:+44%2020%203249%208448>
>
>
>
> E: michal.borowiecki@openbet.com
> <ma...@openbet.com>
> W: www.openbet.com <http://www.openbet.com/>
>
>
> OpenBet Ltd
>
> Chiswick Park Building 9
>
> 566 Chiswick High Rd
>
> London
>
> W4 5XT
>
> UK
>
>
> <https://www.openbet.com/email_promo>
>
> This message is confidential and intended only for the addressee.
> If you have received this message in error, please immediately
> notify the postmaster@openbet.com <ma...@openbet.com>
> and delete it from your system as well as any copies. The content
> of e-mails as well as traffic data may be monitored by OpenBet for
> employment and security purposes. To protect the environment
> please do not print this e-mail unless necessary. OpenBet Ltd.
> Registered Office: Chiswick Park Building 9, 566 Chiswick High
> Road, London, W4 5XT, United Kingdom. A company registered in
> England and Wales. Registered no. 3134634. VAT no. GB927523612
>
>
--
Signature
<http://www.openbet.com/> Michal Borowiecki
Senior Software Engineer L4
T: +44 208 742 1600
+44 203 249 8448
E: michal.borowiecki@openbet.com
W: www.openbet.com <http://www.openbet.com/>
OpenBet Ltd
Chiswick Park Building 9
566 Chiswick High Rd
London
W4 5XT
UK
<https://www.openbet.com/email_promo>
This message is confidential and intended only for the addressee. If you
have received this message in error, please immediately notify the
postmaster@openbet.com <ma...@openbet.com> and delete it
from your system as well as any copies. The content of e-mails as well
as traffic data may be monitored by OpenBet for employment and security
purposes. To protect the environment please do not print this e-mail
unless necessary. OpenBet Ltd. Registered Office: Chiswick Park Building
9, 566 Chiswick High Road, London, W4 5XT, United Kingdom. A company
registered in England and Wales. Registered no. 3134634. VAT no.
GB927523612
Re: Zookeeper on same server as Kafka
Posted by Tom Crayford <tc...@heroku.com>.
Hi,
I would not recommend running this kind of set up in production. Busy Kafka
brokers use up a lot of disk and network bandwidth, which zookeeper does
not deal well with. This means that a burst of traffic to 1 node carries
the risk of disrupting the ZK ensemble.
Secondly, this will cause problems down the line, because you will want to
scale Kafka independently from ZK. ZK gets slower as you add nodes, but
Kafka can scale out for quite a while. For production clusters, I'd
recommend to always have 5 ZK nodes, but for Kafka, you can scale well past
that, or keep it small while you are starting out.
Thanks,
Tom Crayford
Heroku Kafka
On Sat, Jun 3, 2017 at 8:20 AM, Michal Borowiecki <
michal.borowiecki@openbet.com> wrote:
> I'm not an expert but I prefer keeping zookeepers on the same hosts as
> kafka brokers and mimic each-others topology. The reason is to minimize the
> chance of e.g. kafka brokers being able to talk to one another but
> zookeepers not, or vice-versa. So, I'd say I *do* want my kafka broker
> and the co-located zookeeper to go down together - for simplicity - prefer
> that to some asymmetric failures to debug. This comes from past experience,
> albeit using other technologies, when relying on 2 different clustering
> mechanism made failures in one but not the other very difficult to debug.
>
> Also, I think I read this advice somewhere a long time ago (don't recall
> where) and it made sense to me (given the prior experience) and we've never
> tried a different arrangement.
>
> As to the overheads, I believe it's mostly disk IO and can hopefully be
> addressed by separate disks for each but it's never been a bottleneck for
> us, so can't really say.
>
> Thanks,
>
> Michał
>
> On 02/06/17 21:47, Mohammed Manna wrote:
>
> Usually, the overhead comes when you have kafka and zookeeper doing the
> housekeeping (i.e. Disk IO) on the same server. ZK even suggests that you
> should keep their logs on totally different physical machine for better
> performance. Furthermore, if a mechanical failure occurs, you might not
> want both zookeeper and broker going down together.
>
> Can anyone else chime in for some better points?
>
>
> On 2 Jun 2017 7:57 pm, "Meghana Narasimhan" <mn...@bandwidth.com> <mn...@bandwidth.com>
> wrote:
>
> Hi,
> What are the pros and cons of setting up Zookeeper on the same server as
> the Kafka broker ? Earlier offsets were being written to zookeeper which
> was a major overhead but with offsets being written to Kafka now, what
> other requirements should be taken into consideration for setting up
> Zookeeper on the same server as Kafka vs having a separate zookeeper
> cluster ?
>
> Thanks,
> Meghana
>
>
>
> --
> <http://www.openbet.com/> Michal Borowiecki
> Senior Software Engineer L4
> T: +44 208 742 1600 <+44%2020%208742%201600>
>
>
> +44 203 249 8448 <+44%2020%203249%208448>
>
>
>
> E: michal.borowiecki@openbet.com
> W: www.openbet.com
> OpenBet Ltd
>
> Chiswick Park Building 9
>
> 566 Chiswick High Rd
>
> London
>
> W4 5XT
>
> UK
> <https://www.openbet.com/email_promo>
> This message is confidential and intended only for the addressee. If you
> have received this message in error, please immediately notify the
> postmaster@openbet.com and delete it from your system as well as any
> copies. The content of e-mails as well as traffic data may be monitored by
> OpenBet for employment and security purposes. To protect the environment
> please do not print this e-mail unless necessary. OpenBet Ltd. Registered
> Office: Chiswick Park Building 9, 566 Chiswick High Road, London, W4 5XT,
> United Kingdom. A company registered in England and Wales. Registered no.
> 3134634. VAT no. GB927523612
>
Re: Zookeeper on same server as Kafka
Posted by Michal Borowiecki <mi...@openbet.com>.
I'm not an expert but I prefer keeping zookeepers on the same hosts as
kafka brokers and mimic each-others topology. The reason is to minimize
the chance of e.g. kafka brokers being able to talk to one another but
zookeepers not, or vice-versa. So, I'd say I /do/ want my kafka broker
and the co-located zookeeper to go down together - for simplicity -
prefer that to some asymmetric failures to debug. This comes from past
experience, albeit using other technologies, when relying on 2 different
clustering mechanism made failures in one but not the other very
difficult to debug.
Also, I think I read this advice somewhere a long time ago (don't recall
where) and it made sense to me (given the prior experience) and we've
never tried a different arrangement.
As to the overheads, I believe it's mostly disk IO and can hopefully be
addressed by separate disks for each but it's never been a bottleneck
for us, so can't really say.
Thanks,
Michał
On 02/06/17 21:47, Mohammed Manna wrote:
> Usually, the overhead comes when you have kafka and zookeeper doing the
> housekeeping (i.e. Disk IO) on the same server. ZK even suggests that you
> should keep their logs on totally different physical machine for better
> performance. Furthermore, if a mechanical failure occurs, you might not
> want both zookeeper and broker going down together.
>
> Can anyone else chime in for some better points?
>
>
> On 2 Jun 2017 7:57 pm, "Meghana Narasimhan" <mn...@bandwidth.com>
> wrote:
>
> Hi,
> What are the pros and cons of setting up Zookeeper on the same server as
> the Kafka broker ? Earlier offsets were being written to zookeeper which
> was a major overhead but with offsets being written to Kafka now, what
> other requirements should be taken into consideration for setting up
> Zookeeper on the same server as Kafka vs having a separate zookeeper
> cluster ?
>
> Thanks,
> Meghana
>
--
Signature
<http://www.openbet.com/> Michal Borowiecki
Senior Software Engineer L4
T: +44 208 742 1600
+44 203 249 8448
E: michal.borowiecki@openbet.com
W: www.openbet.com <http://www.openbet.com/>
OpenBet Ltd
Chiswick Park Building 9
566 Chiswick High Rd
London
W4 5XT
UK
<https://www.openbet.com/email_promo>
This message is confidential and intended only for the addressee. If you
have received this message in error, please immediately notify the
postmaster@openbet.com <ma...@openbet.com> and delete it
from your system as well as any copies. The content of e-mails as well
as traffic data may be monitored by OpenBet for employment and security
purposes. To protect the environment please do not print this e-mail
unless necessary. OpenBet Ltd. Registered Office: Chiswick Park Building
9, 566 Chiswick High Road, London, W4 5XT, United Kingdom. A company
registered in England and Wales. Registered no. 3134634. VAT no.
GB927523612
Re: Zookeeper on same server as Kafka
Posted by Mohammed Manna <ma...@gmail.com>.
Usually, the overhead comes when you have kafka and zookeeper doing the
housekeeping (i.e. Disk IO) on the same server. ZK even suggests that you
should keep their logs on totally different physical machine for better
performance. Furthermore, if a mechanical failure occurs, you might not
want both zookeeper and broker going down together.
Can anyone else chime in for some better points?
On 2 Jun 2017 7:57 pm, "Meghana Narasimhan" <mn...@bandwidth.com>
wrote:
Hi,
What are the pros and cons of setting up Zookeeper on the same server as
the Kafka broker ? Earlier offsets were being written to zookeeper which
was a major overhead but with offsets being written to Kafka now, what
other requirements should be taken into consideration for setting up
Zookeeper on the same server as Kafka vs having a separate zookeeper
cluster ?
Thanks,
Meghana