colocating zookeeper on the kafka broker nodes

[Robert Friberg <ro...@devrex.se>]

Hi,

I'm preparing a production grade install of Kafka. The message volume will be very low, peak load in messages/second would be measured in the 100's. Message size is on average 0.3kb. The number of topics will be < 100 and the typical number of consumers per topic 2 or 3.

Reading the docs and other best practices found it seems like a lot of the recommendations are focused on performance. I'm guessing that some practices are more important than others. In particular I'm wondering if it would be suitable to co-locate the zookeepers on the same VM's as I've heard from multiple sources that this is not recommended. My guess is that this is acceptable given the small workload.

Does anyone here disagree or care to elaborate why this would be bad/good?

--
Robert Friberg
Building OrigoDB at Devrex Labs
+46733839080

RE: colocating zookeeper on the kafka broker nodes

[Srinath C <sr...@gmail.com>]

Ok Robert. For the scale you mentioned, is should be fine in my opinion.

Regards,
Srinath

On Jul 27, 2017 6:23 AM, "Robert Friberg" <ro...@devrex.se> wrote:

>
> Thank you Srinath,
>
> I see now that my question was unclear and actually had an incomplete
> sentence...
>
> My idea is to have 3 VM cluster and each VM will run both zookeeper and
> kafka.
>
> --
> Robert Friberg
> Building OrigoDB at Devrex Labs
> +46733839080
>
> -----Original Message-----
> From: Srinath C [mailto:srinath.c@gmail.com]
> Sent: den 26 juli 2017 20:39
> To: users@kafka.apache.org
> Subject: Re: colocating zookeeper on the kafka broker nodes
>
> Hi Robert,
>
> By colocating the zookeeper servers on the same VM we could have a
> downtime if the VM happened to go down. Having the servers on different VMs
> will reduce the risk factor.
>
> For example, if the zookeeper cluster has 5 servers you can tolerate upto
> 2 servers going down and still have no downtime.
>
> Regards,
> Srinath
>
>
> On Jul 27, 2017 5:55 AM, "Robert Friberg" <ro...@devrex.se>
> wrote:
>
> Hi,
>
> I'm preparing a production grade install of Kafka. The message volume will
> be very low, peak load in messages/second would be measured in the 100's.
> Message size is on average 0.3kb. The number of topics will be < 100 and
> the typical number of consumers per topic 2 or 3.
>
> Reading the docs and other best practices found it seems like a lot of the
> recommendations are focused on performance. I'm guessing that some
> practices are more important than others. In particular I'm wondering if it
> would be suitable to co-locate the zookeepers on the same VM's as I've
> heard from multiple sources that this is not recommended. My guess is that
> this is acceptable given the small workload.
>
> Does anyone here disagree or care to elaborate why this would be bad/good?
>
> --
> Robert Friberg
> Building OrigoDB at Devrex Labs
> +46733839080
>

RE: colocating zookeeper on the kafka broker nodes

[Robert Friberg <ro...@devrex.se>]

Thank you Srinath,

I see now that my question was unclear and actually had an incomplete sentence...

My idea is to have 3 VM cluster and each VM will run both zookeeper and kafka.

--
Robert Friberg
Building OrigoDB at Devrex Labs
+46733839080

-----Original Message-----
From: Srinath C [mailto:srinath.c@gmail.com] 
Sent: den 26 juli 2017 20:39
To: users@kafka.apache.org
Subject: Re: colocating zookeeper on the kafka broker nodes

Hi Robert,

By colocating the zookeeper servers on the same VM we could have a downtime if the VM happened to go down. Having the servers on different VMs will reduce the risk factor.

For example, if the zookeeper cluster has 5 servers you can tolerate upto 2 servers going down and still have no downtime.

Regards,
Srinath


On Jul 27, 2017 5:55 AM, "Robert Friberg" <ro...@devrex.se> wrote:

Hi,

I'm preparing a production grade install of Kafka. The message volume will be very low, peak load in messages/second would be measured in the 100's.
Message size is on average 0.3kb. The number of topics will be < 100 and the typical number of consumers per topic 2 or 3.

Reading the docs and other best practices found it seems like a lot of the recommendations are focused on performance. I'm guessing that some practices are more important than others. In particular I'm wondering if it would be suitable to co-locate the zookeepers on the same VM's as I've heard from multiple sources that this is not recommended. My guess is that this is acceptable given the small workload.

Does anyone here disagree or care to elaborate why this would be bad/good?

--
Robert Friberg
Building OrigoDB at Devrex Labs
+46733839080

Re: colocating zookeeper on the kafka broker nodes

[Srinath C <sr...@gmail.com>]

Hi Robert,

By colocating the zookeeper servers on the same VM we could have a downtime
if the VM happened to go down. Having the servers on different VMs will
reduce the risk factor.

For example, if the zookeeper cluster has 5 servers you can tolerate upto 2
servers going down and still have no downtime.

Regards,
Srinath


On Jul 27, 2017 5:55 AM, "Robert Friberg" <ro...@devrex.se> wrote:

Hi,

I'm preparing a production grade install of Kafka. The message volume will
be very low, peak load in messages/second would be measured in the 100's.
Message size is on average 0.3kb. The number of topics will be < 100 and
the typical number of consumers per topic 2 or 3.

Reading the docs and other best practices found it seems like a lot of the
recommendations are focused on performance. I'm guessing that some
practices are more important than others. In particular I'm wondering if it
would be suitable to co-locate the zookeepers on the same VM's as I've
heard from multiple sources that this is not recommended. My guess is that
this is acceptable given the small workload.

Does anyone here disagree or care to elaborate why this would be bad/good?

--
Robert Friberg
Building OrigoDB at Devrex Labs
+46733839080