You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Erik Forsberg <fo...@opera.com> on 2016/01/14 12:06:45 UTC
Kafka + ZooKeeper on the same hardware?
Hi!
Pondering how to configure Kafka clusters and avoid having too many
machines to manage.. Would it be recommended to run say a 3 node kafka
cluster where you also run your 3 node zookeeper cluster on the same
machines?
I guess the answer is that "it depends on load", but would be interested
in any opinions on this anyway.
Thanks!
\EF
Re: Kafka + ZooKeeper on the same hardware?
Posted by James Cheng <jc...@tivo.com>.
> On Jan 18, 2016, at 12:21 PM, Dick Davies <di...@hellooperator.net> wrote:
>
> Started an Ansible playbook using the Confluent platform RPM distro,
> and it seems that co-locates zookeepers
> on the brokers.
>
> So I'm assuming it's fine (at least on 0.9.x for the reasons Todd mentioned).
>
> Does anyone know if the Confluent distro is supposed to be production-ready?
> I switched our test setup over to the Confluent distro to get
> RPM-based installs etc and got a 3-node
> cluster up in HA, but there were a few things I had to roll myself
> (init scripts etc) which I'd have expected
> in a full release.
>
> (sorry if this is OT, I don't know of a specific Confluent mailing list).
>
Confluent-specific mailing list is linked off of here:
http://www.confluent.io/developer#documentation
-James
>
>
> On 14 January 2016 at 18:31, Todd Palino <tp...@gmail.com> wrote:
>> I’d say it depends on load and usage. It can definitely be done, and we’ve
>> done it here in places, though we don’t anymore. Part of the luxury of
>> being able to get the hardware we want. In general, it’s probably easier to
>> do with 0.9 and Kafka-committed offsets, because the consumers don’t need
>> to talk to ZK as much. It’s probably even even easier with the new
>> consumer, but I can’t speak to that at all.
>>
>> One of the gotchas is that ZK really should have its transaction log on an
>> isolate device so that sequential writes do not require seeks. This could
>> be a separate disk or an SSD drive. An example of a really bad place to put
>> it would be on the same device as your Kafka log segments :) Depending on
>> your load, it may not be critical to use a separate device.
>>
>> As Gwen noted, it all comes down to load. Your availability will be fine,
>> you just need to figure out if the services can share the load.
>>
>> -Todd
>>
>>
>> On Thu, Jan 14, 2016 at 9:25 AM, Gwen Shapira <gw...@confluent.io> wrote:
>>
>>> It depends on load :)
>>> As long as there is no contention, you are fine.
>>>
>>> On Thu, Jan 14, 2016 at 6:06 AM, Erik Forsberg <fo...@opera.com> wrote:
>>>
>>>> Hi!
>>>>
>>>> Pondering how to configure Kafka clusters and avoid having too many
>>>> machines to manage.. Would it be recommended to run say a 3 node kafka
>>>> cluster where you also run your 3 node zookeeper cluster on the same
>>>> machines?
>>>>
>>>> I guess the answer is that "it depends on load", but would be interested
>>>> in any opinions on this anyway.
>>>>
>>>> Thanks!
>>>> \EF
>>>>
>>>
>>
>>
>>
>> --
>> *—-*
>> *Todd Palino*
>> Staff Site Reliability Engineer
>> Data Infrastructure Streaming
>>
>>
>>
>> linkedin.com/in/toddpalino
________________________________
This email and any attachments may contain confidential and privileged material for the sole use of the intended recipient. Any review, copying, or distribution of this email (or any attachments) by others is prohibited. If you are not the intended recipient, please contact the sender immediately and permanently delete this email and any attachments. No employee or agent of TiVo Inc. is authorized to conclude any binding agreement on behalf of TiVo Inc. by email. Binding agreements with TiVo Inc. may only be made by a signed written agreement.
Re: Kafka + ZooKeeper on the same hardware?
Posted by Dick Davies <di...@hellooperator.net>.
Started an Ansible playbook using the Confluent platform RPM distro,
and it seems that co-locates zookeepers
on the brokers.
So I'm assuming it's fine (at least on 0.9.x for the reasons Todd mentioned).
Does anyone know if the Confluent distro is supposed to be production-ready?
I switched our test setup over to the Confluent distro to get
RPM-based installs etc and got a 3-node
cluster up in HA, but there were a few things I had to roll myself
(init scripts etc) which I'd have expected
in a full release.
(sorry if this is OT, I don't know of a specific Confluent mailing list).
On 14 January 2016 at 18:31, Todd Palino <tp...@gmail.com> wrote:
> I’d say it depends on load and usage. It can definitely be done, and we’ve
> done it here in places, though we don’t anymore. Part of the luxury of
> being able to get the hardware we want. In general, it’s probably easier to
> do with 0.9 and Kafka-committed offsets, because the consumers don’t need
> to talk to ZK as much. It’s probably even even easier with the new
> consumer, but I can’t speak to that at all.
>
> One of the gotchas is that ZK really should have its transaction log on an
> isolate device so that sequential writes do not require seeks. This could
> be a separate disk or an SSD drive. An example of a really bad place to put
> it would be on the same device as your Kafka log segments :) Depending on
> your load, it may not be critical to use a separate device.
>
> As Gwen noted, it all comes down to load. Your availability will be fine,
> you just need to figure out if the services can share the load.
>
> -Todd
>
>
> On Thu, Jan 14, 2016 at 9:25 AM, Gwen Shapira <gw...@confluent.io> wrote:
>
>> It depends on load :)
>> As long as there is no contention, you are fine.
>>
>> On Thu, Jan 14, 2016 at 6:06 AM, Erik Forsberg <fo...@opera.com> wrote:
>>
>> > Hi!
>> >
>> > Pondering how to configure Kafka clusters and avoid having too many
>> > machines to manage.. Would it be recommended to run say a 3 node kafka
>> > cluster where you also run your 3 node zookeeper cluster on the same
>> > machines?
>> >
>> > I guess the answer is that "it depends on load", but would be interested
>> > in any opinions on this anyway.
>> >
>> > Thanks!
>> > \EF
>> >
>>
>
>
>
> --
> *—-*
> *Todd Palino*
> Staff Site Reliability Engineer
> Data Infrastructure Streaming
>
>
>
> linkedin.com/in/toddpalino
Re: Kafka + ZooKeeper on the same hardware?
Posted by Todd Palino <tp...@gmail.com>.
I’d say it depends on load and usage. It can definitely be done, and we’ve
done it here in places, though we don’t anymore. Part of the luxury of
being able to get the hardware we want. In general, it’s probably easier to
do with 0.9 and Kafka-committed offsets, because the consumers don’t need
to talk to ZK as much. It’s probably even even easier with the new
consumer, but I can’t speak to that at all.
One of the gotchas is that ZK really should have its transaction log on an
isolate device so that sequential writes do not require seeks. This could
be a separate disk or an SSD drive. An example of a really bad place to put
it would be on the same device as your Kafka log segments :) Depending on
your load, it may not be critical to use a separate device.
As Gwen noted, it all comes down to load. Your availability will be fine,
you just need to figure out if the services can share the load.
-Todd
On Thu, Jan 14, 2016 at 9:25 AM, Gwen Shapira <gw...@confluent.io> wrote:
> It depends on load :)
> As long as there is no contention, you are fine.
>
> On Thu, Jan 14, 2016 at 6:06 AM, Erik Forsberg <fo...@opera.com> wrote:
>
> > Hi!
> >
> > Pondering how to configure Kafka clusters and avoid having too many
> > machines to manage.. Would it be recommended to run say a 3 node kafka
> > cluster where you also run your 3 node zookeeper cluster on the same
> > machines?
> >
> > I guess the answer is that "it depends on load", but would be interested
> > in any opinions on this anyway.
> >
> > Thanks!
> > \EF
> >
>
--
*—-*
*Todd Palino*
Staff Site Reliability Engineer
Data Infrastructure Streaming
linkedin.com/in/toddpalino
Re: Kafka + ZooKeeper on the same hardware?
Posted by Gwen Shapira <gw...@confluent.io>.
It depends on load :)
As long as there is no contention, you are fine.
On Thu, Jan 14, 2016 at 6:06 AM, Erik Forsberg <fo...@opera.com> wrote:
> Hi!
>
> Pondering how to configure Kafka clusters and avoid having too many
> machines to manage.. Would it be recommended to run say a 3 node kafka
> cluster where you also run your 3 node zookeeper cluster on the same
> machines?
>
> I guess the answer is that "it depends on load", but would be interested
> in any opinions on this anyway.
>
> Thanks!
> \EF
>
Re: Kafka + ZooKeeper on the same hardware?
Posted by Kamal C <ka...@gmail.com>.
Yes, it can sustain one failure. Misunderstood your question..
On Thu, Jan 14, 2016 at 5:14 PM, Erik Forsberg <fo...@opera.com> wrote:
>
>
> On 2016-01-14 12:42, Kamal C wrote:
>
>> It's a single point of failure. You may lose high-availability.
>>
>
> In this case I would like to protect myself from 1 machine going down, and
> my replication factor for Kafka would be 2. So in the case of one machine
> going down, Zookeeper cluster would still be operational, and Kafka would
> as well if I understand things correctly.
>
> Regards,
> \EF
>
>
Re: Kafka + ZooKeeper on the same hardware?
Posted by Erik Forsberg <fo...@opera.com>.
On 2016-01-14 12:42, Kamal C wrote:
> It's a single point of failure. You may lose high-availability.
In this case I would like to protect myself from 1 machine going down,
and my replication factor for Kafka would be 2. So in the case of one
machine going down, Zookeeper cluster would still be operational, and
Kafka would as well if I understand things correctly.
Regards,
\EF
Re: Kafka + ZooKeeper on the same hardware?
Posted by Kamal C <ka...@gmail.com>.
It's a single point of failure. You may lose high-availability.
On Thu, Jan 14, 2016 at 4:36 PM, Erik Forsberg <fo...@opera.com> wrote:
> Hi!
>
> Pondering how to configure Kafka clusters and avoid having too many
> machines to manage.. Would it be recommended to run say a 3 node kafka
> cluster where you also run your 3 node zookeeper cluster on the same
> machines?
>
> I guess the answer is that "it depends on load", but would be interested
> in any opinions on this anyway.
>
> Thanks!
> \EF
>