ZooKeeper performance

[Dima Gutzeit <di...@mailvision.com>]

Hi,

I have some performance related question.

I am running a cluster of 5 zookeeper machines, each one has dual quad
core Xeon 2.5 Ghz, 8 gb RAM.

I want achieve the following numbers:

10 clients producing in total (at peaks):

3K add nodes, 3K delete nodes and 10K watches. Per second.

Does those numbers make any sense ?

Thanks in advance.

Regards,
Dima Gutzeit.

Sent from my iPhone

Re: ZooKeeper performance

[Ted Dunning <te...@gmail.com>]

You can do that, but you don't have very many clients so sharding won't take
you very far.

But don't let me rain on your parade.  ZK is awesome and can be pushed to
pretty incredible lengths.

On Mon, Oct 3, 2011 at 11:29 AM, Dima Gutzeit
<di...@mailvision.com>wrote:

> Multiple zookeeper clusters with data partitioning (sharding) ?
>
> Regards,
> Dima Gutzeit.
>
> Sent from my iPhone
>
> On 3 באוק 2011, at 20:25, Ted Dunning <te...@gmail.com> wrote:
>
> > It definitely can be done, but what if your application grows by 10x?
> >
> > On Mon, Oct 3, 2011 at 10:56 AM, Dima Gutzeit
> > <di...@mailvision.com>wrote:
> >
> >> Very nice analysis at the link, thanks.
> >>
> >> Assuming my nodes are around the same size ~100 bytes, ensemble of 5
> >> servers and 10 clients can easily support the required ~20,000
> >> operations per second, no ?
> >>
> >> I will run a smoke-test to prove it ofcourse ...
> >>
> >> Regards,
> >> Dima Gutzeit.
> >>
> >> Sent from my iPhone
> >>
> >> On 3 באוק 2011, at 19:41, "Fournier, Camille F."
> >> <Ca...@gs.com> wrote:
> >>
> >>> It's pretty easy to set up a zk-smoketest to simulate what you are
> doing.
> >> We can't answer this question without knowing how big the data you're
> >> writing, etc etc. I would recommend testing it out yourself on realistic
> >> data sizes.
> >>>
> >>>
> >>
> https://cwiki.apache.org/confluence/display/ZOOKEEPER/ServiceLatencyOverview
> >>>
> >>> -----Original Message-----
> >>> From: Dima Gutzeit [mailto:dima.gutzeit@mailvision.com]
> >>> Sent: Monday, October 03, 2011 1:36 PM
> >>> To: user@zookeeper.apache.org
> >>> Subject: Re: ZooKeeper performance
> >>>
> >>> Inline.
> >>>
> >>> Regards,
> >>> Dima Gutzeit.
> >>>
> >>> Sent from my iPhone
> >>>
> >>> On 3 באוק 2011, at 19:31, Ted Dunning <te...@gmail.com> wrote:
> >>>
> >>>> Questions in-line
> >>>>
> >>>> On Mon, Oct 3, 2011 at 10:19 AM, Dima Gutzeit
> >>>> <di...@mailvision.com>wrote:
> >>>>
> >>>>> Hi,
> >>>>>
> >>>>> I have some performance related question.
> >>>>>
> >>>>> I am running a cluster of 5 zookeeper machines, each one has dual
> quad
> >>>>> core Xeon 2.5 Ghz, 8 gb RAM.
> >>>>>
> >>>>> I want achieve the following numbers:
> >>>>>
> >>>>> 10 clients producing in total (at peaks):
> >>>>>
> >>>>
> >>>> Each producing this?
> >>>
> >>> Total.
> >>>>
> >>>>
> >>>>> 3K add nodes, 3K delete nodes and 10K watches. Per second.
> >>>>>
> >>>>
> >>>> What do you mean by 10K watches?  10,000 watch notifications?
> >>>
> >>> Yes.
> >>>>
> >>>> Does those numbers make any sense ?
> >>>>>
> >>>>
> >>>> This is a bit high.  It sounds like you are trying to use ZK as a
> >> message
> >>>> bus rather than a coordination service.  You can do this, but the
> >> throughput
> >>>> you can achieve is limited.  If you want higher throughput, it is
> better
> >> to
> >>>> have ZK coordinate a higher performance messaging system such as
> Kafka.
> >>
>

Re: ZooKeeper performance

[Dima Gutzeit <di...@mailvision.com>]

Multiple zookeeper clusters with data partitioning (sharding) ?

Regards,
Dima Gutzeit.

Sent from my iPhone

On 3 באוק 2011, at 20:25, Ted Dunning <te...@gmail.com> wrote:

> It definitely can be done, but what if your application grows by 10x?
>
> On Mon, Oct 3, 2011 at 10:56 AM, Dima Gutzeit
> <di...@mailvision.com>wrote:
>
>> Very nice analysis at the link, thanks.
>>
>> Assuming my nodes are around the same size ~100 bytes, ensemble of 5
>> servers and 10 clients can easily support the required ~20,000
>> operations per second, no ?
>>
>> I will run a smoke-test to prove it ofcourse ...
>>
>> Regards,
>> Dima Gutzeit.
>>
>> Sent from my iPhone
>>
>> On 3 באוק 2011, at 19:41, "Fournier, Camille F."
>> <Ca...@gs.com> wrote:
>>
>>> It's pretty easy to set up a zk-smoketest to simulate what you are doing.
>> We can't answer this question without knowing how big the data you're
>> writing, etc etc. I would recommend testing it out yourself on realistic
>> data sizes.
>>>
>>>
>> https://cwiki.apache.org/confluence/display/ZOOKEEPER/ServiceLatencyOverview
>>>
>>> -----Original Message-----
>>> From: Dima Gutzeit [mailto:dima.gutzeit@mailvision.com]
>>> Sent: Monday, October 03, 2011 1:36 PM
>>> To: user@zookeeper.apache.org
>>> Subject: Re: ZooKeeper performance
>>>
>>> Inline.
>>>
>>> Regards,
>>> Dima Gutzeit.
>>>
>>> Sent from my iPhone
>>>
>>> On 3 באוק 2011, at 19:31, Ted Dunning <te...@gmail.com> wrote:
>>>
>>>> Questions in-line
>>>>
>>>> On Mon, Oct 3, 2011 at 10:19 AM, Dima Gutzeit
>>>> <di...@mailvision.com>wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I have some performance related question.
>>>>>
>>>>> I am running a cluster of 5 zookeeper machines, each one has dual quad
>>>>> core Xeon 2.5 Ghz, 8 gb RAM.
>>>>>
>>>>> I want achieve the following numbers:
>>>>>
>>>>> 10 clients producing in total (at peaks):
>>>>>
>>>>
>>>> Each producing this?
>>>
>>> Total.
>>>>
>>>>
>>>>> 3K add nodes, 3K delete nodes and 10K watches. Per second.
>>>>>
>>>>
>>>> What do you mean by 10K watches?  10,000 watch notifications?
>>>
>>> Yes.
>>>>
>>>> Does those numbers make any sense ?
>>>>>
>>>>
>>>> This is a bit high.  It sounds like you are trying to use ZK as a
>> message
>>>> bus rather than a coordination service.  You can do this, but the
>> throughput
>>>> you can achieve is limited.  If you want higher throughput, it is better
>> to
>>>> have ZK coordinate a higher performance messaging system such as Kafka.
>>

Re: ZooKeeper performance

[Ted Dunning <te...@gmail.com>]

It definitely can be done, but what if your application grows by 10x?

On Mon, Oct 3, 2011 at 10:56 AM, Dima Gutzeit
<di...@mailvision.com>wrote:

> Very nice analysis at the link, thanks.
>
> Assuming my nodes are around the same size ~100 bytes, ensemble of 5
> servers and 10 clients can easily support the required ~20,000
> operations per second, no ?
>
> I will run a smoke-test to prove it ofcourse ...
>
> Regards,
> Dima Gutzeit.
>
> Sent from my iPhone
>
> On 3 באוק 2011, at 19:41, "Fournier, Camille F."
> <Ca...@gs.com> wrote:
>
> > It's pretty easy to set up a zk-smoketest to simulate what you are doing.
> We can't answer this question without knowing how big the data you're
> writing, etc etc. I would recommend testing it out yourself on realistic
> data sizes.
> >
> >
> https://cwiki.apache.org/confluence/display/ZOOKEEPER/ServiceLatencyOverview
> >
> > -----Original Message-----
> > From: Dima Gutzeit [mailto:dima.gutzeit@mailvision.com]
> > Sent: Monday, October 03, 2011 1:36 PM
> > To: user@zookeeper.apache.org
> > Subject: Re: ZooKeeper performance
> >
> > Inline.
> >
> > Regards,
> > Dima Gutzeit.
> >
> > Sent from my iPhone
> >
> > On 3 באוק 2011, at 19:31, Ted Dunning <te...@gmail.com> wrote:
> >
> >> Questions in-line
> >>
> >> On Mon, Oct 3, 2011 at 10:19 AM, Dima Gutzeit
> >> <di...@mailvision.com>wrote:
> >>
> >>> Hi,
> >>>
> >>> I have some performance related question.
> >>>
> >>> I am running a cluster of 5 zookeeper machines, each one has dual quad
> >>> core Xeon 2.5 Ghz, 8 gb RAM.
> >>>
> >>> I want achieve the following numbers:
> >>>
> >>> 10 clients producing in total (at peaks):
> >>>
> >>
> >> Each producing this?
> >
> > Total.
> >>
> >>
> >>> 3K add nodes, 3K delete nodes and 10K watches. Per second.
> >>>
> >>
> >> What do you mean by 10K watches?  10,000 watch notifications?
> >
> > Yes.
> >>
> >> Does those numbers make any sense ?
> >>>
> >>
> >> This is a bit high.  It sounds like you are trying to use ZK as a
> message
> >> bus rather than a coordination service.  You can do this, but the
> throughput
> >> you can achieve is limited.  If you want higher throughput, it is better
> to
> >> have ZK coordinate a higher performance messaging system such as Kafka.
>

Re: ZooKeeper performance

[Dima Gutzeit <di...@mailvision.com>]

Very nice analysis at the link, thanks.

Assuming my nodes are around the same size ~100 bytes, ensemble of 5
servers and 10 clients can easily support the required ~20,000
operations per second, no ?

I will run a smoke-test to prove it ofcourse ...

Regards,
Dima Gutzeit.

Sent from my iPhone

On 3 באוק 2011, at 19:41, "Fournier, Camille F."
<Ca...@gs.com> wrote:

> It's pretty easy to set up a zk-smoketest to simulate what you are doing. We can't answer this question without knowing how big the data you're writing, etc etc. I would recommend testing it out yourself on realistic data sizes.
>
> https://cwiki.apache.org/confluence/display/ZOOKEEPER/ServiceLatencyOverview
>
> -----Original Message-----
> From: Dima Gutzeit [mailto:dima.gutzeit@mailvision.com]
> Sent: Monday, October 03, 2011 1:36 PM
> To: user@zookeeper.apache.org
> Subject: Re: ZooKeeper performance
>
> Inline.
>
> Regards,
> Dima Gutzeit.
>
> Sent from my iPhone
>
> On 3 באוק 2011, at 19:31, Ted Dunning <te...@gmail.com> wrote:
>
>> Questions in-line
>>
>> On Mon, Oct 3, 2011 at 10:19 AM, Dima Gutzeit
>> <di...@mailvision.com>wrote:
>>
>>> Hi,
>>>
>>> I have some performance related question.
>>>
>>> I am running a cluster of 5 zookeeper machines, each one has dual quad
>>> core Xeon 2.5 Ghz, 8 gb RAM.
>>>
>>> I want achieve the following numbers:
>>>
>>> 10 clients producing in total (at peaks):
>>>
>>
>> Each producing this?
>
> Total.
>>
>>
>>> 3K add nodes, 3K delete nodes and 10K watches. Per second.
>>>
>>
>> What do you mean by 10K watches?  10,000 watch notifications?
>
> Yes.
>>
>> Does those numbers make any sense ?
>>>
>>
>> This is a bit high.  It sounds like you are trying to use ZK as a message
>> bus rather than a coordination service.  You can do this, but the throughput
>> you can achieve is limited.  If you want higher throughput, it is better to
>> have ZK coordinate a higher performance messaging system such as Kafka.

RE: ZooKeeper performance

["Fournier, Camille F." <Ca...@gs.com>]

It's pretty easy to set up a zk-smoketest to simulate what you are doing. We can't answer this question without knowing how big the data you're writing, etc etc. I would recommend testing it out yourself on realistic data sizes.

https://cwiki.apache.org/confluence/display/ZOOKEEPER/ServiceLatencyOverview

-----Original Message-----
From: Dima Gutzeit [mailto:dima.gutzeit@mailvision.com] 
Sent: Monday, October 03, 2011 1:36 PM
To: user@zookeeper.apache.org
Subject: Re: ZooKeeper performance

Inline.

Regards,
Dima Gutzeit.

Sent from my iPhone

On 3 באוק 2011, at 19:31, Ted Dunning <te...@gmail.com> wrote:

> Questions in-line
>
> On Mon, Oct 3, 2011 at 10:19 AM, Dima Gutzeit
> <di...@mailvision.com>wrote:
>
>> Hi,
>>
>> I have some performance related question.
>>
>> I am running a cluster of 5 zookeeper machines, each one has dual quad
>> core Xeon 2.5 Ghz, 8 gb RAM.
>>
>> I want achieve the following numbers:
>>
>> 10 clients producing in total (at peaks):
>>
>
> Each producing this?

Total.
>
>
>> 3K add nodes, 3K delete nodes and 10K watches. Per second.
>>
>
> What do you mean by 10K watches?  10,000 watch notifications?

Yes.
>
> Does those numbers make any sense ?
>>
>
> This is a bit high.  It sounds like you are trying to use ZK as a message
> bus rather than a coordination service.  You can do this, but the throughput
> you can achieve is limited.  If you want higher throughput, it is better to
> have ZK coordinate a higher performance messaging system such as Kafka.

Re: ZooKeeper performance

[Ted Dunning <te...@gmail.com>]

Correct.

But if your state is changing that fast, pub-sub might well be a useful way
to direct the firehose in the right direction.

On Mon, Oct 3, 2011 at 10:58 AM, Dima Gutzeit
<di...@mailvision.com>wrote:

> I read about Kafka as you suggested but it looks to me as a
> memory-less pub-sub while I want to keep a state in addition of
> notifying other watchers ...
>
> Regards,
> Dima Gutzeit.
>
> Sent from my iPhone
>
> On 3 באוק 2011, at 19:40, Ted Dunning <te...@gmail.com> wrote:
>
> > I think that these levels are achievable with a correctly tuned ZK
> cluster,
> > but you won't have as much head-room as I would like.  I really think
> that
> > using other systems built on top of ZK might be more useful.
> >
> > On Mon, Oct 3, 2011 at 10:35 AM, Dima Gutzeit
> > <di...@mailvision.com>wrote:
> >
> >> Inline.
> >>
> >> Regards,
> >> Dima Gutzeit.
> >>
> >> Sent from my iPhone
> >>
> >> On 3 באוק 2011, at 19:31, Ted Dunning <te...@gmail.com> wrote:
> >>
> >>> Questions in-line
> >>>
> >>> On Mon, Oct 3, 2011 at 10:19 AM, Dima Gutzeit
> >>> <di...@mailvision.com>wrote:
> >>>
> >>>> Hi,
> >>>>
> >>>> I have some performance related question.
> >>>>
> >>>> I am running a cluster of 5 zookeeper machines, each one has dual quad
> >>>> core Xeon 2.5 Ghz, 8 gb RAM.
> >>>>
> >>>> I want achieve the following numbers:
> >>>>
> >>>> 10 clients producing in total (at peaks):
> >>>>
> >>>
> >>> Each producing this?
> >>
> >> Total.
> >>>
> >>>
> >>>> 3K add nodes, 3K delete nodes and 10K watches. Per second.
> >>>>
> >>>
> >>> What do you mean by 10K watches?  10,000 watch notifications?
> >>
> >> Yes.
> >>>
> >>> Does those numbers make any sense ?
> >>>>
> >>>
> >>> This is a bit high.  It sounds like you are trying to use ZK as a
> message
> >>> bus rather than a coordination service.  You can do this, but the
> >> throughput
> >>> you can achieve is limited.  If you want higher throughput, it is
> better
> >> to
> >>> have ZK coordinate a higher performance messaging system such as Kafka.
> >>
>

Re: ZooKeeper performance

[Dima Gutzeit <di...@mailvision.com>]

I read about Kafka as you suggested but it looks to me as a
memory-less pub-sub while I want to keep a state in addition of
notifying other watchers ...

Regards,
Dima Gutzeit.

Sent from my iPhone

On 3 באוק 2011, at 19:40, Ted Dunning <te...@gmail.com> wrote:

> I think that these levels are achievable with a correctly tuned ZK cluster,
> but you won't have as much head-room as I would like.  I really think that
> using other systems built on top of ZK might be more useful.
>
> On Mon, Oct 3, 2011 at 10:35 AM, Dima Gutzeit
> <di...@mailvision.com>wrote:
>
>> Inline.
>>
>> Regards,
>> Dima Gutzeit.
>>
>> Sent from my iPhone
>>
>> On 3 באוק 2011, at 19:31, Ted Dunning <te...@gmail.com> wrote:
>>
>>> Questions in-line
>>>
>>> On Mon, Oct 3, 2011 at 10:19 AM, Dima Gutzeit
>>> <di...@mailvision.com>wrote:
>>>
>>>> Hi,
>>>>
>>>> I have some performance related question.
>>>>
>>>> I am running a cluster of 5 zookeeper machines, each one has dual quad
>>>> core Xeon 2.5 Ghz, 8 gb RAM.
>>>>
>>>> I want achieve the following numbers:
>>>>
>>>> 10 clients producing in total (at peaks):
>>>>
>>>
>>> Each producing this?
>>
>> Total.
>>>
>>>
>>>> 3K add nodes, 3K delete nodes and 10K watches. Per second.
>>>>
>>>
>>> What do you mean by 10K watches?  10,000 watch notifications?
>>
>> Yes.
>>>
>>> Does those numbers make any sense ?
>>>>
>>>
>>> This is a bit high.  It sounds like you are trying to use ZK as a message
>>> bus rather than a coordination service.  You can do this, but the
>> throughput
>>> you can achieve is limited.  If you want higher throughput, it is better
>> to
>>> have ZK coordinate a higher performance messaging system such as Kafka.
>>

Re: ZooKeeper performance

[Ted Dunning <te...@gmail.com>]

I think that these levels are achievable with a correctly tuned ZK cluster,
but you won't have as much head-room as I would like.  I really think that
using other systems built on top of ZK might be more useful.

On Mon, Oct 3, 2011 at 10:35 AM, Dima Gutzeit
<di...@mailvision.com>wrote:

> Inline.
>
> Regards,
> Dima Gutzeit.
>
> Sent from my iPhone
>
> On 3 באוק 2011, at 19:31, Ted Dunning <te...@gmail.com> wrote:
>
> > Questions in-line
> >
> > On Mon, Oct 3, 2011 at 10:19 AM, Dima Gutzeit
> > <di...@mailvision.com>wrote:
> >
> >> Hi,
> >>
> >> I have some performance related question.
> >>
> >> I am running a cluster of 5 zookeeper machines, each one has dual quad
> >> core Xeon 2.5 Ghz, 8 gb RAM.
> >>
> >> I want achieve the following numbers:
> >>
> >> 10 clients producing in total (at peaks):
> >>
> >
> > Each producing this?
>
> Total.
> >
> >
> >> 3K add nodes, 3K delete nodes and 10K watches. Per second.
> >>
> >
> > What do you mean by 10K watches?  10,000 watch notifications?
>
> Yes.
> >
> > Does those numbers make any sense ?
> >>
> >
> > This is a bit high.  It sounds like you are trying to use ZK as a message
> > bus rather than a coordination service.  You can do this, but the
> throughput
> > you can achieve is limited.  If you want higher throughput, it is better
> to
> > have ZK coordinate a higher performance messaging system such as Kafka.
>

Re: ZooKeeper performance

[Dima Gutzeit <di...@mailvision.com>]

Inline.

Regards,
Dima Gutzeit.

Sent from my iPhone

On 3 באוק 2011, at 19:31, Ted Dunning <te...@gmail.com> wrote:

> Questions in-line
>
> On Mon, Oct 3, 2011 at 10:19 AM, Dima Gutzeit
> <di...@mailvision.com>wrote:
>
>> Hi,
>>
>> I have some performance related question.
>>
>> I am running a cluster of 5 zookeeper machines, each one has dual quad
>> core Xeon 2.5 Ghz, 8 gb RAM.
>>
>> I want achieve the following numbers:
>>
>> 10 clients producing in total (at peaks):
>>
>
> Each producing this?

Total.
>
>
>> 3K add nodes, 3K delete nodes and 10K watches. Per second.
>>
>
> What do you mean by 10K watches?  10,000 watch notifications?

Yes.
>
> Does those numbers make any sense ?
>>
>
> This is a bit high.  It sounds like you are trying to use ZK as a message
> bus rather than a coordination service.  You can do this, but the throughput
> you can achieve is limited.  If you want higher throughput, it is better to
> have ZK coordinate a higher performance messaging system such as Kafka.

Re: ZooKeeper performance

[Ted Dunning <te...@gmail.com>]

Questions in-line

On Mon, Oct 3, 2011 at 10:19 AM, Dima Gutzeit
<di...@mailvision.com>wrote:

> Hi,
>
> I have some performance related question.
>
> I am running a cluster of 5 zookeeper machines, each one has dual quad
> core Xeon 2.5 Ghz, 8 gb RAM.
>
> I want achieve the following numbers:
>
> 10 clients producing in total (at peaks):
>

Each producing this?


> 3K add nodes, 3K delete nodes and 10K watches. Per second.
>

What do you mean by 10K watches?  10,000 watch notifications?

Does those numbers make any sense ?
>

This is a bit high.  It sounds like you are trying to use ZK as a message
bus rather than a coordination service.  You can do this, but the throughput
you can achieve is limited.  If you want higher throughput, it is better to
have ZK coordinate a higher performance messaging system such as Kafka.