Zookeeper on 60+Gb mem

[Maarten Koopmans <ma...@vrijheid.net>]

Hi,

I just wondered: has anybody ever ran zookeeper "to the max" on a 68GB quadruple extra large high memory EC2 instance? With, say, 60GB allocated or so?

Because EC2 with EBS is a nice way to grow your zookeeper cluster (data on the ebs columes, upgrade as your memory utilization grows....)  - I just wonder what the limits are there, or if I am foing where angels fear to tread...

--Maarten

Re: Zookeeper on 60+Gb mem

[Ted Dunning <te...@gmail.com>]

That would be an interesting experiment although it is way outside normal
usage as a coordination store.

I have used ZK as a session store for PHP with OK results.  I never
implemented an expiration mechanism so things
had to be cleared out manually sometimes.  It worked pretty well until
things filled up.

On Tue, Oct 5, 2010 at 11:03 AM, Maarten Koopmans <ma...@vrijheid.net>wrote:

> Hi,
>
> I just wondered: has anybody ever ran zookeeper "to the max" on a 68GB
> quadruple extra large high memory EC2 instance? With, say, 60GB allocated or
> so?
>
> Because EC2 with EBS is a nice way to grow your zookeeper cluster (data on
> the ebs columes, upgrade as your memory utilization grows....)  - I just
> wonder what the limits are there, or if I am foing where angels fear to
> tread...
>
> --Maarten

Re: Zookeeper on 60+Gb mem

[Maarten Koopmans <ma...@vrijheid.net>]

Yup, and that's ironic, isn't it? The gc tuning is so specialistic, as is the profiling, that automated memory management (to me) hasn't brought what I hoped it would. I've had some conversations about this topic a few years back with a well respected OS designer, and his point is that we (humans) can trace back almost all problems because we're adding complexity, in stead of reducing it.

Sorry for the slight rant.... Anyway, it's one of the things I like about zookeeper (and, e.g. voldemort): it makes a hard thing doable.

--Maarten


Op 5 okt. 2010 om 23:27 heeft Patrick Hunt <ph...@apache.org> het volgende geschreven:

> Tuning GC is going to be critical, otw all the sessions will timeout (and
> potentially expire) during GC pauses.
> 
> Patrick
> 
> On Tue, Oct 5, 2010 at 1:18 PM, Maarten Koopmans <ma...@vrijheid.net>wrote:
> 
>> Yes, and syncing after a crash will be interesting as well. Off note; I am
>> running it with a 6GB heap now, but it's not filled yet. I do have smoke
>> tests thoug, so maybe I'll give it a try.
>> 
>> 
>> 
>> Op 5 okt. 2010 om 21:13 heeft Benjamin Reed <br...@yahoo-inc.com> het
>> volgende geschreven:
>> 
>>> 
>>> you will need to time how long it takes to read all that state back in
>> and adjust the initTime accordingly. it will probably take a while to pull
>> all that data into memory.
>>> 
>>> ben
>>> 
>>> On 10/05/2010 11:36 AM, Avinash Lakshman wrote:
>>>> I have run it over 5 GB of heap with over 10M znodes. We will definitely
>> run
>>>> it with over 64 GB of heap. Technically I do not see any limitiation.
>>>> However I will the experts chime in.
>>>> 
>>>> Avinash
>>>> 
>>>> On Tue, Oct 5, 2010 at 11:14 AM, Mahadev Konar<mahadev@yahoo-inc.com
>>> wrote:
>>>> 
>>>>> Hi Maarteen,
>>>>> I definitely know of a group which uses around 3GB of memory heap for
>>>>> zookeeper but never heard of someone with such huge requirements. I
>> would
>>>>> say it definitely would be a learning experience with such high memory
>>>>> which
>>>>> I definitely think would be very very useful for others in the
>> community as
>>>>> well.
>>>>> 
>>>>> Thanks
>>>>> mahadev
>>>>> 
>>>>> 
>>>>> On 10/5/10 11:03 AM, "Maarten Koopmans"<ma...@vrijheid.net>  wrote:
>>>>> 
>>>>>> Hi,
>>>>>> 
>>>>>> I just wondered: has anybody ever ran zookeeper "to the max" on a 68GB
>>>>>> quadruple extra large high memory EC2 instance? With, say, 60GB
>> allocated
>>>>> or
>>>>>> so?
>>>>>> 
>>>>>> Because EC2 with EBS is a nice way to grow your zookeeper cluster
>> (data
>>>>> on the
>>>>>> ebs columes, upgrade as your memory utilization grows....)  - I just
>>>>> wonder
>>>>>> what the limits are there, or if I am foing where angels fear to
>> tread...
>>>>>> 
>>>>>> --Maarten
>>>>>> 
>>>>> 
>>> 
>>> 
>> 

Re: Zookeeper on 60+Gb mem

[Maarten Koopmans <ma...@vrijheid.net>]

Good point. And Cassandra is a no-go for me for now. I get the model, but I don't like, check, dislike, things like Thrift.  



Op 5 okt. 2010 om 23:54 heeft Dave Wright <wr...@gmail.com> het volgende geschreven:

> I think the issue of having to write a full ~60GB snapshot file at
> intervals would make this prohibitive, particularly on EC2 via EBS. At
> a scale like that I think you'd be better off with a traditional
> database or a nosql database like Cassandra, possibly using Zookeeper
> for transaction locking/coordination on top.
> 
> 
> -Dave Wright
> 
> On Tue, Oct 5, 2010 at 5:27 PM, Patrick Hunt <ph...@apache.org> wrote:
>> Tuning GC is going to be critical, otw all the sessions will timeout (and
>> potentially expire) during GC pauses.
>> 
>> Patrick
>> 
>> On Tue, Oct 5, 2010 at 1:18 PM, Maarten Koopmans <ma...@vrijheid.net>wrote:
>> 
>>> Yes, and syncing after a crash will be interesting as well. Off note; I am
>>> running it with a 6GB heap now, but it's not filled yet. I do have smoke
>>> tests thoug, so maybe I'll give it a try.
>>> 
>>> 
>>> 
>>> Op 5 okt. 2010 om 21:13 heeft Benjamin Reed <br...@yahoo-inc.com> het
>>> volgende geschreven:
>>> 
>>>> 
>>>> you will need to time how long it takes to read all that state back in
>>> and adjust the initTime accordingly. it will probably take a while to pull
>>> all that data into memory.
>>>> 
>>>> ben
>>>> 
>>>> On 10/05/2010 11:36 AM, Avinash Lakshman wrote:
>>>>> I have run it over 5 GB of heap with over 10M znodes. We will definitely
>>> run
>>>>> it with over 64 GB of heap. Technically I do not see any limitiation.
>>>>> However I will the experts chime in.
>>>>> 
>>>>> Avinash
>>>>> 
>>>>> On Tue, Oct 5, 2010 at 11:14 AM, Mahadev Konar<mahadev@yahoo-inc.com
>>>> wrote:
>>>>> 
>>>>>> Hi Maarteen,
>>>>>>  I definitely know of a group which uses around 3GB of memory heap for
>>>>>> zookeeper but never heard of someone with such huge requirements. I
>>> would
>>>>>> say it definitely would be a learning experience with such high memory
>>>>>> which
>>>>>> I definitely think would be very very useful for others in the
>>> community as
>>>>>> well.
>>>>>> 
>>>>>> Thanks
>>>>>> mahadev
>>>>>> 
>>>>>> 
>>>>>> On 10/5/10 11:03 AM, "Maarten Koopmans"<ma...@vrijheid.net>  wrote:
>>>>>> 
>>>>>>> Hi,
>>>>>>> 
>>>>>>> I just wondered: has anybody ever ran zookeeper "to the max" on a 68GB
>>>>>>> quadruple extra large high memory EC2 instance? With, say, 60GB
>>> allocated
>>>>>> or
>>>>>>> so?
>>>>>>> 
>>>>>>> Because EC2 with EBS is a nice way to grow your zookeeper cluster
>>> (data
>>>>>> on the
>>>>>>> ebs columes, upgrade as your memory utilization grows....)  - I just
>>>>>> wonder
>>>>>>> what the limits are there, or if I am foing where angels fear to
>>> tread...
>>>>>>> 
>>>>>>> --Maarten
>>>>>>> 
>>>>>> 
>>>> 
>>>> 
>>> 
>> 
> 

Re: Zookeeper on 60+Gb mem

[Dave Wright <wr...@gmail.com>]

I think the issue of having to write a full ~60GB snapshot file at
intervals would make this prohibitive, particularly on EC2 via EBS. At
a scale like that I think you'd be better off with a traditional
database or a nosql database like Cassandra, possibly using Zookeeper
for transaction locking/coordination on top.


-Dave Wright

On Tue, Oct 5, 2010 at 5:27 PM, Patrick Hunt <ph...@apache.org> wrote:
> Tuning GC is going to be critical, otw all the sessions will timeout (and
> potentially expire) during GC pauses.
>
> Patrick
>
> On Tue, Oct 5, 2010 at 1:18 PM, Maarten Koopmans <ma...@vrijheid.net>wrote:
>
>> Yes, and syncing after a crash will be interesting as well. Off note; I am
>> running it with a 6GB heap now, but it's not filled yet. I do have smoke
>> tests thoug, so maybe I'll give it a try.
>>
>>
>>
>> Op 5 okt. 2010 om 21:13 heeft Benjamin Reed <br...@yahoo-inc.com> het
>> volgende geschreven:
>>
>> >
>> > you will need to time how long it takes to read all that state back in
>> and adjust the initTime accordingly. it will probably take a while to pull
>> all that data into memory.
>> >
>> > ben
>> >
>> > On 10/05/2010 11:36 AM, Avinash Lakshman wrote:
>> >> I have run it over 5 GB of heap with over 10M znodes. We will definitely
>> run
>> >> it with over 64 GB of heap. Technically I do not see any limitiation.
>> >> However I will the experts chime in.
>> >>
>> >> Avinash
>> >>
>> >> On Tue, Oct 5, 2010 at 11:14 AM, Mahadev Konar<mahadev@yahoo-inc.com
>> >wrote:
>> >>
>> >>> Hi Maarteen,
>> >>>  I definitely know of a group which uses around 3GB of memory heap for
>> >>> zookeeper but never heard of someone with such huge requirements. I
>> would
>> >>> say it definitely would be a learning experience with such high memory
>> >>> which
>> >>> I definitely think would be very very useful for others in the
>> community as
>> >>> well.
>> >>>
>> >>> Thanks
>> >>> mahadev
>> >>>
>> >>>
>> >>> On 10/5/10 11:03 AM, "Maarten Koopmans"<ma...@vrijheid.net>  wrote:
>> >>>
>> >>>> Hi,
>> >>>>
>> >>>> I just wondered: has anybody ever ran zookeeper "to the max" on a 68GB
>> >>>> quadruple extra large high memory EC2 instance? With, say, 60GB
>> allocated
>> >>> or
>> >>>> so?
>> >>>>
>> >>>> Because EC2 with EBS is a nice way to grow your zookeeper cluster
>> (data
>> >>> on the
>> >>>> ebs columes, upgrade as your memory utilization grows....)  - I just
>> >>> wonder
>> >>>> what the limits are there, or if I am foing where angels fear to
>> tread...
>> >>>>
>> >>>> --Maarten
>> >>>>
>> >>>
>> >
>> >
>>
>

Re: Zookeeper on 60+Gb mem

[Patrick Hunt <ph...@apache.org>]

Tuning GC is going to be critical, otw all the sessions will timeout (and
potentially expire) during GC pauses.

Patrick

On Tue, Oct 5, 2010 at 1:18 PM, Maarten Koopmans <ma...@vrijheid.net>wrote:

> Yes, and syncing after a crash will be interesting as well. Off note; I am
> running it with a 6GB heap now, but it's not filled yet. I do have smoke
> tests thoug, so maybe I'll give it a try.
>
>
>
> Op 5 okt. 2010 om 21:13 heeft Benjamin Reed <br...@yahoo-inc.com> het
> volgende geschreven:
>
> >
> > you will need to time how long it takes to read all that state back in
> and adjust the initTime accordingly. it will probably take a while to pull
> all that data into memory.
> >
> > ben
> >
> > On 10/05/2010 11:36 AM, Avinash Lakshman wrote:
> >> I have run it over 5 GB of heap with over 10M znodes. We will definitely
> run
> >> it with over 64 GB of heap. Technically I do not see any limitiation.
> >> However I will the experts chime in.
> >>
> >> Avinash
> >>
> >> On Tue, Oct 5, 2010 at 11:14 AM, Mahadev Konar<mahadev@yahoo-inc.com
> >wrote:
> >>
> >>> Hi Maarteen,
> >>>  I definitely know of a group which uses around 3GB of memory heap for
> >>> zookeeper but never heard of someone with such huge requirements. I
> would
> >>> say it definitely would be a learning experience with such high memory
> >>> which
> >>> I definitely think would be very very useful for others in the
> community as
> >>> well.
> >>>
> >>> Thanks
> >>> mahadev
> >>>
> >>>
> >>> On 10/5/10 11:03 AM, "Maarten Koopmans"<ma...@vrijheid.net>  wrote:
> >>>
> >>>> Hi,
> >>>>
> >>>> I just wondered: has anybody ever ran zookeeper "to the max" on a 68GB
> >>>> quadruple extra large high memory EC2 instance? With, say, 60GB
> allocated
> >>> or
> >>>> so?
> >>>>
> >>>> Because EC2 with EBS is a nice way to grow your zookeeper cluster
> (data
> >>> on the
> >>>> ebs columes, upgrade as your memory utilization grows....)  - I just
> >>> wonder
> >>>> what the limits are there, or if I am foing where angels fear to
> tread...
> >>>>
> >>>> --Maarten
> >>>>
> >>>
> >
> >
>

Re: Zookeeper on 60+Gb mem

[Maarten Koopmans <ma...@vrijheid.net>]

Yes, and syncing after a crash will be interesting as well. Off note; I am running it with a 6GB heap now, but it's not filled yet. I do have smoke tests thoug, so maybe I'll give it a try.



Op 5 okt. 2010 om 21:13 heeft Benjamin Reed <br...@yahoo-inc.com> het volgende geschreven:

> 
> you will need to time how long it takes to read all that state back in and adjust the initTime accordingly. it will probably take a while to pull all that data into memory.
> 
> ben
> 
> On 10/05/2010 11:36 AM, Avinash Lakshman wrote:
>> I have run it over 5 GB of heap with over 10M znodes. We will definitely run
>> it with over 64 GB of heap. Technically I do not see any limitiation.
>> However I will the experts chime in.
>> 
>> Avinash
>> 
>> On Tue, Oct 5, 2010 at 11:14 AM, Mahadev Konar<ma...@yahoo-inc.com>wrote:
>> 
>>> Hi Maarteen,
>>>  I definitely know of a group which uses around 3GB of memory heap for
>>> zookeeper but never heard of someone with such huge requirements. I would
>>> say it definitely would be a learning experience with such high memory
>>> which
>>> I definitely think would be very very useful for others in the community as
>>> well.
>>> 
>>> Thanks
>>> mahadev
>>> 
>>> 
>>> On 10/5/10 11:03 AM, "Maarten Koopmans"<ma...@vrijheid.net>  wrote:
>>> 
>>>> Hi,
>>>> 
>>>> I just wondered: has anybody ever ran zookeeper "to the max" on a 68GB
>>>> quadruple extra large high memory EC2 instance? With, say, 60GB allocated
>>> or
>>>> so?
>>>> 
>>>> Because EC2 with EBS is a nice way to grow your zookeeper cluster (data
>>> on the
>>>> ebs columes, upgrade as your memory utilization grows....)  - I just
>>> wonder
>>>> what the limits are there, or if I am foing where angels fear to tread...
>>>> 
>>>> --Maarten
>>>> 
>>> 
> 
> 

Re: Zookeeper on 60+Gb mem

[Benjamin Reed <br...@yahoo-inc.com>]

  you will need to time how long it takes to read all that state back in 
and adjust the initTime accordingly. it will probably take a while to 
pull all that data into memory.

ben

On 10/05/2010 11:36 AM, Avinash Lakshman wrote:
> I have run it over 5 GB of heap with over 10M znodes. We will definitely run
> it with over 64 GB of heap. Technically I do not see any limitiation.
> However I will the experts chime in.
>
> Avinash
>
> On Tue, Oct 5, 2010 at 11:14 AM, Mahadev Konar<ma...@yahoo-inc.com>wrote:
>
>> Hi Maarteen,
>>   I definitely know of a group which uses around 3GB of memory heap for
>> zookeeper but never heard of someone with such huge requirements. I would
>> say it definitely would be a learning experience with such high memory
>> which
>> I definitely think would be very very useful for others in the community as
>> well.
>>
>> Thanks
>> mahadev
>>
>>
>> On 10/5/10 11:03 AM, "Maarten Koopmans"<ma...@vrijheid.net>  wrote:
>>
>>> Hi,
>>>
>>> I just wondered: has anybody ever ran zookeeper "to the max" on a 68GB
>>> quadruple extra large high memory EC2 instance? With, say, 60GB allocated
>> or
>>> so?
>>>
>>> Because EC2 with EBS is a nice way to grow your zookeeper cluster (data
>> on the
>>> ebs columes, upgrade as your memory utilization grows....)  - I just
>> wonder
>>> what the limits are there, or if I am foing where angels fear to tread...
>>>
>>> --Maarten
>>>
>>

Re: Zookeeper on 60+Gb mem

[Avinash Lakshman <av...@gmail.com>]

I have run it over 5 GB of heap with over 10M znodes. We will definitely run
it with over 64 GB of heap. Technically I do not see any limitiation.
However I will the experts chime in.

Avinash

On Tue, Oct 5, 2010 at 11:14 AM, Mahadev Konar <ma...@yahoo-inc.com>wrote:

> Hi Maarteen,
>  I definitely know of a group which uses around 3GB of memory heap for
> zookeeper but never heard of someone with such huge requirements. I would
> say it definitely would be a learning experience with such high memory
> which
> I definitely think would be very very useful for others in the community as
> well.
>
> Thanks
> mahadev
>
>
> On 10/5/10 11:03 AM, "Maarten Koopmans" <ma...@vrijheid.net> wrote:
>
> > Hi,
> >
> > I just wondered: has anybody ever ran zookeeper "to the max" on a 68GB
> > quadruple extra large high memory EC2 instance? With, say, 60GB allocated
> or
> > so?
> >
> > Because EC2 with EBS is a nice way to grow your zookeeper cluster (data
> on the
> > ebs columes, upgrade as your memory utilization grows....)  - I just
> wonder
> > what the limits are there, or if I am foing where angels fear to tread...
> >
> > --Maarten
> >
>
>

Re: Zookeeper on 60+Gb mem

[Mahadev Konar <ma...@yahoo-inc.com>]

Hi Maarteen,
  I definitely know of a group which uses around 3GB of memory heap for
zookeeper but never heard of someone with such huge requirements. I would
say it definitely would be a learning experience with such high memory which
I definitely think would be very very useful for others in the community as
well. 

Thanks
mahadev


On 10/5/10 11:03 AM, "Maarten Koopmans" <ma...@vrijheid.net> wrote:

> Hi,
> 
> I just wondered: has anybody ever ran zookeeper "to the max" on a 68GB
> quadruple extra large high memory EC2 instance? With, say, 60GB allocated or
> so?
> 
> Because EC2 with EBS is a nice way to grow your zookeeper cluster (data on the
> ebs columes, upgrade as your memory utilization grows....)  - I just wonder
> what the limits are there, or if I am foing where angels fear to tread...
> 
> --Maarten 
>