You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@zookeeper.apache.org by Ed Sexton <se...@gmail.com> on 2011/11/10 22:47:38 UTC
setting zookeeper heap size
Hello Group-
I am running CDH zookeeper version 3.3.3+12.12 and noticed that zookeeper
is running with a heapsize of 6477m.
I start zookeeper standalone, via /etc/init.d/hadoop-zookeeper-server. I
do not see reference to any ZOOKEEPER_HEAPIZE in the startup or zoo.cfg,
nor do I see any special JVMFLAGS in export/usr/bin/zookeeper-server, which
is what is called by the startup script, it just has this definition:
JVMFLAGS=-Dzookeeper.log.threshold=INFO
Can someone guide me to where the 6GB heapsize is being referenced? This
seems a bit large of a heap.
Thanks for your guidance.
Sincerely,
Ed
Re: setting zookeeper heap size
Posted by Patrick Hunt <ph...@apache.org>.
It changed "recently":
http://www.oracle.com/technetwork/java/javase/6u18-142093.html
"The default maximum heap size is half of the physical memory up to a
physical memory size of 192 megabytes and otherwise one fourth of the
physical memory up to a physical memory size of 1 gigabyte. "
"Server JVM heap configuration ergonomics are now the same as the
Client, except that the default maximum heap size for 32-bit JVMs is 1
gigabyte, corresponding to a physical memory size of 4 gigabytes, and
for 64-bit JVMs is 32 gigabytes, corresponding to a physical memory
size of 128 gigabytes."
Patrick
On Thu, Nov 10, 2011 at 4:14 PM, Camille Fournier <ca...@apache.org> wrote:
> It's very odd for the jvm to chose a 6g heap. On a 64bit machine it might
> choose 2g, I've never seen it go larger without xmx. Check to see if you
> have some sort of flags unintentionally added.
>
> C
> From my phone
> On Nov 10, 2011 4:59 PM, "Patrick Hunt" <ph...@apache.org> wrote:
>
>> In general ZK does not bound the heap size. The default is to use
>> whatever the JVM prefers, which is typically based on available host
>> memory. You can override this using JVMFLAGS.
>>
>> Patrick
>>
>> On Thu, Nov 10, 2011 at 1:47 PM, Ed Sexton <se...@gmail.com> wrote:
>> > Hello Group-
>> >
>> > I am running CDH zookeeper version 3.3.3+12.12 and noticed that zookeeper
>> > is running with a heapsize of 6477m.
>> >
>> > I start zookeeper standalone, via /etc/init.d/hadoop-zookeeper-server. I
>> > do not see reference to any ZOOKEEPER_HEAPIZE in the startup or zoo.cfg,
>> > nor do I see any special JVMFLAGS in export/usr/bin/zookeeper-server,
>> which
>> > is what is called by the startup script, it just has this definition:
>> >
>> > JVMFLAGS=-Dzookeeper.log.threshold=INFO
>> >
>> > Can someone guide me to where the 6GB heapsize is being referenced? This
>> > seems a bit large of a heap.
>> >
>> > Thanks for your guidance.
>> >
>> > Sincerely,
>> > Ed
>> >
>>
>
Re: setting zookeeper heap size
Posted by Henry Robinson <he...@cloudera.com>.
If you are using RHEL 6 chances are you're running into the newer per
thread allocator in recent glibc. See
http://www.quora.com/Why-do-some-applications-use-significantly-more-virtual-memory-on-RHEL-6-compared-to-RHEL-5
If so, it's harmless. Virtual memory address space is not a scarce
resource. If it concerns you, you can set MALLOC_ARENA_MAX to something
like 4 in the JVM's environment.
Henry
On Thursday, 10 November 2011, Camille Fournier <ca...@apache.org> wrote:
> It's very odd for the jvm to chose a 6g heap. On a 64bit machine it might
> choose 2g, I've never seen it go larger without xmx. Check to see if you
> have some sort of flags unintentionally added.
>
> C
> From my phone
> On Nov 10, 2011 4:59 PM, "Patrick Hunt" <ph...@apache.org> wrote:
>
>> In general ZK does not bound the heap size. The default is to use
>> whatever the JVM prefers, which is typically based on available host
>> memory. You can override this using JVMFLAGS.
>>
>> Patrick
>>
>> On Thu, Nov 10, 2011 at 1:47 PM, Ed Sexton <se...@gmail.com>
wrote:
>> > Hello Group-
>> >
>> > I am running CDH zookeeper version 3.3.3+12.12 and noticed that
zookeeper
>> > is running with a heapsize of 6477m.
>> >
>> > I start zookeeper standalone, via /etc/init.d/hadoop-zookeeper-server.
I
>> > do not see reference to any ZOOKEEPER_HEAPIZE in the startup or
zoo.cfg,
>> > nor do I see any special JVMFLAGS in export/usr/bin/zookeeper-server,
>> which
>> > is what is called by the startup script, it just has this definition:
>> >
>> > JVMFLAGS=-Dzookeeper.log.threshold=INFO
>> >
>> > Can someone guide me to where the 6GB heapsize is being referenced?
This
>> > seems a bit large of a heap.
>> >
>> > Thanks for your guidance.
>> >
>> > Sincerely,
>> > Ed
>> >
>>
>
--
Henry Robinson
Software Engineer
Cloudera
415-994-6679
Re: setting zookeeper heap size
Posted by Camille Fournier <ca...@apache.org>.
It's very odd for the jvm to chose a 6g heap. On a 64bit machine it might
choose 2g, I've never seen it go larger without xmx. Check to see if you
have some sort of flags unintentionally added.
C
>From my phone
On Nov 10, 2011 4:59 PM, "Patrick Hunt" <ph...@apache.org> wrote:
> In general ZK does not bound the heap size. The default is to use
> whatever the JVM prefers, which is typically based on available host
> memory. You can override this using JVMFLAGS.
>
> Patrick
>
> On Thu, Nov 10, 2011 at 1:47 PM, Ed Sexton <se...@gmail.com> wrote:
> > Hello Group-
> >
> > I am running CDH zookeeper version 3.3.3+12.12 and noticed that zookeeper
> > is running with a heapsize of 6477m.
> >
> > I start zookeeper standalone, via /etc/init.d/hadoop-zookeeper-server. I
> > do not see reference to any ZOOKEEPER_HEAPIZE in the startup or zoo.cfg,
> > nor do I see any special JVMFLAGS in export/usr/bin/zookeeper-server,
> which
> > is what is called by the startup script, it just has this definition:
> >
> > JVMFLAGS=-Dzookeeper.log.threshold=INFO
> >
> > Can someone guide me to where the 6GB heapsize is being referenced? This
> > seems a bit large of a heap.
> >
> > Thanks for your guidance.
> >
> > Sincerely,
> > Ed
> >
>
Re: setting zookeeper heap size
Posted by Patrick Hunt <ph...@apache.org>.
In general ZK does not bound the heap size. The default is to use
whatever the JVM prefers, which is typically based on available host
memory. You can override this using JVMFLAGS.
Patrick
On Thu, Nov 10, 2011 at 1:47 PM, Ed Sexton <se...@gmail.com> wrote:
> Hello Group-
>
> I am running CDH zookeeper version 3.3.3+12.12 and noticed that zookeeper
> is running with a heapsize of 6477m.
>
> I start zookeeper standalone, via /etc/init.d/hadoop-zookeeper-server. I
> do not see reference to any ZOOKEEPER_HEAPIZE in the startup or zoo.cfg,
> nor do I see any special JVMFLAGS in export/usr/bin/zookeeper-server, which
> is what is called by the startup script, it just has this definition:
>
> JVMFLAGS=-Dzookeeper.log.threshold=INFO
>
> Can someone guide me to where the 6GB heapsize is being referenced? This
> seems a bit large of a heap.
>
> Thanks for your guidance.
>
> Sincerely,
> Ed
>
Re: setting zookeeper heap size
Posted by Patrick Hunt <ph...@apache.org>.
Re wiseness - that all depends on the size of the content you are storing. ;-)
You could get more insight using jmap and the like to get an idea of
how much mem you are using. (in a test setup say)
Patrick
On Thu, Nov 10, 2011 at 5:32 PM, Ed Sexton <se...@gmail.com> wrote:
> Thanks all, I am understanding things a bit better.
>
> Here is the output of the heap. 1/4 of the available memory (24gb) does
> put it at ~6gb.
>
> I will override using the JVMFLAG and test it out. If this is not a wise
> thing to do, please let me know - thanks.
>
> - Ed
>
> # /usr/java/jdk1.6.0_25/bin/jmap -heap 11508
> Attaching to process ID 11508, please wait...
> Debugger attached successfully.
> Server compiler detected.
> JVM version is 20.0-b11
>
> using thread-local object allocation.
> Parallel GC with 13 thread(s)
>
> Heap Configuration:
> MinHeapFreeRatio = 40
> MaxHeapFreeRatio = 70
> MaxHeapSize = 6318718976 (6026.0MB)
> NewSize = 1310720 (1.25MB)
> MaxNewSize = 17592186044415 MB
> OldSize = 5439488 (5.1875MB)
> NewRatio = 2
> SurvivorRatio = 8
> PermSize = 21757952 (20.75MB)
> MaxPermSize = 85983232 (82.0MB)
>
> Heap Usage:
> PS Young Generation
> Eden Space:
> capacity = 262144 (0.25MB)
> used = 132088 (0.12596893310546875MB)
> free = 130056 (0.12403106689453125MB)
> 50.3875732421875% used
> From Space:
> capacity = 1900544 (1.8125MB)
> used = 163840 (0.15625MB)
> free = 1736704 (1.65625MB)
> 8.620689655172415% used
> To Space:
> capacity = 1900544 (1.8125MB)
> used = 0 (0.0MB)
> free = 1900544 (1.8125MB)
> 0.0% used
> PS Old Generation
> capacity = 4587520 (4.375MB)
> used = 2752448 (2.62493896484375MB)
> free = 1835072 (1.75006103515625MB)
> 59.998604910714285% used
> PS Perm Generation
> capacity = 21757952 (20.75MB)
> used = 10180360 (9.708747863769531MB)
> free = 11577592 (11.041252136230469MB)
> 46.78914633141943% used
>
>
>
>
> On Thu, Nov 10, 2011 at 4:51 PM, Patrick Hunt <ph...@apache.org> wrote:
>
>> Perhaps easier to just use jmap?
>>
>> sudo jmap -heap <jvmpid>
>>
>> Patrick
>>
>> On Thu, Nov 10, 2011 at 3:49 PM, Ed Sexton <se...@gmail.com> wrote:
>> > Hi Henry,
>> >
>> > I checked the process listing via ps and top, and saw VIRT=6477m and
>> > RES=83m.
>> >
>> > How would one check if it is the jvm heap vs mapped space?
>> >
>> > Thanks as always Henry.
>> > Ed
>> >
>> > On Thu, Nov 10, 2011 at 2:04 PM, Henry Robinson <he...@cloudera.com>
>> wrote:
>> >
>> >> Ed - how are you making the observation that the heap size is 6477m? Is
>> it
>> >> from a profiler, or from top or similar? The amount of virtual memory
>> >> mapped for Java processes can often be very large, especially on recent
>> >> RHEL systems with the new arena allocator in glibc. Worth checking
>> whether
>> >> it's the JVM heap or the amount of mapped space.
>> >>
>> >> cheers,
>> >> Henry
>> >>
>> >> On 10 November 2011 13:47, Ed Sexton <se...@gmail.com> wrote:
>> >>
>> >> > Hello Group-
>> >> >
>> >> > I am running CDH zookeeper version 3.3.3+12.12 and noticed that
>> zookeeper
>> >> > is running with a heapsize of 6477m.
>> >> >
>> >> > I start zookeeper standalone, via
>> /etc/init.d/hadoop-zookeeper-server. I
>> >> > do not see reference to any ZOOKEEPER_HEAPIZE in the startup or
>> zoo.cfg,
>> >> > nor do I see any special JVMFLAGS in export/usr/bin/zookeeper-server,
>> >> which
>> >> > is what is called by the startup script, it just has this definition:
>> >> >
>> >> > JVMFLAGS=-Dzookeeper.log.threshold=INFO
>> >> >
>> >> > Can someone guide me to where the 6GB heapsize is being referenced?
>> This
>> >> > seems a bit large of a heap.
>> >> >
>> >> > Thanks for your guidance.
>> >> >
>> >> > Sincerely,
>> >> > Ed
>> >> >
>> >>
>> >>
>> >>
>> >> --
>> >> Henry Robinson
>> >> Software Engineer
>> >> Cloudera
>> >> 415-994-6679
>> >>
>> >
>> >
>> >
>>
>
Re: setting zookeeper heap size
Posted by Ed Sexton <se...@gmail.com>.
Thanks all, I am understanding things a bit better.
Here is the output of the heap. 1/4 of the available memory (24gb) does
put it at ~6gb.
I will override using the JVMFLAG and test it out. If this is not a wise
thing to do, please let me know - thanks.
- Ed
# /usr/java/jdk1.6.0_25/bin/jmap -heap 11508
Attaching to process ID 11508, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 20.0-b11
using thread-local object allocation.
Parallel GC with 13 thread(s)
Heap Configuration:
MinHeapFreeRatio = 40
MaxHeapFreeRatio = 70
MaxHeapSize = 6318718976 (6026.0MB)
NewSize = 1310720 (1.25MB)
MaxNewSize = 17592186044415 MB
OldSize = 5439488 (5.1875MB)
NewRatio = 2
SurvivorRatio = 8
PermSize = 21757952 (20.75MB)
MaxPermSize = 85983232 (82.0MB)
Heap Usage:
PS Young Generation
Eden Space:
capacity = 262144 (0.25MB)
used = 132088 (0.12596893310546875MB)
free = 130056 (0.12403106689453125MB)
50.3875732421875% used
>From Space:
capacity = 1900544 (1.8125MB)
used = 163840 (0.15625MB)
free = 1736704 (1.65625MB)
8.620689655172415% used
To Space:
capacity = 1900544 (1.8125MB)
used = 0 (0.0MB)
free = 1900544 (1.8125MB)
0.0% used
PS Old Generation
capacity = 4587520 (4.375MB)
used = 2752448 (2.62493896484375MB)
free = 1835072 (1.75006103515625MB)
59.998604910714285% used
PS Perm Generation
capacity = 21757952 (20.75MB)
used = 10180360 (9.708747863769531MB)
free = 11577592 (11.041252136230469MB)
46.78914633141943% used
On Thu, Nov 10, 2011 at 4:51 PM, Patrick Hunt <ph...@apache.org> wrote:
> Perhaps easier to just use jmap?
>
> sudo jmap -heap <jvmpid>
>
> Patrick
>
> On Thu, Nov 10, 2011 at 3:49 PM, Ed Sexton <se...@gmail.com> wrote:
> > Hi Henry,
> >
> > I checked the process listing via ps and top, and saw VIRT=6477m and
> > RES=83m.
> >
> > How would one check if it is the jvm heap vs mapped space?
> >
> > Thanks as always Henry.
> > Ed
> >
> > On Thu, Nov 10, 2011 at 2:04 PM, Henry Robinson <he...@cloudera.com>
> wrote:
> >
> >> Ed - how are you making the observation that the heap size is 6477m? Is
> it
> >> from a profiler, or from top or similar? The amount of virtual memory
> >> mapped for Java processes can often be very large, especially on recent
> >> RHEL systems with the new arena allocator in glibc. Worth checking
> whether
> >> it's the JVM heap or the amount of mapped space.
> >>
> >> cheers,
> >> Henry
> >>
> >> On 10 November 2011 13:47, Ed Sexton <se...@gmail.com> wrote:
> >>
> >> > Hello Group-
> >> >
> >> > I am running CDH zookeeper version 3.3.3+12.12 and noticed that
> zookeeper
> >> > is running with a heapsize of 6477m.
> >> >
> >> > I start zookeeper standalone, via
> /etc/init.d/hadoop-zookeeper-server. I
> >> > do not see reference to any ZOOKEEPER_HEAPIZE in the startup or
> zoo.cfg,
> >> > nor do I see any special JVMFLAGS in export/usr/bin/zookeeper-server,
> >> which
> >> > is what is called by the startup script, it just has this definition:
> >> >
> >> > JVMFLAGS=-Dzookeeper.log.threshold=INFO
> >> >
> >> > Can someone guide me to where the 6GB heapsize is being referenced?
> This
> >> > seems a bit large of a heap.
> >> >
> >> > Thanks for your guidance.
> >> >
> >> > Sincerely,
> >> > Ed
> >> >
> >>
> >>
> >>
> >> --
> >> Henry Robinson
> >> Software Engineer
> >> Cloudera
> >> 415-994-6679
> >>
> >
> >
> >
>
Re: setting zookeeper heap size
Posted by Patrick Hunt <ph...@apache.org>.
Perhaps easier to just use jmap?
sudo jmap -heap <jvmpid>
Patrick
On Thu, Nov 10, 2011 at 3:49 PM, Ed Sexton <se...@gmail.com> wrote:
> Hi Henry,
>
> I checked the process listing via ps and top, and saw VIRT=6477m and
> RES=83m.
>
> How would one check if it is the jvm heap vs mapped space?
>
> Thanks as always Henry.
> Ed
>
> On Thu, Nov 10, 2011 at 2:04 PM, Henry Robinson <he...@cloudera.com> wrote:
>
>> Ed - how are you making the observation that the heap size is 6477m? Is it
>> from a profiler, or from top or similar? The amount of virtual memory
>> mapped for Java processes can often be very large, especially on recent
>> RHEL systems with the new arena allocator in glibc. Worth checking whether
>> it's the JVM heap or the amount of mapped space.
>>
>> cheers,
>> Henry
>>
>> On 10 November 2011 13:47, Ed Sexton <se...@gmail.com> wrote:
>>
>> > Hello Group-
>> >
>> > I am running CDH zookeeper version 3.3.3+12.12 and noticed that zookeeper
>> > is running with a heapsize of 6477m.
>> >
>> > I start zookeeper standalone, via /etc/init.d/hadoop-zookeeper-server. I
>> > do not see reference to any ZOOKEEPER_HEAPIZE in the startup or zoo.cfg,
>> > nor do I see any special JVMFLAGS in export/usr/bin/zookeeper-server,
>> which
>> > is what is called by the startup script, it just has this definition:
>> >
>> > JVMFLAGS=-Dzookeeper.log.threshold=INFO
>> >
>> > Can someone guide me to where the 6GB heapsize is being referenced? This
>> > seems a bit large of a heap.
>> >
>> > Thanks for your guidance.
>> >
>> > Sincerely,
>> > Ed
>> >
>>
>>
>>
>> --
>> Henry Robinson
>> Software Engineer
>> Cloudera
>> 415-994-6679
>>
>
>
>
> --
> Sincerely,
> Ed Sexton
> e: sextonhadoop@gmail.com
> gv: (408) 475-2358
>
Re: setting zookeeper heap size
Posted by Patrick Hunt <ph...@apache.org>.
I agree. I've been meaning to create a set of jiras for this, all
related to resource availability/env, it's on my todo list (came up
recently at a customer site). I see a number of things:
1) heap:
a) sane default for max mem (easy), plus docs and such.
b) turn on CMS/parallelGC by default
c) verify that the server correctly handles OOM in all cases (hard)
2) verify that there are sufficient FDs available (ie ulimit - also
came up recently at customer site)
a) warn if too low
b) try to set to some sane default, perhaps 2k unless already set higher
c) we might want to have the server not accept client connections
after some fd limit is reached. This would protect the storage system
from failure.
3) also see ZOOKEEPER-1239
a) in addition to fsync stall we should have a check for GC stop the
world pauses.
HBase seems to do all these things (e.g. see 1239). We might copy what
they are doing.
These are also all pretty easy things for new ZK contributors to take
on. (hint hint). Esp ones that are appreciative when experienced
contributors quickly takeup/fix the particularly nasty bugs. (hint
hint) ;-)
Patrick
On Fri, Nov 11, 2011 at 6:41 AM, Camille Fournier <ca...@apache.org> wrote:
> The problem in my mind is that it raises the spectre of users
> overcommitting their servers accidentally, and instead of seeing the
> result as a hard fail if their VM runs out of available memory, they
> will see it as a useless system when everything starts swapping to
> disk because the total heap size for VMs on the machine has grown
> beyond the reasonably allocatable physical memory of the machine.
> I don't really care about the ergonomics of client VMs, but server VMs
> in 64bit servers defaulting to 1/4 the memory of the system seems
> excessive. Anyway, whether it is reasonable for the vast case of VMs
> out there in the wild or not, it is way overboard for the vast case of
> ZooKeepers out there in the wild. I think we should consider offering
> guidance on this in documentation or the scripts.
>
> C
>
> On Thu, Nov 10, 2011 at 11:59 PM, Ted Dunning <te...@gmail.com> wrote:
>> The behavior without xmx is actually pretty reasonable. Note that the
>> default of half is only up to a few hundred MB of memory (not GB). Even
>> when lots of memory is allocated, the jvm avoids expanding the actual use
>> beyond reasonable need.
>>
>> On Thu, Nov 10, 2011 at 8:35 PM, Camille Fournier <ca...@apache.org>wrote:
>>
>>> Btw I'm pretty sure res is used heap and virt is just mapped but yet to be
>>> filled. Use jstat -gc to see actual usage for the various collectors.
>>>
>>> Also, if it's true that the jvm defaults to a xmx of half the ram (wtf bad
>>> decision) we had better ship zk start scripts with more reasonable xmx
>>> configured.
>>> Although I've used u18 and u23 extensively on 64bit servers and never seen
>>> it default to half the physical memory, or 32gig.
>>>
>>> C
>>>
>>> From my phone
>>> On Nov 10, 2011 6:50 PM, "Ed Sexton" <se...@gmail.com> wrote:
>>>
>>> > Hi Henry,
>>> >
>>> > I checked the process listing via ps and top, and saw VIRT=6477m and
>>> > RES=83m.
>>> >
>>> > How would one check if it is the jvm heap vs mapped space?
>>> >
>>> > Thanks as always Henry.
>>> > Ed
>>> >
>>> > On Thu, Nov 10, 2011 at 2:04 PM, Henry Robinson <he...@cloudera.com>
>>> > wrote:
>>> >
>>> > > Ed - how are you making the observation that the heap size is 6477m? Is
>>> > it
>>> > > from a profiler, or from top or similar? The amount of virtual memory
>>> > > mapped for Java processes can often be very large, especially on recent
>>> > > RHEL systems with the new arena allocator in glibc. Worth checking
>>> > whether
>>> > > it's the JVM heap or the amount of mapped space.
>>> > >
>>> > > cheers,
>>> > > Henry
>>> > >
>>> > > On 10 November 2011 13:47, Ed Sexton <se...@gmail.com> wrote:
>>> > >
>>> > > > Hello Group-
>>> > > >
>>> > > > I am running CDH zookeeper version 3.3.3+12.12 and noticed that
>>> > zookeeper
>>> > > > is running with a heapsize of 6477m.
>>> > > >
>>> > > > I start zookeeper standalone, via
>>> /etc/init.d/hadoop-zookeeper-server.
>>> > I
>>> > > > do not see reference to any ZOOKEEPER_HEAPIZE in the startup or
>>> > zoo.cfg,
>>> > > > nor do I see any special JVMFLAGS in export/usr/bin/zookeeper-server,
>>> > > which
>>> > > > is what is called by the startup script, it just has this definition:
>>> > > >
>>> > > > JVMFLAGS=-Dzookeeper.log.threshold=INFO
>>> > > >
>>> > > > Can someone guide me to where the 6GB heapsize is being referenced?
>>> > This
>>> > > > seems a bit large of a heap.
>>> > > >
>>> > > > Thanks for your guidance.
>>> > > >
>>> > > > Sincerely,
>>> > > > Ed
>>> > > >
>>> > >
>>> > >
>>> > >
>>> > > --
>>> > > Henry Robinson
>>> > > Software Engineer
>>> > > Cloudera
>>> > > 415-994-6679
>>> > >
>>> >
>>> >
>>> >
>>> > --
>>> > Sincerely,
>>> > Ed Sexton
>>> > e: sextonhadoop@gmail.com
>>> > gv: (408) 475-2358
>>> >
>>>
>>
>
Re: setting zookeeper heap size
Posted by Patrick Hunt <ph...@apache.org>.
I agree. I've been meaning to create a set of jiras for this, all
related to resource availability/env, it's on my todo list (came up
recently at a customer site). I see a number of things:
1) heap:
a) sane default for max mem (easy), plus docs and such.
b) turn on CMS/parallelGC by default
c) verify that the server correctly handles OOM in all cases (hard)
2) verify that there are sufficient FDs available (ie ulimit - also
came up recently at customer site)
a) warn if too low
b) try to set to some sane default, perhaps 2k unless already set higher
c) we might want to have the server not accept client connections
after some fd limit is reached. This would protect the storage system
from failure.
3) also see ZOOKEEPER-1239
a) in addition to fsync stall we should have a check for GC stop the
world pauses.
HBase seems to do all these things (e.g. see 1239). We might copy what
they are doing.
These are also all pretty easy things for new ZK contributors to take
on. (hint hint). Esp ones that are appreciative when experienced
contributors quickly takeup/fix the particularly nasty bugs. (hint
hint) ;-)
Patrick
On Fri, Nov 11, 2011 at 6:41 AM, Camille Fournier <ca...@apache.org> wrote:
> The problem in my mind is that it raises the spectre of users
> overcommitting their servers accidentally, and instead of seeing the
> result as a hard fail if their VM runs out of available memory, they
> will see it as a useless system when everything starts swapping to
> disk because the total heap size for VMs on the machine has grown
> beyond the reasonably allocatable physical memory of the machine.
> I don't really care about the ergonomics of client VMs, but server VMs
> in 64bit servers defaulting to 1/4 the memory of the system seems
> excessive. Anyway, whether it is reasonable for the vast case of VMs
> out there in the wild or not, it is way overboard for the vast case of
> ZooKeepers out there in the wild. I think we should consider offering
> guidance on this in documentation or the scripts.
>
> C
>
> On Thu, Nov 10, 2011 at 11:59 PM, Ted Dunning <te...@gmail.com> wrote:
>> The behavior without xmx is actually pretty reasonable. Note that the
>> default of half is only up to a few hundred MB of memory (not GB). Even
>> when lots of memory is allocated, the jvm avoids expanding the actual use
>> beyond reasonable need.
>>
>> On Thu, Nov 10, 2011 at 8:35 PM, Camille Fournier <ca...@apache.org>wrote:
>>
>>> Btw I'm pretty sure res is used heap and virt is just mapped but yet to be
>>> filled. Use jstat -gc to see actual usage for the various collectors.
>>>
>>> Also, if it's true that the jvm defaults to a xmx of half the ram (wtf bad
>>> decision) we had better ship zk start scripts with more reasonable xmx
>>> configured.
>>> Although I've used u18 and u23 extensively on 64bit servers and never seen
>>> it default to half the physical memory, or 32gig.
>>>
>>> C
>>>
>>> From my phone
>>> On Nov 10, 2011 6:50 PM, "Ed Sexton" <se...@gmail.com> wrote:
>>>
>>> > Hi Henry,
>>> >
>>> > I checked the process listing via ps and top, and saw VIRT=6477m and
>>> > RES=83m.
>>> >
>>> > How would one check if it is the jvm heap vs mapped space?
>>> >
>>> > Thanks as always Henry.
>>> > Ed
>>> >
>>> > On Thu, Nov 10, 2011 at 2:04 PM, Henry Robinson <he...@cloudera.com>
>>> > wrote:
>>> >
>>> > > Ed - how are you making the observation that the heap size is 6477m? Is
>>> > it
>>> > > from a profiler, or from top or similar? The amount of virtual memory
>>> > > mapped for Java processes can often be very large, especially on recent
>>> > > RHEL systems with the new arena allocator in glibc. Worth checking
>>> > whether
>>> > > it's the JVM heap or the amount of mapped space.
>>> > >
>>> > > cheers,
>>> > > Henry
>>> > >
>>> > > On 10 November 2011 13:47, Ed Sexton <se...@gmail.com> wrote:
>>> > >
>>> > > > Hello Group-
>>> > > >
>>> > > > I am running CDH zookeeper version 3.3.3+12.12 and noticed that
>>> > zookeeper
>>> > > > is running with a heapsize of 6477m.
>>> > > >
>>> > > > I start zookeeper standalone, via
>>> /etc/init.d/hadoop-zookeeper-server.
>>> > I
>>> > > > do not see reference to any ZOOKEEPER_HEAPIZE in the startup or
>>> > zoo.cfg,
>>> > > > nor do I see any special JVMFLAGS in export/usr/bin/zookeeper-server,
>>> > > which
>>> > > > is what is called by the startup script, it just has this definition:
>>> > > >
>>> > > > JVMFLAGS=-Dzookeeper.log.threshold=INFO
>>> > > >
>>> > > > Can someone guide me to where the 6GB heapsize is being referenced?
>>> > This
>>> > > > seems a bit large of a heap.
>>> > > >
>>> > > > Thanks for your guidance.
>>> > > >
>>> > > > Sincerely,
>>> > > > Ed
>>> > > >
>>> > >
>>> > >
>>> > >
>>> > > --
>>> > > Henry Robinson
>>> > > Software Engineer
>>> > > Cloudera
>>> > > 415-994-6679
>>> > >
>>> >
>>> >
>>> >
>>> > --
>>> > Sincerely,
>>> > Ed Sexton
>>> > e: sextonhadoop@gmail.com
>>> > gv: (408) 475-2358
>>> >
>>>
>>
>
Re: setting zookeeper heap size
Posted by Camille Fournier <ca...@apache.org>.
The problem in my mind is that it raises the spectre of users
overcommitting their servers accidentally, and instead of seeing the
result as a hard fail if their VM runs out of available memory, they
will see it as a useless system when everything starts swapping to
disk because the total heap size for VMs on the machine has grown
beyond the reasonably allocatable physical memory of the machine.
I don't really care about the ergonomics of client VMs, but server VMs
in 64bit servers defaulting to 1/4 the memory of the system seems
excessive. Anyway, whether it is reasonable for the vast case of VMs
out there in the wild or not, it is way overboard for the vast case of
ZooKeepers out there in the wild. I think we should consider offering
guidance on this in documentation or the scripts.
C
On Thu, Nov 10, 2011 at 11:59 PM, Ted Dunning <te...@gmail.com> wrote:
> The behavior without xmx is actually pretty reasonable. Note that the
> default of half is only up to a few hundred MB of memory (not GB). Even
> when lots of memory is allocated, the jvm avoids expanding the actual use
> beyond reasonable need.
>
> On Thu, Nov 10, 2011 at 8:35 PM, Camille Fournier <ca...@apache.org>wrote:
>
>> Btw I'm pretty sure res is used heap and virt is just mapped but yet to be
>> filled. Use jstat -gc to see actual usage for the various collectors.
>>
>> Also, if it's true that the jvm defaults to a xmx of half the ram (wtf bad
>> decision) we had better ship zk start scripts with more reasonable xmx
>> configured.
>> Although I've used u18 and u23 extensively on 64bit servers and never seen
>> it default to half the physical memory, or 32gig.
>>
>> C
>>
>> From my phone
>> On Nov 10, 2011 6:50 PM, "Ed Sexton" <se...@gmail.com> wrote:
>>
>> > Hi Henry,
>> >
>> > I checked the process listing via ps and top, and saw VIRT=6477m and
>> > RES=83m.
>> >
>> > How would one check if it is the jvm heap vs mapped space?
>> >
>> > Thanks as always Henry.
>> > Ed
>> >
>> > On Thu, Nov 10, 2011 at 2:04 PM, Henry Robinson <he...@cloudera.com>
>> > wrote:
>> >
>> > > Ed - how are you making the observation that the heap size is 6477m? Is
>> > it
>> > > from a profiler, or from top or similar? The amount of virtual memory
>> > > mapped for Java processes can often be very large, especially on recent
>> > > RHEL systems with the new arena allocator in glibc. Worth checking
>> > whether
>> > > it's the JVM heap or the amount of mapped space.
>> > >
>> > > cheers,
>> > > Henry
>> > >
>> > > On 10 November 2011 13:47, Ed Sexton <se...@gmail.com> wrote:
>> > >
>> > > > Hello Group-
>> > > >
>> > > > I am running CDH zookeeper version 3.3.3+12.12 and noticed that
>> > zookeeper
>> > > > is running with a heapsize of 6477m.
>> > > >
>> > > > I start zookeeper standalone, via
>> /etc/init.d/hadoop-zookeeper-server.
>> > I
>> > > > do not see reference to any ZOOKEEPER_HEAPIZE in the startup or
>> > zoo.cfg,
>> > > > nor do I see any special JVMFLAGS in export/usr/bin/zookeeper-server,
>> > > which
>> > > > is what is called by the startup script, it just has this definition:
>> > > >
>> > > > JVMFLAGS=-Dzookeeper.log.threshold=INFO
>> > > >
>> > > > Can someone guide me to where the 6GB heapsize is being referenced?
>> > This
>> > > > seems a bit large of a heap.
>> > > >
>> > > > Thanks for your guidance.
>> > > >
>> > > > Sincerely,
>> > > > Ed
>> > > >
>> > >
>> > >
>> > >
>> > > --
>> > > Henry Robinson
>> > > Software Engineer
>> > > Cloudera
>> > > 415-994-6679
>> > >
>> >
>> >
>> >
>> > --
>> > Sincerely,
>> > Ed Sexton
>> > e: sextonhadoop@gmail.com
>> > gv: (408) 475-2358
>> >
>>
>
Re: setting zookeeper heap size
Posted by Ted Dunning <te...@gmail.com>.
The behavior without xmx is actually pretty reasonable. Note that the
default of half is only up to a few hundred MB of memory (not GB). Even
when lots of memory is allocated, the jvm avoids expanding the actual use
beyond reasonable need.
On Thu, Nov 10, 2011 at 8:35 PM, Camille Fournier <ca...@apache.org>wrote:
> Btw I'm pretty sure res is used heap and virt is just mapped but yet to be
> filled. Use jstat -gc to see actual usage for the various collectors.
>
> Also, if it's true that the jvm defaults to a xmx of half the ram (wtf bad
> decision) we had better ship zk start scripts with more reasonable xmx
> configured.
> Although I've used u18 and u23 extensively on 64bit servers and never seen
> it default to half the physical memory, or 32gig.
>
> C
>
> From my phone
> On Nov 10, 2011 6:50 PM, "Ed Sexton" <se...@gmail.com> wrote:
>
> > Hi Henry,
> >
> > I checked the process listing via ps and top, and saw VIRT=6477m and
> > RES=83m.
> >
> > How would one check if it is the jvm heap vs mapped space?
> >
> > Thanks as always Henry.
> > Ed
> >
> > On Thu, Nov 10, 2011 at 2:04 PM, Henry Robinson <he...@cloudera.com>
> > wrote:
> >
> > > Ed - how are you making the observation that the heap size is 6477m? Is
> > it
> > > from a profiler, or from top or similar? The amount of virtual memory
> > > mapped for Java processes can often be very large, especially on recent
> > > RHEL systems with the new arena allocator in glibc. Worth checking
> > whether
> > > it's the JVM heap or the amount of mapped space.
> > >
> > > cheers,
> > > Henry
> > >
> > > On 10 November 2011 13:47, Ed Sexton <se...@gmail.com> wrote:
> > >
> > > > Hello Group-
> > > >
> > > > I am running CDH zookeeper version 3.3.3+12.12 and noticed that
> > zookeeper
> > > > is running with a heapsize of 6477m.
> > > >
> > > > I start zookeeper standalone, via
> /etc/init.d/hadoop-zookeeper-server.
> > I
> > > > do not see reference to any ZOOKEEPER_HEAPIZE in the startup or
> > zoo.cfg,
> > > > nor do I see any special JVMFLAGS in export/usr/bin/zookeeper-server,
> > > which
> > > > is what is called by the startup script, it just has this definition:
> > > >
> > > > JVMFLAGS=-Dzookeeper.log.threshold=INFO
> > > >
> > > > Can someone guide me to where the 6GB heapsize is being referenced?
> > This
> > > > seems a bit large of a heap.
> > > >
> > > > Thanks for your guidance.
> > > >
> > > > Sincerely,
> > > > Ed
> > > >
> > >
> > >
> > >
> > > --
> > > Henry Robinson
> > > Software Engineer
> > > Cloudera
> > > 415-994-6679
> > >
> >
> >
> >
> > --
> > Sincerely,
> > Ed Sexton
> > e: sextonhadoop@gmail.com
> > gv: (408) 475-2358
> >
>
Re: setting zookeeper heap size
Posted by Camille Fournier <ca...@apache.org>.
Btw I'm pretty sure res is used heap and virt is just mapped but yet to be
filled. Use jstat -gc to see actual usage for the various collectors.
Also, if it's true that the jvm defaults to a xmx of half the ram (wtf bad
decision) we had better ship zk start scripts with more reasonable xmx
configured.
Although I've used u18 and u23 extensively on 64bit servers and never seen
it default to half the physical memory, or 32gig.
C
>From my phone
On Nov 10, 2011 6:50 PM, "Ed Sexton" <se...@gmail.com> wrote:
> Hi Henry,
>
> I checked the process listing via ps and top, and saw VIRT=6477m and
> RES=83m.
>
> How would one check if it is the jvm heap vs mapped space?
>
> Thanks as always Henry.
> Ed
>
> On Thu, Nov 10, 2011 at 2:04 PM, Henry Robinson <he...@cloudera.com>
> wrote:
>
> > Ed - how are you making the observation that the heap size is 6477m? Is
> it
> > from a profiler, or from top or similar? The amount of virtual memory
> > mapped for Java processes can often be very large, especially on recent
> > RHEL systems with the new arena allocator in glibc. Worth checking
> whether
> > it's the JVM heap or the amount of mapped space.
> >
> > cheers,
> > Henry
> >
> > On 10 November 2011 13:47, Ed Sexton <se...@gmail.com> wrote:
> >
> > > Hello Group-
> > >
> > > I am running CDH zookeeper version 3.3.3+12.12 and noticed that
> zookeeper
> > > is running with a heapsize of 6477m.
> > >
> > > I start zookeeper standalone, via /etc/init.d/hadoop-zookeeper-server.
> I
> > > do not see reference to any ZOOKEEPER_HEAPIZE in the startup or
> zoo.cfg,
> > > nor do I see any special JVMFLAGS in export/usr/bin/zookeeper-server,
> > which
> > > is what is called by the startup script, it just has this definition:
> > >
> > > JVMFLAGS=-Dzookeeper.log.threshold=INFO
> > >
> > > Can someone guide me to where the 6GB heapsize is being referenced?
> This
> > > seems a bit large of a heap.
> > >
> > > Thanks for your guidance.
> > >
> > > Sincerely,
> > > Ed
> > >
> >
> >
> >
> > --
> > Henry Robinson
> > Software Engineer
> > Cloudera
> > 415-994-6679
> >
>
>
>
> --
> Sincerely,
> Ed Sexton
> e: sextonhadoop@gmail.com
> gv: (408) 475-2358
>
Re: setting zookeeper heap size
Posted by Ed Sexton <se...@gmail.com>.
Hi Henry,
I checked the process listing via ps and top, and saw VIRT=6477m and
RES=83m.
How would one check if it is the jvm heap vs mapped space?
Thanks as always Henry.
Ed
On Thu, Nov 10, 2011 at 2:04 PM, Henry Robinson <he...@cloudera.com> wrote:
> Ed - how are you making the observation that the heap size is 6477m? Is it
> from a profiler, or from top or similar? The amount of virtual memory
> mapped for Java processes can often be very large, especially on recent
> RHEL systems with the new arena allocator in glibc. Worth checking whether
> it's the JVM heap or the amount of mapped space.
>
> cheers,
> Henry
>
> On 10 November 2011 13:47, Ed Sexton <se...@gmail.com> wrote:
>
> > Hello Group-
> >
> > I am running CDH zookeeper version 3.3.3+12.12 and noticed that zookeeper
> > is running with a heapsize of 6477m.
> >
> > I start zookeeper standalone, via /etc/init.d/hadoop-zookeeper-server. I
> > do not see reference to any ZOOKEEPER_HEAPIZE in the startup or zoo.cfg,
> > nor do I see any special JVMFLAGS in export/usr/bin/zookeeper-server,
> which
> > is what is called by the startup script, it just has this definition:
> >
> > JVMFLAGS=-Dzookeeper.log.threshold=INFO
> >
> > Can someone guide me to where the 6GB heapsize is being referenced? This
> > seems a bit large of a heap.
> >
> > Thanks for your guidance.
> >
> > Sincerely,
> > Ed
> >
>
>
>
> --
> Henry Robinson
> Software Engineer
> Cloudera
> 415-994-6679
>
--
Sincerely,
Ed Sexton
e: sextonhadoop@gmail.com
gv: (408) 475-2358
Re: setting zookeeper heap size
Posted by Henry Robinson <he...@cloudera.com>.
Ed - how are you making the observation that the heap size is 6477m? Is it
from a profiler, or from top or similar? The amount of virtual memory
mapped for Java processes can often be very large, especially on recent
RHEL systems with the new arena allocator in glibc. Worth checking whether
it's the JVM heap or the amount of mapped space.
cheers,
Henry
On 10 November 2011 13:47, Ed Sexton <se...@gmail.com> wrote:
> Hello Group-
>
> I am running CDH zookeeper version 3.3.3+12.12 and noticed that zookeeper
> is running with a heapsize of 6477m.
>
> I start zookeeper standalone, via /etc/init.d/hadoop-zookeeper-server. I
> do not see reference to any ZOOKEEPER_HEAPIZE in the startup or zoo.cfg,
> nor do I see any special JVMFLAGS in export/usr/bin/zookeeper-server, which
> is what is called by the startup script, it just has this definition:
>
> JVMFLAGS=-Dzookeeper.log.threshold=INFO
>
> Can someone guide me to where the 6GB heapsize is being referenced? This
> seems a bit large of a heap.
>
> Thanks for your guidance.
>
> Sincerely,
> Ed
>
--
Henry Robinson
Software Engineer
Cloudera
415-994-6679