You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Wayne <wa...@gmail.com> on 2011/01/03 14:13:23 UTC
CMF & NodeIsDeadException
After heavily loading a 10 node cluster for 3-4 days I got a concurrent mode
failure of 53 seconds followed by a NodeIsDeadException which caused the
node to be shut down. Is there is a timeout that can be increased so this
does not occur in the future? From my experience with Cassandra Concurrent
Mode Failures are a reality of Java, so my assumption is the timeouts need
to accept them in terms of zookeeper and the master.
Any suggestions? The node was not even under load when this occurred. It was
idle after being under load for several days. I also saw other CMF errors on
this node and others so I assume this one just crossed the 60 second mark
which is the timeout exception somewhere?
Any help or suggestions would be appreciated. Parnew was getting large and
taking too long (> 100ms) so I will try to limit the size with the
suggestion from the performance tuning page (-XX:NewSize=6m
-XX:MaxNewSize=6m).
Thanks
Re: CMF & NodeIsDeadException
Posted by Todd Lipcon <to...@cloudera.com>.
On Mon, Jan 3, 2011 at 9:40 AM, Stack <st...@duboce.net> wrote:
> zookeeper.session.timeout is the config. to toggle. Its set to
> 180seconds in 0.90.0RC. Is it not so in your deploy?
>
> On Mon, Jan 3, 2011 at 5:13 AM, Wayne <wa...@gmail.com> wrote:
> >
> > Any help or suggestions would be appreciated. Parnew was getting large
> and
> > taking too long (> 100ms) so I will try to limit the size with the
> > suggestion from the performance tuning page (-XX:NewSize=6m
> > -XX:MaxNewSize=6m).
> >
>
> The CMS concurrent mode failure will be about trying to promote from
> new space up into the tenured heap but there's not the space in
> tenured heap to take the promotion because of fragmentation. You
> could try putting an upper bound on the new size (What size had your
> eden space grown too?). That would put off the CMF some but in long
> running app., CMF seems unavoidable, yeah.
>
Still working on this one on a backgroud thread over here, bugging the
hotspot guys :) I think our best bet is going to be basically doing a slow
rolling full GC in the cluster - if we can detect when the heap is
fragmented, we can shed regions gracefully, do GC, then pick them back up.
Detecting the fragmentation is possible from within the JVM source code, but
can't quite figure out how to expose it.
> A newsize of 6M is way too small given the heap sizes you've been
> bandy'ing about (You were thinking 64M? Even then, that seems too
> small).
>
+1. I'd recommend at least 64m new size.. if reasonably frequent 200-300ms
pauses are acceptable, go to 128m or larger. You can also tune SurvivorRatio
down and use a larger new size for some workloads, but it's a little messy
to figure this out.
-Todd
--
Todd Lipcon
Software Engineer, Cloudera
Re: CMF & NodeIsDeadException
Posted by Ted Yu <yu...@gmail.com>.
In an ideal minor collection the live objects are copied from one part of
the *young* generation (the eden space plus the first survivor space) to
another part of the *young* generation (the second survivor space).
See:
http://java.sun.com/docs/hotspot/gc1.4.2/
On Mon, Jan 3, 2011 at 12:50 PM, Wayne <wa...@gmail.com> wrote:
> We have an 8GB heap. What should newsize be? I just had another node die
> hard after going into a CMF storm. I swear it had solid CMFs 30+ in a row.
>
> I have no idea what eden space is or how to see what it is. ??
>
> Not knowing what else to do I will start using some of the Cassandra
> settings I used to improve it by setting the occupancy fraction. Any other
> ideas???
>
> Thanks.
>
> On Mon, Jan 3, 2011 at 12:40 PM, Stack <st...@duboce.net> wrote:
>
> > zookeeper.session.timeout is the config. to toggle. Its set to
> > 180seconds in 0.90.0RC. Is it not so in your deploy?
> >
> > On Mon, Jan 3, 2011 at 5:13 AM, Wayne <wa...@gmail.com> wrote:
> > >
> > > Any help or suggestions would be appreciated. Parnew was getting large
> > and
> > > taking too long (> 100ms) so I will try to limit the size with the
> > > suggestion from the performance tuning page (-XX:NewSize=6m
> > > -XX:MaxNewSize=6m).
> > >
> >
> > The CMS concurrent mode failure will be about trying to promote from
> > new space up into the tenured heap but there's not the space in
> > tenured heap to take the promotion because of fragmentation. You
> > could try putting an upper bound on the new size (What size had your
> > eden space grown too?). That would put off the CMF some but in long
> > running app., CMF seems unavoidable, yeah.
> >
> > A newsize of 6M is way too small given the heap sizes you've been
> > bandy'ing about (You were thinking 64M? Even then, that seems too
> > small).
> >
> > On failure of the node, all the regions came up again on new servers OK?
> >
> > St.Ack
> >
>
Re: CMF & NodeIsDeadException
Posted by Stack <st...@duboce.net>.
On Mon, Jan 3, 2011 at 2:13 PM, Wayne <wa...@gmail.com> wrote:
> Here are the new settings we are trying out. They seemed to "help" with
> cass. In the end I assume we will need a script to do rolling restarts or
> better yet hbase does it on its own!!
>
> Thanks for the help!
>
> -XX:+UseCMSInitiatingOccupancyOnly
> -XX:CMSInitiatingOccupancyFraction=60
This seems low. Means lots of CPU spent GC'ing. But that said, good
to start low then you can work up from there.
> -XX:+CMSParallelRemarkEnabled
> -XX:SurvivorRatio=8
> -XX:NewRatio=3
This is fine to start with but if it were me, I'd make the young gen
bigger (if objects don't make it up into the tenured heap, they'll not
get in the way of subsequent promotions). What proportion of heap was
it when you had long pauses?
> -XX:MaxTenuringThreshold=1
>
Setting this to 1 means stuff objects get promoted to tenured heap
after surviving only one young GC. I wonder if you set this to a
higher number how things would run? (Again, my rationale is that if
objects don't get into the tenured space in the first place, then they
can't be in the way when comes time to promote subsequent objects from
young to tenured.) It might be something to mess with later.
GC tuning, the "joy of java", is a little bit of a black art. Its
particularly black given that a bunch think there is no tuning that
will get you away from an occasional stop-the-world GC, at least when
running the CMS collector.
Keep us posted.
St.Ack
> On Mon, Jan 3, 2011 at 5:05 PM, Stack <st...@duboce.net> wrote:
>
>> On Mon, Jan 3, 2011 at 12:50 PM, Wayne <wa...@gmail.com> wrote:
>> > We have an 8GB heap. What should newsize be? I just had another node die
>> > hard after going into a CMF storm. I swear it had solid CMFs 30+ in a
>> row.
>> >
>>
>> Did a full stop-the-world GC run in between? It should have cleaned
>> up fragmentation.
>>
>> > I have no idea what eden space is or how to see what it is. ??
>> >
>>
>> Sorry. There's a bunch of 'cute' terms used for describing the two
>> heap areas in the JVM. Basically, new stuff goes into the 'new' or
>> 'eden' area first. If it sticks around through N (configurable) GCs,
>> it gets promoted to old or tenured generation (there are other names
>> for these notions of young and old). The garbage collection
>> algorithms done in the two heaps differ. See the Ted citation for
>> more on the gruesome details (though come up to a newer version of
>> that doc). The JVM is supposed to work ergonomically but it just
>> ain't smart enough dealing w/ HBase/Cass loadings it seems (e.g. it
>> keeps growing the new/eden space pathologically it would seem).
>>
>>
>> > Not knowing what else to do I will start using some of the Cassandra
>> > settings I used to improve it by setting the occupancy fraction. Any
>> other
>> > ideas???
>> >
>>
>> Which config. you talking of? -XX:+CMSInitiatingOccupancyFraction?
>> Thats a good one to toggle down from defaullts. Should help put off
>> promotion failures a while.
>>
>>
>> St.Ack
>>
>
Re: CMF & NodeIsDeadException
Posted by Ted Dunning <td...@maprtech.com>.
That tenuring threshold is scary.
I recommend printing the tenuring distribution to see if you are tenuring
garbage.
Remember, for server apps you want memory allocations to either die before
tenuring or live forever. Anything that gets tenured and then becomes
garbage will eventually trigger a full-GC. Of course, caches violate that,
but the more closely you can come to that ideal, the better.
On Mon, Jan 3, 2011 at 2:13 PM, Wayne <wa...@gmail.com> wrote:
> Here are the new settings we are trying out. They seemed to "help" with
> cass. In the end I assume we will need a script to do rolling restarts or
> better yet hbase does it on its own!!
>
> Thanks for the help!
>
> -XX:+UseCMSInitiatingOccupancyOnly
> -XX:CMSInitiatingOccupancyFraction=60
> -XX:+CMSParallelRemarkEnabled
> -XX:SurvivorRatio=8
> -XX:NewRatio=3
> -XX:MaxTenuringThreshold=1
>
> On Mon, Jan 3, 2011 at 5:05 PM, Stack <st...@duboce.net> wrote:
>
> > On Mon, Jan 3, 2011 at 12:50 PM, Wayne <wa...@gmail.com> wrote:
> > > We have an 8GB heap. What should newsize be? I just had another node
> die
> > > hard after going into a CMF storm. I swear it had solid CMFs 30+ in a
> > row.
> > >
> >
> > Did a full stop-the-world GC run in between? It should have cleaned
> > up fragmentation.
> >
> > > I have no idea what eden space is or how to see what it is. ??
> > >
> >
> > Sorry. There's a bunch of 'cute' terms used for describing the two
> > heap areas in the JVM. Basically, new stuff goes into the 'new' or
> > 'eden' area first. If it sticks around through N (configurable) GCs,
> > it gets promoted to old or tenured generation (there are other names
> > for these notions of young and old). The garbage collection
> > algorithms done in the two heaps differ. See the Ted citation for
> > more on the gruesome details (though come up to a newer version of
> > that doc). The JVM is supposed to work ergonomically but it just
> > ain't smart enough dealing w/ HBase/Cass loadings it seems (e.g. it
> > keeps growing the new/eden space pathologically it would seem).
> >
> >
> > > Not knowing what else to do I will start using some of the Cassandra
> > > settings I used to improve it by setting the occupancy fraction. Any
> > other
> > > ideas???
> > >
> >
> > Which config. you talking of? -XX:+CMSInitiatingOccupancyFraction?
> > Thats a good one to toggle down from defaullts. Should help put off
> > promotion failures a while.
> >
> >
> > St.Ack
> >
>
Re: CMF & NodeIsDeadException
Posted by Wayne <wa...@gmail.com>.
Here are the new settings we are trying out. They seemed to "help" with
cass. In the end I assume we will need a script to do rolling restarts or
better yet hbase does it on its own!!
Thanks for the help!
-XX:+UseCMSInitiatingOccupancyOnly
-XX:CMSInitiatingOccupancyFraction=60
-XX:+CMSParallelRemarkEnabled
-XX:SurvivorRatio=8
-XX:NewRatio=3
-XX:MaxTenuringThreshold=1
On Mon, Jan 3, 2011 at 5:05 PM, Stack <st...@duboce.net> wrote:
> On Mon, Jan 3, 2011 at 12:50 PM, Wayne <wa...@gmail.com> wrote:
> > We have an 8GB heap. What should newsize be? I just had another node die
> > hard after going into a CMF storm. I swear it had solid CMFs 30+ in a
> row.
> >
>
> Did a full stop-the-world GC run in between? It should have cleaned
> up fragmentation.
>
> > I have no idea what eden space is or how to see what it is. ??
> >
>
> Sorry. There's a bunch of 'cute' terms used for describing the two
> heap areas in the JVM. Basically, new stuff goes into the 'new' or
> 'eden' area first. If it sticks around through N (configurable) GCs,
> it gets promoted to old or tenured generation (there are other names
> for these notions of young and old). The garbage collection
> algorithms done in the two heaps differ. See the Ted citation for
> more on the gruesome details (though come up to a newer version of
> that doc). The JVM is supposed to work ergonomically but it just
> ain't smart enough dealing w/ HBase/Cass loadings it seems (e.g. it
> keeps growing the new/eden space pathologically it would seem).
>
>
> > Not knowing what else to do I will start using some of the Cassandra
> > settings I used to improve it by setting the occupancy fraction. Any
> other
> > ideas???
> >
>
> Which config. you talking of? -XX:+CMSInitiatingOccupancyFraction?
> Thats a good one to toggle down from defaullts. Should help put off
> promotion failures a while.
>
>
> St.Ack
>
Re: CMF & NodeIsDeadException
Posted by Stack <st...@duboce.net>.
On Mon, Jan 3, 2011 at 12:50 PM, Wayne <wa...@gmail.com> wrote:
> We have an 8GB heap. What should newsize be? I just had another node die
> hard after going into a CMF storm. I swear it had solid CMFs 30+ in a row.
>
Did a full stop-the-world GC run in between? It should have cleaned
up fragmentation.
> I have no idea what eden space is or how to see what it is. ??
>
Sorry. There's a bunch of 'cute' terms used for describing the two
heap areas in the JVM. Basically, new stuff goes into the 'new' or
'eden' area first. If it sticks around through N (configurable) GCs,
it gets promoted to old or tenured generation (there are other names
for these notions of young and old). The garbage collection
algorithms done in the two heaps differ. See the Ted citation for
more on the gruesome details (though come up to a newer version of
that doc). The JVM is supposed to work ergonomically but it just
ain't smart enough dealing w/ HBase/Cass loadings it seems (e.g. it
keeps growing the new/eden space pathologically it would seem).
> Not knowing what else to do I will start using some of the Cassandra
> settings I used to improve it by setting the occupancy fraction. Any other
> ideas???
>
Which config. you talking of? -XX:+CMSInitiatingOccupancyFraction?
Thats a good one to toggle down from defaullts. Should help put off
promotion failures a while.
St.Ack
Re: CMF & NodeIsDeadException
Posted by Wayne <wa...@gmail.com>.
We have an 8GB heap. What should newsize be? I just had another node die
hard after going into a CMF storm. I swear it had solid CMFs 30+ in a row.
I have no idea what eden space is or how to see what it is. ??
Not knowing what else to do I will start using some of the Cassandra
settings I used to improve it by setting the occupancy fraction. Any other
ideas???
Thanks.
On Mon, Jan 3, 2011 at 12:40 PM, Stack <st...@duboce.net> wrote:
> zookeeper.session.timeout is the config. to toggle. Its set to
> 180seconds in 0.90.0RC. Is it not so in your deploy?
>
> On Mon, Jan 3, 2011 at 5:13 AM, Wayne <wa...@gmail.com> wrote:
> >
> > Any help or suggestions would be appreciated. Parnew was getting large
> and
> > taking too long (> 100ms) so I will try to limit the size with the
> > suggestion from the performance tuning page (-XX:NewSize=6m
> > -XX:MaxNewSize=6m).
> >
>
> The CMS concurrent mode failure will be about trying to promote from
> new space up into the tenured heap but there's not the space in
> tenured heap to take the promotion because of fragmentation. You
> could try putting an upper bound on the new size (What size had your
> eden space grown too?). That would put off the CMF some but in long
> running app., CMF seems unavoidable, yeah.
>
> A newsize of 6M is way too small given the heap sizes you've been
> bandy'ing about (You were thinking 64M? Even then, that seems too
> small).
>
> On failure of the node, all the regions came up again on new servers OK?
>
> St.Ack
>
Re: CMF & NodeIsDeadException
Posted by Stack <st...@duboce.net>.
zookeeper.session.timeout is the config. to toggle. Its set to
180seconds in 0.90.0RC. Is it not so in your deploy?
On Mon, Jan 3, 2011 at 5:13 AM, Wayne <wa...@gmail.com> wrote:
>
> Any help or suggestions would be appreciated. Parnew was getting large and
> taking too long (> 100ms) so I will try to limit the size with the
> suggestion from the performance tuning page (-XX:NewSize=6m
> -XX:MaxNewSize=6m).
>
The CMS concurrent mode failure will be about trying to promote from
new space up into the tenured heap but there's not the space in
tenured heap to take the promotion because of fragmentation. You
could try putting an upper bound on the new size (What size had your
eden space grown too?). That would put off the CMF some but in long
running app., CMF seems unavoidable, yeah.
A newsize of 6M is way too small given the heap sizes you've been
bandy'ing about (You were thinking 64M? Even then, that seems too
small).
On failure of the node, all the regions came up again on new servers OK?
St.Ack