You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-user@hadoop.apache.org by felix gao <gr...@gmail.com> on 2012/01/19 00:23:16 UTC
namenode grows overtime
Hi guys,
we are running Hadoop 0.20.2+228, the namenode process's memory grows
overtime to occupy over 18GB. However, if I restart the namenode, it only
occupies about 10GB after it is stable. I am wondering if there is anyway
to figure out what is going on with namenode that causes it to grow very
rapidly and if there are any tools to make the namenode printout some
useful information on what is holding onto that memory.
Thanks,
Felix
Re: namenode grows overtime
Posted by felix gao <gr...@gmail.com>.
I noticed this from doing a top of the process and it is reporting the RES
is about 80% of the allocated.
On Wed, Jan 18, 2012 at 6:56 PM, Harsh J <ha...@cloudera.com> wrote:
> Felix,
>
> Are you noticing this from the NN Web UI or from proper JVM monitoring
> tools? Just so you know, the UI had a flaw that has since been fixed:
> https://issues.apache.org/jira/browse/HDFS-94
>
> On Thu, Jan 19, 2012 at 4:53 AM, felix gao <gr...@gmail.com> wrote:
> > Hi guys,
> >
> > we are running Hadoop 0.20.2+228, the namenode process's memory grows
> > overtime to occupy over 18GB. However, if I restart the namenode, it
> only
> > occupies about 10GB after it is stable. I am wondering if there is
> anyway
> > to figure out what is going on with namenode that causes it to grow very
> > rapidly and if there are any tools to make the namenode printout some
> useful
> > information on what is holding onto that memory.
> >
> > Thanks,
> >
> > Felix
>
>
>
> --
> Harsh J
> Customer Ops. Engineer, Cloudera
>
Re: namenode grows overtime
Posted by Harsh J <ha...@cloudera.com>.
Felix,
Are you noticing this from the NN Web UI or from proper JVM monitoring
tools? Just so you know, the UI had a flaw that has since been fixed:
https://issues.apache.org/jira/browse/HDFS-94
On Thu, Jan 19, 2012 at 4:53 AM, felix gao <gr...@gmail.com> wrote:
> Hi guys,
>
> we are running Hadoop 0.20.2+228, the namenode process's memory grows
> overtime to occupy over 18GB. However, if I restart the namenode, it only
> occupies about 10GB after it is stable. I am wondering if there is anyway
> to figure out what is going on with namenode that causes it to grow very
> rapidly and if there are any tools to make the namenode printout some useful
> information on what is holding onto that memory.
>
> Thanks,
>
> Felix
--
Harsh J
Customer Ops. Engineer, Cloudera
RE: namenode grows overtime
Posted by Uma Maheswara Rao G <ma...@huawei.com>.
Can you please check in UI, what is the heap usage. Then we can confirm whether java heap is growing or not.
top will consider native memory usage also and nio uses directByteBuffers internally.
This is good write up from Jonathan https://groups.google.com/group/asynchbase/browse_thread/thread/c45bc7ba788b2357
Regards,
Uma
________________________________
From: felix gao [gre1600@gmail.com]
Sent: Friday, January 20, 2012 6:42 AM
To: hdfs-user@hadoop.apache.org
Subject: Re: namenode grows overtime
Koji,
There is no Java options specified other than -Xmx24g, what are some of the recommended options for namenode?
Thanks,
Felix
On Wed, Jan 18, 2012 at 3:39 PM, Koji Noguchi <kn...@yahoo-inc.com>> wrote:
Hi Felix,
Taking jmap –histo:live <pid> would tell you what’s occupying the heap.
Are you using UseConcMarkSweepGC?
If yes, and if you see bunch of
java.net.SocksSocketImpl
sun.nio.ch.SocketChannelImpl
from jmap histo outputs, try passing -XX:-CMSConcurrentMTEnabled .
Background: http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=7113118
Koji
On 1/18/12 3:23 PM, "felix gao" <gr...@gmail.com>> wrote:
Hi guys,
we are running Hadoop 0.20.2+228, the namenode process's memory grows overtime to occupy over 18GB. However, if I restart the namenode, it only occupies about 10GB after it is stable. I am wondering if there is anyway to figure out what is going on with namenode that causes it to grow very rapidly and if there are any tools to make the namenode printout some useful information on what is holding onto that memory.
Thanks,
Felix
Re: namenode grows overtime
Posted by felix gao <gr...@gmail.com>.
Koji,
There is no Java options specified other than -Xmx24g, what are some of the
recommended options for namenode?
Thanks,
Felix
On Wed, Jan 18, 2012 at 3:39 PM, Koji Noguchi <kn...@yahoo-inc.com>wrote:
> Hi Felix,
>
> Taking jmap –histo:live <pid> would tell you what’s occupying the heap.
>
> Are you using UseConcMarkSweepGC?
> If yes, and if you see bunch of
> java.net.SocksSocketImpl
> sun.nio.ch.SocketChannelImpl
> from jmap histo outputs, try passing -XX:-CMSConcurrentMTEnabled .
>
> Background: http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=7113118
>
> Koji
>
>
>
>
> On 1/18/12 3:23 PM, "felix gao" <gr...@gmail.com> wrote:
>
> Hi guys,
>
> we are running Hadoop 0.20.2+228, the namenode process's memory grows
> overtime to occupy over 18GB. However, if I restart the namenode, it only
> occupies about 10GB after it is stable. I am wondering if there is anyway
> to figure out what is going on with namenode that causes it to grow very
> rapidly and if there are any tools to make the namenode printout some
> useful information on what is holding onto that memory.
>
> Thanks,
>
> Felix
>
>
Re: namenode grows overtime
Posted by Koji Noguchi <kn...@yahoo-inc.com>.
Hi Felix,
Taking jmap –histo:live <pid> would tell you what’s occupying the heap.
Are you using UseConcMarkSweepGC?
If yes, and if you see bunch of
java.net.SocksSocketImpl
sun.nio.ch.SocketChannelImpl
from jmap histo outputs, try passing -XX:-CMSConcurrentMTEnabled .
Background: http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=7113118
Koji
On 1/18/12 3:23 PM, "felix gao" <gr...@gmail.com> wrote:
Hi guys,
we are running Hadoop 0.20.2+228, the namenode process's memory grows overtime to occupy over 18GB. However, if I restart the namenode, it only occupies about 10GB after it is stable. I am wondering if there is anyway to figure out what is going on with namenode that causes it to grow very rapidly and if there are any tools to make the namenode printout some useful information on what is holding onto that memory.
Thanks,
Felix