You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Shahnawaz Saifi <sh...@gmail.com> on 2011/03/22 11:41:43 UTC
HRegionserver memory usage
Hi,
While uploading data to hbase memory usage getting too high (95%). we have a
cluster of seven nodes (hadoop master namenode, secondary namenode, hbase
master node, zookeeper node, slave datanode1, 2,3). Slave data nodes are
getting memory usage upto 99% and same time later some of them are
unresponsive.
--
Thanks,
Shah
Re: HRegionserver memory usage
Posted by Shahnawaz Saifi <sh...@gmail.com>.
After making below changes. It shows "heapdumponoutofmemoryerror".
On Thu, Mar 24, 2011 at 8:54 AM, Shahnawaz Saifi <sh...@gmail.com>wrote:
> Thank you, I changed heap size to 4GB and enabled gc. Along this also
> defined :
> export HBASE_REGIONSERVER_OPTS="$HBASE_REGIONSERVER_OPTS $HBASE_JMX_BASE
> -Dcom.sun.management.jmxremote.port=10102 -Xmx5120m"
> Will check the performance.
>
> -Shah
>
> On Thu, Mar 24, 2011 at 6:23 AM, Buttler, David <bu...@llnl.gov> wrote:
>
>> You might want to think about upping your hbase heap size to at least 4GB,
>> preferably 8GB. That only leaves about 6-7GB for map/reduce tasks (1 GB for
>> data node, 1 GB for OS), but unless you have a lot of them or are doing
>> extremely memory intensive operations it will be better.
>>
>> Also, consider upgrading to 0.90.1 and enabling MSLAB using the CMS
>> garbage collector as pointed out earlier on the list.
>>
>> Dave
>>
>> -----Original Message-----
>> From: Shahnawaz Saifi [mailto:shahsaifi@gmail.com]
>> Sent: Tuesday, March 22, 2011 8:59 AM
>> To: user@hbase.apache.org
>> Cc: Stack
>> Subject: Re: HRegionserver memory usage
>>
>> we are using hbase-0.20.6
>> hbasemaster, master & secondary nn has 36GB of Memory, zookeeper & slave
>> datanodes got 16GB. And heap size is default in hbase-env.sh.
>>
>> unresponsive: box is not accessible it hangs and shows java error on
>> screen.
>>
>>
>> DN uses entire available resources. Client are able to store data into
>> hbase
>> but not completely because job crashes as box goes out of memory and hangs
>> :(.
>>
>> Thanks,
>> Shah
>>
>> On Tue, Mar 22, 2011 at 8:58 PM, Stack <st...@duboce.net> wrote:
>>
>> > Can you provide more detail on versions and memory sizes.
>> >
>> > In general, the JVM grows to occupy the allotted heap so your reports
>> > below about its using 95% and 99% could be just 'normal' behavior.
>> >
>> > When its 'unresponsive', what do the logs say? What do you mean by
>> > 'unresponsive'? DN won't take any load? Your client can't put any
>> > data into hbase?
>> >
>> > Thanks,
>> > St>Ack
>> >
>> > On Tue, Mar 22, 2011 at 3:41 AM, Shahnawaz Saifi <sh...@gmail.com>
>> > wrote:
>> > > Hi,
>> > >
>> > > While uploading data to hbase memory usage getting too high (95%). we
>> > have a
>> > > cluster of seven nodes (hadoop master namenode, secondary namenode,
>> hbase
>> > > master node, zookeeper node, slave datanode1, 2,3). Slave data nodes
>> are
>> > > getting memory usage upto 99% and same time later some of them are
>> > > unresponsive.
>> > >
>> > > --
>> > > Thanks,
>> > > Shah
>> > >
>> >
>>
>>
>>
>> --
>> Thanks,
>> Shah
>>
>
>
>
> --
> Thanks,
> Shah
>
>
--
Thanks,
Shah
Re: HRegionserver memory usage
Posted by Shahnawaz Saifi <sh...@gmail.com>.
Thank you, I changed heap size to 4GB and enabled gc. Along this also
defined :
export HBASE_REGIONSERVER_OPTS="$HBASE_REGIONSERVER_OPTS $HBASE_JMX_BASE
-Dcom.sun.management.jmxremote.port=10102 -Xmx5120m"
Will check the performance.
-Shah
On Thu, Mar 24, 2011 at 6:23 AM, Buttler, David <bu...@llnl.gov> wrote:
> You might want to think about upping your hbase heap size to at least 4GB,
> preferably 8GB. That only leaves about 6-7GB for map/reduce tasks (1 GB for
> data node, 1 GB for OS), but unless you have a lot of them or are doing
> extremely memory intensive operations it will be better.
>
> Also, consider upgrading to 0.90.1 and enabling MSLAB using the CMS garbage
> collector as pointed out earlier on the list.
>
> Dave
>
> -----Original Message-----
> From: Shahnawaz Saifi [mailto:shahsaifi@gmail.com]
> Sent: Tuesday, March 22, 2011 8:59 AM
> To: user@hbase.apache.org
> Cc: Stack
> Subject: Re: HRegionserver memory usage
>
> we are using hbase-0.20.6
> hbasemaster, master & secondary nn has 36GB of Memory, zookeeper & slave
> datanodes got 16GB. And heap size is default in hbase-env.sh.
>
> unresponsive: box is not accessible it hangs and shows java error on
> screen.
>
>
> DN uses entire available resources. Client are able to store data into
> hbase
> but not completely because job crashes as box goes out of memory and hangs
> :(.
>
> Thanks,
> Shah
>
> On Tue, Mar 22, 2011 at 8:58 PM, Stack <st...@duboce.net> wrote:
>
> > Can you provide more detail on versions and memory sizes.
> >
> > In general, the JVM grows to occupy the allotted heap so your reports
> > below about its using 95% and 99% could be just 'normal' behavior.
> >
> > When its 'unresponsive', what do the logs say? What do you mean by
> > 'unresponsive'? DN won't take any load? Your client can't put any
> > data into hbase?
> >
> > Thanks,
> > St>Ack
> >
> > On Tue, Mar 22, 2011 at 3:41 AM, Shahnawaz Saifi <sh...@gmail.com>
> > wrote:
> > > Hi,
> > >
> > > While uploading data to hbase memory usage getting too high (95%). we
> > have a
> > > cluster of seven nodes (hadoop master namenode, secondary namenode,
> hbase
> > > master node, zookeeper node, slave datanode1, 2,3). Slave data nodes
> are
> > > getting memory usage upto 99% and same time later some of them are
> > > unresponsive.
> > >
> > > --
> > > Thanks,
> > > Shah
> > >
> >
>
>
>
> --
> Thanks,
> Shah
>
--
Thanks,
Shah
RE: HRegionserver memory usage
Posted by "Buttler, David" <bu...@llnl.gov>.
You might want to think about upping your hbase heap size to at least 4GB, preferably 8GB. That only leaves about 6-7GB for map/reduce tasks (1 GB for data node, 1 GB for OS), but unless you have a lot of them or are doing extremely memory intensive operations it will be better.
Also, consider upgrading to 0.90.1 and enabling MSLAB using the CMS garbage collector as pointed out earlier on the list.
Dave
-----Original Message-----
From: Shahnawaz Saifi [mailto:shahsaifi@gmail.com]
Sent: Tuesday, March 22, 2011 8:59 AM
To: user@hbase.apache.org
Cc: Stack
Subject: Re: HRegionserver memory usage
we are using hbase-0.20.6
hbasemaster, master & secondary nn has 36GB of Memory, zookeeper & slave
datanodes got 16GB. And heap size is default in hbase-env.sh.
unresponsive: box is not accessible it hangs and shows java error on screen.
DN uses entire available resources. Client are able to store data into hbase
but not completely because job crashes as box goes out of memory and hangs
:(.
Thanks,
Shah
On Tue, Mar 22, 2011 at 8:58 PM, Stack <st...@duboce.net> wrote:
> Can you provide more detail on versions and memory sizes.
>
> In general, the JVM grows to occupy the allotted heap so your reports
> below about its using 95% and 99% could be just 'normal' behavior.
>
> When its 'unresponsive', what do the logs say? What do you mean by
> 'unresponsive'? DN won't take any load? Your client can't put any
> data into hbase?
>
> Thanks,
> St>Ack
>
> On Tue, Mar 22, 2011 at 3:41 AM, Shahnawaz Saifi <sh...@gmail.com>
> wrote:
> > Hi,
> >
> > While uploading data to hbase memory usage getting too high (95%). we
> have a
> > cluster of seven nodes (hadoop master namenode, secondary namenode, hbase
> > master node, zookeeper node, slave datanode1, 2,3). Slave data nodes are
> > getting memory usage upto 99% and same time later some of them are
> > unresponsive.
> >
> > --
> > Thanks,
> > Shah
> >
>
--
Thanks,
Shah
Re: HRegionserver memory usage
Posted by Shahnawaz Saifi <sh...@gmail.com>.
we are using hbase-0.20.6
hbasemaster, master & secondary nn has 36GB of Memory, zookeeper & slave
datanodes got 16GB. And heap size is default in hbase-env.sh.
unresponsive: box is not accessible it hangs and shows java error on screen.
DN uses entire available resources. Client are able to store data into hbase
but not completely because job crashes as box goes out of memory and hangs
:(.
Thanks,
Shah
On Tue, Mar 22, 2011 at 8:58 PM, Stack <st...@duboce.net> wrote:
> Can you provide more detail on versions and memory sizes.
>
> In general, the JVM grows to occupy the allotted heap so your reports
> below about its using 95% and 99% could be just 'normal' behavior.
>
> When its 'unresponsive', what do the logs say? What do you mean by
> 'unresponsive'? DN won't take any load? Your client can't put any
> data into hbase?
>
> Thanks,
> St>Ack
>
> On Tue, Mar 22, 2011 at 3:41 AM, Shahnawaz Saifi <sh...@gmail.com>
> wrote:
> > Hi,
> >
> > While uploading data to hbase memory usage getting too high (95%). we
> have a
> > cluster of seven nodes (hadoop master namenode, secondary namenode, hbase
> > master node, zookeeper node, slave datanode1, 2,3). Slave data nodes are
> > getting memory usage upto 99% and same time later some of them are
> > unresponsive.
> >
> > --
> > Thanks,
> > Shah
> >
>
--
Thanks,
Shah
Re: HRegionserver memory usage
Posted by Stack <st...@duboce.net>.
Can you provide more detail on versions and memory sizes.
In general, the JVM grows to occupy the allotted heap so your reports
below about its using 95% and 99% could be just 'normal' behavior.
When its 'unresponsive', what do the logs say? What do you mean by
'unresponsive'? DN won't take any load? Your client can't put any
data into hbase?
Thanks,
St>Ack
On Tue, Mar 22, 2011 at 3:41 AM, Shahnawaz Saifi <sh...@gmail.com> wrote:
> Hi,
>
> While uploading data to hbase memory usage getting too high (95%). we have a
> cluster of seven nodes (hadoop master namenode, secondary namenode, hbase
> master node, zookeeper node, slave datanode1, 2,3). Slave data nodes are
> getting memory usage upto 99% and same time later some of them are
> unresponsive.
>
> --
> Thanks,
> Shah
>