You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by "Buckley,Ron" <bu...@oclc.org> on 2013/01/22 22:17:46 UTC
Region server Memory Use is double the -Xmx setting
We have a 50 node cluster replicating to a 6 node cluster. Both clusters
are running CDH4.1.2 and HBase 0.94.2.
Today we noticed that the region servers at our replica site are using
10GB more memory than the '-Xmx12288m' we have defined in hbase-env.sh
These region servers have been up since January 9, 2013.
Does anyone have suggestions about tracking down this additional memory
use?
I'm not necessarily expecting the Region Server to stay right at the
12GB that we allocated, but having it running at 24GB is starting to
cause the servers to swap.
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
28544 prodcon 18 0 24.1g 23g 18m S 20.0 74.0 9071:34 java
28544: /usr/java/jdk1.6.0_31/bin/java -XX:OnOutOfMemoryError=kill -9
%p -Xmx12288m -Dcom.sun.management.jmxr
emote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.port=9021 -ea
-server -XX:+HeapDumpOnOutOfMemoryError -Xmn256m -XX:+UseParNewGC
-XX:+UseConcMarkSweepGC -XX:CMSInitiatingOc
cupancyFraction=70 -verbose:gc -XX:+PrintGCDetails
-XX:+PrintGCDateStamps -Xloggc:/drive1/hadoop/2.0/isoft/../
hbase/logs/gc-hbase.log -ea -server -XX:+HeapDumpOnOutOfMemoryError
-Xmn256m -XX
----------------------------------------------------------------------
Ron Buckley
x6365
http://intranet-wiki.oclc.org/wiki/XWC/XServe/SRU
http://intranet-wiki.oclc.org/wiki/FIND
http://intranet-wiki.oclc.org/wiki/Higgins
http://intranet-wiki.oclc.org/wiki/Firefly
RE: Region server Memory Use is double the -Xmx setting
Posted by "Buckley,Ron" <bu...@oclc.org>.
Anoop,
We use Snappy compression for all our tables.
I tried '-XX:MaxDirectMemorySize=1g' on our test cluster last night.
Ran great until major compactions started, then all region servers took 'java.lang.OutOfMemoryError: Direct buffer memory'
Trying '-XX:MaxDirectMemorySize=2g' tonight.
Ron
-----Original Message-----
From: Anoop Sam John [mailto:anoopsj@huawei.com]
Sent: Wednesday, January 23, 2013 10:22 PM
To: user@hbase.apache.org
Subject: RE: Region server Memory Use is double the -Xmx setting
Are you using compression for HFiles?
Yes we are using "MaxDirectMemorySize" and we dont use off-heap cache.
-Anoop-
________________________________________
From: Buckley,Ron [buckleyr@oclc.org]
Sent: Wednesday, January 23, 2013 8:49 PM
To: user@hbase.apache.org
Subject: RE: Region server Memory Use is double the -Xmx setting
Liang,
Thanks. I wasn’t really aware that the direct memory could get that large. (Full disclosure, we did switch from jdk1.6.0_25 to jdk1.6.0_31 the last time we restarted HBase.)
I've only seen explicit setting of -XX:MaxDirectMemorySize for regionservers associated with the experimental off-heap cache.
Is anyone else running their region servers with -XX:MaxDirectMemorySize (not using the off-heap cache)?
Ron
-----Original Message-----
From: 谢良 [mailto:xieliang@xiaomi.com]
Sent: Tuesday, January 22, 2013 9:20 PM
To: user@hbase.apache.org
Subject: 答复: Region server Memory Use is double the -Xmx setting
Please set "-XX:MaxDirectMemorySize" explicitly, else the default is taking the value like -Xmx in currenty JDK6, at least for jdk1.6.30+
Best Regards,
Liang
________________________________________
发件人: Buckley,Ron [buckleyr@oclc.org]
发送时间: 2013年1月23日 5:17
收件人: user@hbase.apache.org
主题: Region server Memory Use is double the -Xmx setting
We have a 50 node cluster replicating to a 6 node cluster. Both clusters are running CDH4.1.2 and HBase 0.94.2.
Today we noticed that the region servers at our replica site are using 10GB more memory than the '-Xmx12288m' we have defined in hbase-env.sh
These region servers have been up since January 9, 2013.
Does anyone have suggestions about tracking down this additional memory use?
I'm not necessarily expecting the Region Server to stay right at the 12GB that we allocated, but having it running at 24GB is starting to cause the servers to swap.
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
28544 prodcon 18 0 24.1g 23g 18m S 20.0 74.0 9071:34 java
28544: /usr/java/jdk1.6.0_31/bin/java -XX:OnOutOfMemoryError=kill -9
%p -Xmx12288m -Dcom.sun.management.jmxr
emote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.port=9021 -ea
-server -XX:+HeapDumpOnOutOfMemoryError -Xmn256m -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOc
cupancyFraction=70 -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:/drive1/hadoop/2.0/isoft/../
hbase/logs/gc-hbase.log -ea -server -XX:+HeapDumpOnOutOfMemoryError -Xmn256m -XX
----------------------------------------------------------------------
Ron Buckley
x6365
http://intranet-wiki.oclc.org/wiki/XWC/XServe/SRU
http://intranet-wiki.oclc.org/wiki/FIND
http://intranet-wiki.oclc.org/wiki/Higgins
http://intranet-wiki.oclc.org/wiki/Firefly
RE: Region server Memory Use is double the -Xmx setting
Posted by Anoop Sam John <an...@huawei.com>.
Are you using compression for HFiles?
Yes we are using "MaxDirectMemorySize" and we dont use off-heap cache.
-Anoop-
________________________________________
From: Buckley,Ron [buckleyr@oclc.org]
Sent: Wednesday, January 23, 2013 8:49 PM
To: user@hbase.apache.org
Subject: RE: Region server Memory Use is double the -Xmx setting
Liang,
Thanks. I wasn’t really aware that the direct memory could get that large. (Full disclosure, we did switch from jdk1.6.0_25 to jdk1.6.0_31 the last time we restarted HBase.)
I've only seen explicit setting of -XX:MaxDirectMemorySize for regionservers associated with the experimental off-heap cache.
Is anyone else running their region servers with -XX:MaxDirectMemorySize (not using the off-heap cache)?
Ron
-----Original Message-----
From: 谢良 [mailto:xieliang@xiaomi.com]
Sent: Tuesday, January 22, 2013 9:20 PM
To: user@hbase.apache.org
Subject: 答复: Region server Memory Use is double the -Xmx setting
Please set "-XX:MaxDirectMemorySize" explicitly, else the default is taking the value like -Xmx in currenty JDK6, at least for jdk1.6.30+
Best Regards,
Liang
________________________________________
发件人: Buckley,Ron [buckleyr@oclc.org]
发送时间: 2013年1月23日 5:17
收件人: user@hbase.apache.org
主题: Region server Memory Use is double the -Xmx setting
We have a 50 node cluster replicating to a 6 node cluster. Both clusters
are running CDH4.1.2 and HBase 0.94.2.
Today we noticed that the region servers at our replica site are using
10GB more memory than the '-Xmx12288m' we have defined in hbase-env.sh
These region servers have been up since January 9, 2013.
Does anyone have suggestions about tracking down this additional memory
use?
I'm not necessarily expecting the Region Server to stay right at the
12GB that we allocated, but having it running at 24GB is starting to
cause the servers to swap.
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
28544 prodcon 18 0 24.1g 23g 18m S 20.0 74.0 9071:34 java
28544: /usr/java/jdk1.6.0_31/bin/java -XX:OnOutOfMemoryError=kill -9
%p -Xmx12288m -Dcom.sun.management.jmxr
emote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.port=9021 -ea
-server -XX:+HeapDumpOnOutOfMemoryError -Xmn256m -XX:+UseParNewGC
-XX:+UseConcMarkSweepGC -XX:CMSInitiatingOc
cupancyFraction=70 -verbose:gc -XX:+PrintGCDetails
-XX:+PrintGCDateStamps -Xloggc:/drive1/hadoop/2.0/isoft/../
hbase/logs/gc-hbase.log -ea -server -XX:+HeapDumpOnOutOfMemoryError
-Xmn256m -XX
----------------------------------------------------------------------
Ron Buckley
x6365
http://intranet-wiki.oclc.org/wiki/XWC/XServe/SRU
http://intranet-wiki.oclc.org/wiki/FIND
http://intranet-wiki.oclc.org/wiki/Higgins
http://intranet-wiki.oclc.org/wiki/Firefly
RE: Region server Memory Use is double the -Xmx setting
Posted by "Buckley,Ron" <bu...@oclc.org>.
Liang,
Thanks. I wasn’t really aware that the direct memory could get that large. (Full disclosure, we did switch from jdk1.6.0_25 to jdk1.6.0_31 the last time we restarted HBase.)
I've only seen explicit setting of -XX:MaxDirectMemorySize for regionservers associated with the experimental off-heap cache.
Is anyone else running their region servers with -XX:MaxDirectMemorySize (not using the off-heap cache)?
Ron
-----Original Message-----
From: 谢良 [mailto:xieliang@xiaomi.com]
Sent: Tuesday, January 22, 2013 9:20 PM
To: user@hbase.apache.org
Subject: 答复: Region server Memory Use is double the -Xmx setting
Please set "-XX:MaxDirectMemorySize" explicitly, else the default is taking the value like -Xmx in currenty JDK6, at least for jdk1.6.30+
Best Regards,
Liang
________________________________________
发件人: Buckley,Ron [buckleyr@oclc.org]
发送时间: 2013年1月23日 5:17
收件人: user@hbase.apache.org
主题: Region server Memory Use is double the -Xmx setting
We have a 50 node cluster replicating to a 6 node cluster. Both clusters
are running CDH4.1.2 and HBase 0.94.2.
Today we noticed that the region servers at our replica site are using
10GB more memory than the '-Xmx12288m' we have defined in hbase-env.sh
These region servers have been up since January 9, 2013.
Does anyone have suggestions about tracking down this additional memory
use?
I'm not necessarily expecting the Region Server to stay right at the
12GB that we allocated, but having it running at 24GB is starting to
cause the servers to swap.
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
28544 prodcon 18 0 24.1g 23g 18m S 20.0 74.0 9071:34 java
28544: /usr/java/jdk1.6.0_31/bin/java -XX:OnOutOfMemoryError=kill -9
%p -Xmx12288m -Dcom.sun.management.jmxr
emote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.port=9021 -ea
-server -XX:+HeapDumpOnOutOfMemoryError -Xmn256m -XX:+UseParNewGC
-XX:+UseConcMarkSweepGC -XX:CMSInitiatingOc
cupancyFraction=70 -verbose:gc -XX:+PrintGCDetails
-XX:+PrintGCDateStamps -Xloggc:/drive1/hadoop/2.0/isoft/../
hbase/logs/gc-hbase.log -ea -server -XX:+HeapDumpOnOutOfMemoryError
-Xmn256m -XX
----------------------------------------------------------------------
Ron Buckley
x6365
http://intranet-wiki.oclc.org/wiki/XWC/XServe/SRU
http://intranet-wiki.oclc.org/wiki/FIND
http://intranet-wiki.oclc.org/wiki/Higgins
http://intranet-wiki.oclc.org/wiki/Firefly
答复: Region server Memory Use is double the -Xmx setting
Posted by 谢良 <xi...@xiaomi.com>.
Please set "-XX:MaxDirectMemorySize" explicitly, else the default is taking the value like -Xmx in currenty JDK6, at least for jdk1.6.30+
Best Regards,
Liang
________________________________________
发件人: Buckley,Ron [buckleyr@oclc.org]
发送时间: 2013年1月23日 5:17
收件人: user@hbase.apache.org
主题: Region server Memory Use is double the -Xmx setting
We have a 50 node cluster replicating to a 6 node cluster. Both clusters
are running CDH4.1.2 and HBase 0.94.2.
Today we noticed that the region servers at our replica site are using
10GB more memory than the '-Xmx12288m' we have defined in hbase-env.sh
These region servers have been up since January 9, 2013.
Does anyone have suggestions about tracking down this additional memory
use?
I'm not necessarily expecting the Region Server to stay right at the
12GB that we allocated, but having it running at 24GB is starting to
cause the servers to swap.
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
28544 prodcon 18 0 24.1g 23g 18m S 20.0 74.0 9071:34 java
28544: /usr/java/jdk1.6.0_31/bin/java -XX:OnOutOfMemoryError=kill -9
%p -Xmx12288m -Dcom.sun.management.jmxr
emote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.port=9021 -ea
-server -XX:+HeapDumpOnOutOfMemoryError -Xmn256m -XX:+UseParNewGC
-XX:+UseConcMarkSweepGC -XX:CMSInitiatingOc
cupancyFraction=70 -verbose:gc -XX:+PrintGCDetails
-XX:+PrintGCDateStamps -Xloggc:/drive1/hadoop/2.0/isoft/../
hbase/logs/gc-hbase.log -ea -server -XX:+HeapDumpOnOutOfMemoryError
-Xmn256m -XX
----------------------------------------------------------------------
Ron Buckley
x6365
http://intranet-wiki.oclc.org/wiki/XWC/XServe/SRU
http://intranet-wiki.oclc.org/wiki/FIND
http://intranet-wiki.oclc.org/wiki/Higgins
http://intranet-wiki.oclc.org/wiki/Firefly