You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-user@hadoop.apache.org by ch huang <ju...@gmail.com> on 2013/12/10 08:14:12 UTC
issue about datanode memory usage
hi,maillist:
i have a job running very slowly,when i examine cluster,i find
my hdfs user using 170m swap though top command,that's user run datanode
daemon ,ps output show following info,there are two -Xmx value ,and i do
not know which value is the real ,1000m or 10240m
# ps -ef|grep 2853
root 2095 1937 0 15:06 pts/4 00:00:00 grep 2853
hdfs 2853 1 5 Nov07 ? 1-22:34:22
/usr/java/jdk1.7.0_45/bin/java -Dproc_datanode -Xmx1000m
-Dhadoop.log.dir=/var/log/hadoop-hdfs
-Dhadoop.log.file=hadoop-hdfs-datanode-ch14.log
-Dhadoop.home.dir=/usr/lib/hadoop -Dhadoop.id.str=hdfs
-Dhadoop.root.logger=INFO,RFA
-Djava.library.path=/usr/lib/hadoop/lib/native
-Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true
-server -Xmx10240m -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps
-Xloggc:/var/log/hadoop-hdfs/gc-ch14-datanode.log
-Dhadoop.security.logger=INFO,RFAS
org.apache.hadoop.hdfs.server.datanode.DataNode
Re: issue about datanode memory usage
Posted by ch huang <ju...@gmail.com>.
so ,if i examine the output of "curl
http://CHBM224:50075/jmx<http://chbm224:50075/jmx>"
,it can sure which one will be used
{
"beans" : [ {
"name" : "java.lang:type=Memory",
"modelerType" : "sun.management.MemoryImpl",
"Verbose" : false,
"HeapMemoryUsage" : {
"committed" : 79953920,
"init" : 258443648,
"max" : 932118528,
"used" : 55585608
},
On Tue, Dec 10, 2013 at 3:18 PM, Pradeep Gollakota <pr...@gmail.com>wrote:
> Good question... answer is: it depends. I *think* Oracle JDK applies the
> rightmost. See this SO for more info.
> http://stackoverflow.com/questions/2740725/duplicated-java-runtime-options-what-is-the-order-of-preference
>
>
> On Mon, Dec 9, 2013 at 11:14 PM, ch huang <ju...@gmail.com> wrote:
>
>> hi,maillist:
>> i have a job running very slowly,when i examine cluster,i find
>> my hdfs user using 170m swap though top command,that's user run datanode
>> daemon ,ps output show following info,there are two -Xmx value ,and i do
>> not know which value is the real ,1000m or 10240m
>>
>> # ps -ef|grep 2853
>> root 2095 1937 0 15:06 pts/4 00:00:00 grep 2853
>> hdfs 2853 1 5 Nov07 ? 1-22:34:22
>> /usr/java/jdk1.7.0_45/bin/java -Dproc_datanode -Xmx1000m
>> -Dhadoop.log.dir=/var/log/hadoop-hdfs
>> -Dhadoop.log.file=hadoop-hdfs-datanode-ch14.log
>> -Dhadoop.home.dir=/usr/lib/hadoop -Dhadoop.id.str=hdfs
>> -Dhadoop.root.logger=INFO,RFA
>> -Djava.library.path=/usr/lib/hadoop/lib/native
>> -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true
>> -server -Xmx10240m -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps
>> -Xloggc:/var/log/hadoop-hdfs/gc-ch14-datanode.log
>> -Dhadoop.security.logger=INFO,RFAS
>> org.apache.hadoop.hdfs.server.datanode.DataNode
>>
>
>
Re: issue about datanode memory usage
Posted by ch huang <ju...@gmail.com>.
so ,if i examine the output of "curl
http://CHBM224:50075/jmx<http://chbm224:50075/jmx>"
,it can sure which one will be used
{
"beans" : [ {
"name" : "java.lang:type=Memory",
"modelerType" : "sun.management.MemoryImpl",
"Verbose" : false,
"HeapMemoryUsage" : {
"committed" : 79953920,
"init" : 258443648,
"max" : 932118528,
"used" : 55585608
},
On Tue, Dec 10, 2013 at 3:18 PM, Pradeep Gollakota <pr...@gmail.com>wrote:
> Good question... answer is: it depends. I *think* Oracle JDK applies the
> rightmost. See this SO for more info.
> http://stackoverflow.com/questions/2740725/duplicated-java-runtime-options-what-is-the-order-of-preference
>
>
> On Mon, Dec 9, 2013 at 11:14 PM, ch huang <ju...@gmail.com> wrote:
>
>> hi,maillist:
>> i have a job running very slowly,when i examine cluster,i find
>> my hdfs user using 170m swap though top command,that's user run datanode
>> daemon ,ps output show following info,there are two -Xmx value ,and i do
>> not know which value is the real ,1000m or 10240m
>>
>> # ps -ef|grep 2853
>> root 2095 1937 0 15:06 pts/4 00:00:00 grep 2853
>> hdfs 2853 1 5 Nov07 ? 1-22:34:22
>> /usr/java/jdk1.7.0_45/bin/java -Dproc_datanode -Xmx1000m
>> -Dhadoop.log.dir=/var/log/hadoop-hdfs
>> -Dhadoop.log.file=hadoop-hdfs-datanode-ch14.log
>> -Dhadoop.home.dir=/usr/lib/hadoop -Dhadoop.id.str=hdfs
>> -Dhadoop.root.logger=INFO,RFA
>> -Djava.library.path=/usr/lib/hadoop/lib/native
>> -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true
>> -server -Xmx10240m -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps
>> -Xloggc:/var/log/hadoop-hdfs/gc-ch14-datanode.log
>> -Dhadoop.security.logger=INFO,RFAS
>> org.apache.hadoop.hdfs.server.datanode.DataNode
>>
>
>
Re: issue about datanode memory usage
Posted by ch huang <ju...@gmail.com>.
so ,if i examine the output of "curl
http://CHBM224:50075/jmx<http://chbm224:50075/jmx>"
,it can sure which one will be used
{
"beans" : [ {
"name" : "java.lang:type=Memory",
"modelerType" : "sun.management.MemoryImpl",
"Verbose" : false,
"HeapMemoryUsage" : {
"committed" : 79953920,
"init" : 258443648,
"max" : 932118528,
"used" : 55585608
},
On Tue, Dec 10, 2013 at 3:18 PM, Pradeep Gollakota <pr...@gmail.com>wrote:
> Good question... answer is: it depends. I *think* Oracle JDK applies the
> rightmost. See this SO for more info.
> http://stackoverflow.com/questions/2740725/duplicated-java-runtime-options-what-is-the-order-of-preference
>
>
> On Mon, Dec 9, 2013 at 11:14 PM, ch huang <ju...@gmail.com> wrote:
>
>> hi,maillist:
>> i have a job running very slowly,when i examine cluster,i find
>> my hdfs user using 170m swap though top command,that's user run datanode
>> daemon ,ps output show following info,there are two -Xmx value ,and i do
>> not know which value is the real ,1000m or 10240m
>>
>> # ps -ef|grep 2853
>> root 2095 1937 0 15:06 pts/4 00:00:00 grep 2853
>> hdfs 2853 1 5 Nov07 ? 1-22:34:22
>> /usr/java/jdk1.7.0_45/bin/java -Dproc_datanode -Xmx1000m
>> -Dhadoop.log.dir=/var/log/hadoop-hdfs
>> -Dhadoop.log.file=hadoop-hdfs-datanode-ch14.log
>> -Dhadoop.home.dir=/usr/lib/hadoop -Dhadoop.id.str=hdfs
>> -Dhadoop.root.logger=INFO,RFA
>> -Djava.library.path=/usr/lib/hadoop/lib/native
>> -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true
>> -server -Xmx10240m -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps
>> -Xloggc:/var/log/hadoop-hdfs/gc-ch14-datanode.log
>> -Dhadoop.security.logger=INFO,RFAS
>> org.apache.hadoop.hdfs.server.datanode.DataNode
>>
>
>
Re: issue about datanode memory usage
Posted by ch huang <ju...@gmail.com>.
so ,if i examine the output of "curl
http://CHBM224:50075/jmx<http://chbm224:50075/jmx>"
,it can sure which one will be used
{
"beans" : [ {
"name" : "java.lang:type=Memory",
"modelerType" : "sun.management.MemoryImpl",
"Verbose" : false,
"HeapMemoryUsage" : {
"committed" : 79953920,
"init" : 258443648,
"max" : 932118528,
"used" : 55585608
},
On Tue, Dec 10, 2013 at 3:18 PM, Pradeep Gollakota <pr...@gmail.com>wrote:
> Good question... answer is: it depends. I *think* Oracle JDK applies the
> rightmost. See this SO for more info.
> http://stackoverflow.com/questions/2740725/duplicated-java-runtime-options-what-is-the-order-of-preference
>
>
> On Mon, Dec 9, 2013 at 11:14 PM, ch huang <ju...@gmail.com> wrote:
>
>> hi,maillist:
>> i have a job running very slowly,when i examine cluster,i find
>> my hdfs user using 170m swap though top command,that's user run datanode
>> daemon ,ps output show following info,there are two -Xmx value ,and i do
>> not know which value is the real ,1000m or 10240m
>>
>> # ps -ef|grep 2853
>> root 2095 1937 0 15:06 pts/4 00:00:00 grep 2853
>> hdfs 2853 1 5 Nov07 ? 1-22:34:22
>> /usr/java/jdk1.7.0_45/bin/java -Dproc_datanode -Xmx1000m
>> -Dhadoop.log.dir=/var/log/hadoop-hdfs
>> -Dhadoop.log.file=hadoop-hdfs-datanode-ch14.log
>> -Dhadoop.home.dir=/usr/lib/hadoop -Dhadoop.id.str=hdfs
>> -Dhadoop.root.logger=INFO,RFA
>> -Djava.library.path=/usr/lib/hadoop/lib/native
>> -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true
>> -server -Xmx10240m -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps
>> -Xloggc:/var/log/hadoop-hdfs/gc-ch14-datanode.log
>> -Dhadoop.security.logger=INFO,RFAS
>> org.apache.hadoop.hdfs.server.datanode.DataNode
>>
>
>
Re: issue about datanode memory usage
Posted by Pradeep Gollakota <pr...@gmail.com>.
Good question... answer is: it depends. I *think* Oracle JDK applies the
rightmost. See this SO for more info.
http://stackoverflow.com/questions/2740725/duplicated-java-runtime-options-what-is-the-order-of-preference
On Mon, Dec 9, 2013 at 11:14 PM, ch huang <ju...@gmail.com> wrote:
> hi,maillist:
> i have a job running very slowly,when i examine cluster,i find
> my hdfs user using 170m swap though top command,that's user run datanode
> daemon ,ps output show following info,there are two -Xmx value ,and i do
> not know which value is the real ,1000m or 10240m
>
> # ps -ef|grep 2853
> root 2095 1937 0 15:06 pts/4 00:00:00 grep 2853
> hdfs 2853 1 5 Nov07 ? 1-22:34:22
> /usr/java/jdk1.7.0_45/bin/java -Dproc_datanode -Xmx1000m
> -Dhadoop.log.dir=/var/log/hadoop-hdfs
> -Dhadoop.log.file=hadoop-hdfs-datanode-ch14.log
> -Dhadoop.home.dir=/usr/lib/hadoop -Dhadoop.id.str=hdfs
> -Dhadoop.root.logger=INFO,RFA
> -Djava.library.path=/usr/lib/hadoop/lib/native
> -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true
> -server -Xmx10240m -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps
> -Xloggc:/var/log/hadoop-hdfs/gc-ch14-datanode.log
> -Dhadoop.security.logger=INFO,RFAS
> org.apache.hadoop.hdfs.server.datanode.DataNode
>
Re: issue about datanode memory usage
Posted by Pradeep Gollakota <pr...@gmail.com>.
Good question... answer is: it depends. I *think* Oracle JDK applies the
rightmost. See this SO for more info.
http://stackoverflow.com/questions/2740725/duplicated-java-runtime-options-what-is-the-order-of-preference
On Mon, Dec 9, 2013 at 11:14 PM, ch huang <ju...@gmail.com> wrote:
> hi,maillist:
> i have a job running very slowly,when i examine cluster,i find
> my hdfs user using 170m swap though top command,that's user run datanode
> daemon ,ps output show following info,there are two -Xmx value ,and i do
> not know which value is the real ,1000m or 10240m
>
> # ps -ef|grep 2853
> root 2095 1937 0 15:06 pts/4 00:00:00 grep 2853
> hdfs 2853 1 5 Nov07 ? 1-22:34:22
> /usr/java/jdk1.7.0_45/bin/java -Dproc_datanode -Xmx1000m
> -Dhadoop.log.dir=/var/log/hadoop-hdfs
> -Dhadoop.log.file=hadoop-hdfs-datanode-ch14.log
> -Dhadoop.home.dir=/usr/lib/hadoop -Dhadoop.id.str=hdfs
> -Dhadoop.root.logger=INFO,RFA
> -Djava.library.path=/usr/lib/hadoop/lib/native
> -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true
> -server -Xmx10240m -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps
> -Xloggc:/var/log/hadoop-hdfs/gc-ch14-datanode.log
> -Dhadoop.security.logger=INFO,RFAS
> org.apache.hadoop.hdfs.server.datanode.DataNode
>
Re: issue about datanode memory usage
Posted by Pradeep Gollakota <pr...@gmail.com>.
Good question... answer is: it depends. I *think* Oracle JDK applies the
rightmost. See this SO for more info.
http://stackoverflow.com/questions/2740725/duplicated-java-runtime-options-what-is-the-order-of-preference
On Mon, Dec 9, 2013 at 11:14 PM, ch huang <ju...@gmail.com> wrote:
> hi,maillist:
> i have a job running very slowly,when i examine cluster,i find
> my hdfs user using 170m swap though top command,that's user run datanode
> daemon ,ps output show following info,there are two -Xmx value ,and i do
> not know which value is the real ,1000m or 10240m
>
> # ps -ef|grep 2853
> root 2095 1937 0 15:06 pts/4 00:00:00 grep 2853
> hdfs 2853 1 5 Nov07 ? 1-22:34:22
> /usr/java/jdk1.7.0_45/bin/java -Dproc_datanode -Xmx1000m
> -Dhadoop.log.dir=/var/log/hadoop-hdfs
> -Dhadoop.log.file=hadoop-hdfs-datanode-ch14.log
> -Dhadoop.home.dir=/usr/lib/hadoop -Dhadoop.id.str=hdfs
> -Dhadoop.root.logger=INFO,RFA
> -Djava.library.path=/usr/lib/hadoop/lib/native
> -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true
> -server -Xmx10240m -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps
> -Xloggc:/var/log/hadoop-hdfs/gc-ch14-datanode.log
> -Dhadoop.security.logger=INFO,RFAS
> org.apache.hadoop.hdfs.server.datanode.DataNode
>
Re: issue about datanode memory usage
Posted by Pradeep Gollakota <pr...@gmail.com>.
Good question... answer is: it depends. I *think* Oracle JDK applies the
rightmost. See this SO for more info.
http://stackoverflow.com/questions/2740725/duplicated-java-runtime-options-what-is-the-order-of-preference
On Mon, Dec 9, 2013 at 11:14 PM, ch huang <ju...@gmail.com> wrote:
> hi,maillist:
> i have a job running very slowly,when i examine cluster,i find
> my hdfs user using 170m swap though top command,that's user run datanode
> daemon ,ps output show following info,there are two -Xmx value ,and i do
> not know which value is the real ,1000m or 10240m
>
> # ps -ef|grep 2853
> root 2095 1937 0 15:06 pts/4 00:00:00 grep 2853
> hdfs 2853 1 5 Nov07 ? 1-22:34:22
> /usr/java/jdk1.7.0_45/bin/java -Dproc_datanode -Xmx1000m
> -Dhadoop.log.dir=/var/log/hadoop-hdfs
> -Dhadoop.log.file=hadoop-hdfs-datanode-ch14.log
> -Dhadoop.home.dir=/usr/lib/hadoop -Dhadoop.id.str=hdfs
> -Dhadoop.root.logger=INFO,RFA
> -Djava.library.path=/usr/lib/hadoop/lib/native
> -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true
> -server -Xmx10240m -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps
> -Xloggc:/var/log/hadoop-hdfs/gc-ch14-datanode.log
> -Dhadoop.security.logger=INFO,RFAS
> org.apache.hadoop.hdfs.server.datanode.DataNode
>