You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by ch huang <ju...@gmail.com> on 2013/11/26 07:15:43 UTC
issue about set the memory for each container
hi,maillist:
i find each my container just use 200M heap space,how can i
resize it?
# ps -ef|grep -i yarnchild
yarn 24333 8210 99 14:09 ? 00:00:05
/usr/java/jdk1.7.0_25/bin/java -Djava.net.preferIPv4Stack=true
-Dhadoop.metrics.log.level=WARN -Xmx200m
-Djava.io.tmpdir=/data/mrlocal/1/yarn/local/usercache/hdfs/appcache/application_1385445543402_0006/container_1385445543402_0006_01_000017/tmp
-Dlog4j.configuration=container-log4j.properties
-Dyarn.app.mapreduce.container.log.dir=/data/mrlocal/2/yarn/logs/application_1385445543402_0006/container_1385445543402_0006_01_000017
-Dyarn.app.mapreduce.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA
org.apache.hadoop.mapred.YarnChild 192.168.10.224 59237
attempt_1385445543402_0006_m_000015_0 17
Re: issue about set the memory for each container
Posted by Adam Kawa <ka...@gmail.com>.
For MapReduce jobs, you should use you configuration options in
mapred-site.xml or Configuration object in your Driver class.
mapreduce.map.memory.mb1536Larger resource limit for maps.
mapreduce.map.java.opts-Xmx1024MLarger heap-size for child jvms of maps.
mapreduce.reduce.memory.mb3072Larger resource limit for reduces.
mapreduce.reduce.java.opts-Xmx2560MLarger heap-size for child jvms of
reduces.
You can read more here
http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/ClusterSetup.html#Configuring_the_Hadoop_Daemons_in_Non-Secure_Mode
and
http://grokbase.com/t/hadoop/hdfs-user/1324x8nt6d/how-mapred-map-child-java-opts-influence-the-container-resource-allocation
2013/11/26 ch huang <ju...@gmail.com>
> hi,maillist:
> i find each my container just use 200M heap space,how can i
> resize it?
>
> # ps -ef|grep -i yarnchild
> yarn 24333 8210 99 14:09 ? 00:00:05
> /usr/java/jdk1.7.0_25/bin/java -Djava.net.preferIPv4Stack=true
> -Dhadoop.metrics.log.level=WARN -Xmx200m
> -Djava.io.tmpdir=/data/mrlocal/1/yarn/local/usercache/hdfs/appcache/application_1385445543402_0006/container_1385445543402_0006_01_000017/tmp
> -Dlog4j.configuration=container-log4j.properties
> -Dyarn.app.mapreduce.container.log.dir=/data/mrlocal/2/yarn/logs/application_1385445543402_0006/container_1385445543402_0006_01_000017
> -Dyarn.app.mapreduce.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA
> org.apache.hadoop.mapred.YarnChild 192.168.10.224 59237
> attempt_1385445543402_0006_m_000015_0 17
>
Re: issue about set the memory for each container
Posted by Adam Kawa <ka...@gmail.com>.
For MapReduce jobs, you should use you configuration options in
mapred-site.xml or Configuration object in your Driver class.
mapreduce.map.memory.mb1536Larger resource limit for maps.
mapreduce.map.java.opts-Xmx1024MLarger heap-size for child jvms of maps.
mapreduce.reduce.memory.mb3072Larger resource limit for reduces.
mapreduce.reduce.java.opts-Xmx2560MLarger heap-size for child jvms of
reduces.
You can read more here
http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/ClusterSetup.html#Configuring_the_Hadoop_Daemons_in_Non-Secure_Mode
and
http://grokbase.com/t/hadoop/hdfs-user/1324x8nt6d/how-mapred-map-child-java-opts-influence-the-container-resource-allocation
2013/11/26 ch huang <ju...@gmail.com>
> hi,maillist:
> i find each my container just use 200M heap space,how can i
> resize it?
>
> # ps -ef|grep -i yarnchild
> yarn 24333 8210 99 14:09 ? 00:00:05
> /usr/java/jdk1.7.0_25/bin/java -Djava.net.preferIPv4Stack=true
> -Dhadoop.metrics.log.level=WARN -Xmx200m
> -Djava.io.tmpdir=/data/mrlocal/1/yarn/local/usercache/hdfs/appcache/application_1385445543402_0006/container_1385445543402_0006_01_000017/tmp
> -Dlog4j.configuration=container-log4j.properties
> -Dyarn.app.mapreduce.container.log.dir=/data/mrlocal/2/yarn/logs/application_1385445543402_0006/container_1385445543402_0006_01_000017
> -Dyarn.app.mapreduce.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA
> org.apache.hadoop.mapred.YarnChild 192.168.10.224 59237
> attempt_1385445543402_0006_m_000015_0 17
>
Re: issue about set the memory for each container
Posted by Adam Kawa <ka...@gmail.com>.
For MapReduce jobs, you should use you configuration options in
mapred-site.xml or Configuration object in your Driver class.
mapreduce.map.memory.mb1536Larger resource limit for maps.
mapreduce.map.java.opts-Xmx1024MLarger heap-size for child jvms of maps.
mapreduce.reduce.memory.mb3072Larger resource limit for reduces.
mapreduce.reduce.java.opts-Xmx2560MLarger heap-size for child jvms of
reduces.
You can read more here
http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/ClusterSetup.html#Configuring_the_Hadoop_Daemons_in_Non-Secure_Mode
and
http://grokbase.com/t/hadoop/hdfs-user/1324x8nt6d/how-mapred-map-child-java-opts-influence-the-container-resource-allocation
2013/11/26 ch huang <ju...@gmail.com>
> hi,maillist:
> i find each my container just use 200M heap space,how can i
> resize it?
>
> # ps -ef|grep -i yarnchild
> yarn 24333 8210 99 14:09 ? 00:00:05
> /usr/java/jdk1.7.0_25/bin/java -Djava.net.preferIPv4Stack=true
> -Dhadoop.metrics.log.level=WARN -Xmx200m
> -Djava.io.tmpdir=/data/mrlocal/1/yarn/local/usercache/hdfs/appcache/application_1385445543402_0006/container_1385445543402_0006_01_000017/tmp
> -Dlog4j.configuration=container-log4j.properties
> -Dyarn.app.mapreduce.container.log.dir=/data/mrlocal/2/yarn/logs/application_1385445543402_0006/container_1385445543402_0006_01_000017
> -Dyarn.app.mapreduce.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA
> org.apache.hadoop.mapred.YarnChild 192.168.10.224 59237
> attempt_1385445543402_0006_m_000015_0 17
>
Re: issue about set the memory for each container
Posted by Adam Kawa <ka...@gmail.com>.
For MapReduce jobs, you should use you configuration options in
mapred-site.xml or Configuration object in your Driver class.
mapreduce.map.memory.mb1536Larger resource limit for maps.
mapreduce.map.java.opts-Xmx1024MLarger heap-size for child jvms of maps.
mapreduce.reduce.memory.mb3072Larger resource limit for reduces.
mapreduce.reduce.java.opts-Xmx2560MLarger heap-size for child jvms of
reduces.
You can read more here
http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/ClusterSetup.html#Configuring_the_Hadoop_Daemons_in_Non-Secure_Mode
and
http://grokbase.com/t/hadoop/hdfs-user/1324x8nt6d/how-mapred-map-child-java-opts-influence-the-container-resource-allocation
2013/11/26 ch huang <ju...@gmail.com>
> hi,maillist:
> i find each my container just use 200M heap space,how can i
> resize it?
>
> # ps -ef|grep -i yarnchild
> yarn 24333 8210 99 14:09 ? 00:00:05
> /usr/java/jdk1.7.0_25/bin/java -Djava.net.preferIPv4Stack=true
> -Dhadoop.metrics.log.level=WARN -Xmx200m
> -Djava.io.tmpdir=/data/mrlocal/1/yarn/local/usercache/hdfs/appcache/application_1385445543402_0006/container_1385445543402_0006_01_000017/tmp
> -Dlog4j.configuration=container-log4j.properties
> -Dyarn.app.mapreduce.container.log.dir=/data/mrlocal/2/yarn/logs/application_1385445543402_0006/container_1385445543402_0006_01_000017
> -Dyarn.app.mapreduce.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA
> org.apache.hadoop.mapred.YarnChild 192.168.10.224 59237
> attempt_1385445543402_0006_m_000015_0 17
>