You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by Fei Hu <hu...@gmail.com> on 2015/03/22 04:43:07 UTC
Container beyond virtual memory limits
Hi,
I just test my yarn installation, and run a Wordcount program. But it always report the following error, who knows how to solve it? Thank you in advance.
Container [pid=7954,containerID=container_1426992254950_0002_01_000005] is running beyond virtual memory limits. Current usage: 13.6 MB of 1 GB physical memory used; 4.3 GB of 2.1 GB virtual memory used. Killing container.
Dump of the process-tree for container_1426992254950_0002_01_000005 :
|- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
|- 7960 7954 7954 7954 (java) 5 0 4576591872 3199 /usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/jre/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN 1638 -Djava.io.tmpdir=/tmp/hadoop-root/nm-local-dir/usercache/root/appcache/application_1426992254950_0002/container_1426992254950_0002_01_000005/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/home/hadoop-lzl/hadoop-2.6.0/logs/userlogs/application_1426992254950_0002/container_1426992254950_0002_01_000005 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA org.apache.hadoop.mapred.YarnChild 199.26.254.140 36542 attempt_1426992254950_0002_m_000003_0 5
|- 7954 7949 7954 7954 (bash) 0 0 65421312 275 /bin/bash -c /usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/jre/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN 1638 -Djava.io.tmpdir=/tmp/hadoop-root/nm-local-dir/usercache/root/appcache/application_1426992254950_0002/container_1426992254950_0002_01_000005/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/home/hadoop-lzl/hadoop-2.6.0/logs/userlogs/application_1426992254950_0002/container_1426992254950_0002_01_000005 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA org.apache.hadoop.mapred.YarnChild 199.26.254.140 36542 attempt_1426992254950_0002_m_000003_0 5 1>/home/hadoop-lzl/hadoop-2.6.0/logs/userlogs/application_1426992254950_0002/container_1426992254950_0002_01_000005/stdout 2>/home/hadoop-lzl/hadoop-2.6.0/logs/userlogs/application_1426992254950_0002/container_1426992254950_0002_01_000005/stderr
Exception from container-launch.
Container id: container_1426992254950_0002_01_000005
Exit code: 1
Stack trace: ExitCodeException exitCode=1:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
at org.apache.hadoop.util.Shell.run(Shell.java:455)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:715)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:211)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
Thanks,
Fei
Re: Container beyond virtual memory limits
Posted by Fei Hu <hu...@gmail.com>.
Thank you. It works.
Best,
Fei
> On Mar 23, 2015, at 11:27 AM, Gaurav Gupta <ga...@gmail.com> wrote:
>
> Increasing vmem:pmem ratio should help you out. Default value is 2:1, change it to 5:1
>
> On Sun, Mar 22, 2015 at 10:09 PM, Drake민영근 <drake.min@nexr.com <ma...@nexr.com>> wrote:
> Hi,
>
> See "6. Killing of Tasks Due to Virtual Memory Usage" in http://blog.cloudera.com/blog/2014/04/apache-hadoop-yarn-avoiding-6-time-consuming-gotchas/ <http://blog.cloudera.com/blog/2014/04/apache-hadoop-yarn-avoiding-6-time-consuming-gotchas/>
>
> Drake 민영근 Ph.D
> kt NexR
>
> On Sun, Mar 22, 2015 at 12:43 PM, Fei Hu <hufei68@gmail.com <ma...@gmail.com>> wrote:
> Hi,
>
> I just test my yarn installation, and run a Wordcount program. But it always report the following error, who knows how to solve it? Thank you in advance.
>
> Container [pid=7954,containerID=container_1426992254950_0002_01_000005] is running beyond virtual memory limits. Current usage: 13.6 MB of 1 GB physical memory used; 4.3 GB of 2.1 GB virtual memory used. Killing container.
> Dump of the process-tree for container_1426992254950_0002_01_000005 :
> |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
> |- 7960 7954 7954 7954 (java) 5 0 4576591872 3199 /usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/jre/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN 1638 -Djava.io.tmpdir=/tmp/hadoop-root/nm-local-dir/usercache/root/appcache/application_1426992254950_0002/container_1426992254950_0002_01_000005/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/home/hadoop-lzl/hadoop-2.6.0/logs/userlogs/application_1426992254950_0002/container_1426992254950_0002_01_000005 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA org.apache.hadoop.mapred.YarnChild 199.26.254.140 36542 attempt_1426992254950_0002_m_000003_0 5
> |- 7954 7949 7954 7954 (bash) 0 0 65421312 275 /bin/bash -c /usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/jre/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN 1638 -Djava.io.tmpdir=/tmp/hadoop-root/nm-local-dir/usercache/root/appcache/application_1426992254950_0002/container_1426992254950_0002_01_000005/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/home/hadoop-lzl/hadoop-2.6.0/logs/userlogs/application_1426992254950_0002/container_1426992254950_0002_01_000005 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA org.apache.hadoop.mapred.YarnChild 199.26.254.140 36542 attempt_1426992254950_0002_m_000003_0 5 1>/home/hadoop-lzl/hadoop-2.6.0/logs/userlogs/application_1426992254950_0002/container_1426992254950_0002_01_000005/stdout 2>/home/hadoop-lzl/hadoop-2.6.0/logs/userlogs/application_1426992254950_0002/container_1426992254950_0002_01_000005/stderr
>
> Exception from container-launch.
> Container id: container_1426992254950_0002_01_000005
> Exit code: 1
> Stack trace: ExitCodeException exitCode=1:
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
> at org.apache.hadoop.util.Shell.run(Shell.java:455)
> at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:715)
> at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:211)
> at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
> at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
>
> Thanks,
> Fei
>
>
Re: Container beyond virtual memory limits
Posted by Fei Hu <hu...@gmail.com>.
Thank you. It works.
Best,
Fei
> On Mar 23, 2015, at 11:27 AM, Gaurav Gupta <ga...@gmail.com> wrote:
>
> Increasing vmem:pmem ratio should help you out. Default value is 2:1, change it to 5:1
>
> On Sun, Mar 22, 2015 at 10:09 PM, Drake민영근 <drake.min@nexr.com <ma...@nexr.com>> wrote:
> Hi,
>
> See "6. Killing of Tasks Due to Virtual Memory Usage" in http://blog.cloudera.com/blog/2014/04/apache-hadoop-yarn-avoiding-6-time-consuming-gotchas/ <http://blog.cloudera.com/blog/2014/04/apache-hadoop-yarn-avoiding-6-time-consuming-gotchas/>
>
> Drake 민영근 Ph.D
> kt NexR
>
> On Sun, Mar 22, 2015 at 12:43 PM, Fei Hu <hufei68@gmail.com <ma...@gmail.com>> wrote:
> Hi,
>
> I just test my yarn installation, and run a Wordcount program. But it always report the following error, who knows how to solve it? Thank you in advance.
>
> Container [pid=7954,containerID=container_1426992254950_0002_01_000005] is running beyond virtual memory limits. Current usage: 13.6 MB of 1 GB physical memory used; 4.3 GB of 2.1 GB virtual memory used. Killing container.
> Dump of the process-tree for container_1426992254950_0002_01_000005 :
> |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
> |- 7960 7954 7954 7954 (java) 5 0 4576591872 3199 /usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/jre/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN 1638 -Djava.io.tmpdir=/tmp/hadoop-root/nm-local-dir/usercache/root/appcache/application_1426992254950_0002/container_1426992254950_0002_01_000005/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/home/hadoop-lzl/hadoop-2.6.0/logs/userlogs/application_1426992254950_0002/container_1426992254950_0002_01_000005 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA org.apache.hadoop.mapred.YarnChild 199.26.254.140 36542 attempt_1426992254950_0002_m_000003_0 5
> |- 7954 7949 7954 7954 (bash) 0 0 65421312 275 /bin/bash -c /usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/jre/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN 1638 -Djava.io.tmpdir=/tmp/hadoop-root/nm-local-dir/usercache/root/appcache/application_1426992254950_0002/container_1426992254950_0002_01_000005/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/home/hadoop-lzl/hadoop-2.6.0/logs/userlogs/application_1426992254950_0002/container_1426992254950_0002_01_000005 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA org.apache.hadoop.mapred.YarnChild 199.26.254.140 36542 attempt_1426992254950_0002_m_000003_0 5 1>/home/hadoop-lzl/hadoop-2.6.0/logs/userlogs/application_1426992254950_0002/container_1426992254950_0002_01_000005/stdout 2>/home/hadoop-lzl/hadoop-2.6.0/logs/userlogs/application_1426992254950_0002/container_1426992254950_0002_01_000005/stderr
>
> Exception from container-launch.
> Container id: container_1426992254950_0002_01_000005
> Exit code: 1
> Stack trace: ExitCodeException exitCode=1:
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
> at org.apache.hadoop.util.Shell.run(Shell.java:455)
> at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:715)
> at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:211)
> at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
> at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
>
> Thanks,
> Fei
>
>
Re: Container beyond virtual memory limits
Posted by Fei Hu <hu...@gmail.com>.
Thank you. It works.
Best,
Fei
> On Mar 23, 2015, at 11:27 AM, Gaurav Gupta <ga...@gmail.com> wrote:
>
> Increasing vmem:pmem ratio should help you out. Default value is 2:1, change it to 5:1
>
> On Sun, Mar 22, 2015 at 10:09 PM, Drake민영근 <drake.min@nexr.com <ma...@nexr.com>> wrote:
> Hi,
>
> See "6. Killing of Tasks Due to Virtual Memory Usage" in http://blog.cloudera.com/blog/2014/04/apache-hadoop-yarn-avoiding-6-time-consuming-gotchas/ <http://blog.cloudera.com/blog/2014/04/apache-hadoop-yarn-avoiding-6-time-consuming-gotchas/>
>
> Drake 민영근 Ph.D
> kt NexR
>
> On Sun, Mar 22, 2015 at 12:43 PM, Fei Hu <hufei68@gmail.com <ma...@gmail.com>> wrote:
> Hi,
>
> I just test my yarn installation, and run a Wordcount program. But it always report the following error, who knows how to solve it? Thank you in advance.
>
> Container [pid=7954,containerID=container_1426992254950_0002_01_000005] is running beyond virtual memory limits. Current usage: 13.6 MB of 1 GB physical memory used; 4.3 GB of 2.1 GB virtual memory used. Killing container.
> Dump of the process-tree for container_1426992254950_0002_01_000005 :
> |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
> |- 7960 7954 7954 7954 (java) 5 0 4576591872 3199 /usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/jre/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN 1638 -Djava.io.tmpdir=/tmp/hadoop-root/nm-local-dir/usercache/root/appcache/application_1426992254950_0002/container_1426992254950_0002_01_000005/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/home/hadoop-lzl/hadoop-2.6.0/logs/userlogs/application_1426992254950_0002/container_1426992254950_0002_01_000005 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA org.apache.hadoop.mapred.YarnChild 199.26.254.140 36542 attempt_1426992254950_0002_m_000003_0 5
> |- 7954 7949 7954 7954 (bash) 0 0 65421312 275 /bin/bash -c /usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/jre/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN 1638 -Djava.io.tmpdir=/tmp/hadoop-root/nm-local-dir/usercache/root/appcache/application_1426992254950_0002/container_1426992254950_0002_01_000005/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/home/hadoop-lzl/hadoop-2.6.0/logs/userlogs/application_1426992254950_0002/container_1426992254950_0002_01_000005 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA org.apache.hadoop.mapred.YarnChild 199.26.254.140 36542 attempt_1426992254950_0002_m_000003_0 5 1>/home/hadoop-lzl/hadoop-2.6.0/logs/userlogs/application_1426992254950_0002/container_1426992254950_0002_01_000005/stdout 2>/home/hadoop-lzl/hadoop-2.6.0/logs/userlogs/application_1426992254950_0002/container_1426992254950_0002_01_000005/stderr
>
> Exception from container-launch.
> Container id: container_1426992254950_0002_01_000005
> Exit code: 1
> Stack trace: ExitCodeException exitCode=1:
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
> at org.apache.hadoop.util.Shell.run(Shell.java:455)
> at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:715)
> at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:211)
> at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
> at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
>
> Thanks,
> Fei
>
>
Re: Container beyond virtual memory limits
Posted by Fei Hu <hu...@gmail.com>.
Thank you. It works.
Best,
Fei
> On Mar 23, 2015, at 11:27 AM, Gaurav Gupta <ga...@gmail.com> wrote:
>
> Increasing vmem:pmem ratio should help you out. Default value is 2:1, change it to 5:1
>
> On Sun, Mar 22, 2015 at 10:09 PM, Drake민영근 <drake.min@nexr.com <ma...@nexr.com>> wrote:
> Hi,
>
> See "6. Killing of Tasks Due to Virtual Memory Usage" in http://blog.cloudera.com/blog/2014/04/apache-hadoop-yarn-avoiding-6-time-consuming-gotchas/ <http://blog.cloudera.com/blog/2014/04/apache-hadoop-yarn-avoiding-6-time-consuming-gotchas/>
>
> Drake 민영근 Ph.D
> kt NexR
>
> On Sun, Mar 22, 2015 at 12:43 PM, Fei Hu <hufei68@gmail.com <ma...@gmail.com>> wrote:
> Hi,
>
> I just test my yarn installation, and run a Wordcount program. But it always report the following error, who knows how to solve it? Thank you in advance.
>
> Container [pid=7954,containerID=container_1426992254950_0002_01_000005] is running beyond virtual memory limits. Current usage: 13.6 MB of 1 GB physical memory used; 4.3 GB of 2.1 GB virtual memory used. Killing container.
> Dump of the process-tree for container_1426992254950_0002_01_000005 :
> |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
> |- 7960 7954 7954 7954 (java) 5 0 4576591872 3199 /usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/jre/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN 1638 -Djava.io.tmpdir=/tmp/hadoop-root/nm-local-dir/usercache/root/appcache/application_1426992254950_0002/container_1426992254950_0002_01_000005/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/home/hadoop-lzl/hadoop-2.6.0/logs/userlogs/application_1426992254950_0002/container_1426992254950_0002_01_000005 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA org.apache.hadoop.mapred.YarnChild 199.26.254.140 36542 attempt_1426992254950_0002_m_000003_0 5
> |- 7954 7949 7954 7954 (bash) 0 0 65421312 275 /bin/bash -c /usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/jre/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN 1638 -Djava.io.tmpdir=/tmp/hadoop-root/nm-local-dir/usercache/root/appcache/application_1426992254950_0002/container_1426992254950_0002_01_000005/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/home/hadoop-lzl/hadoop-2.6.0/logs/userlogs/application_1426992254950_0002/container_1426992254950_0002_01_000005 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA org.apache.hadoop.mapred.YarnChild 199.26.254.140 36542 attempt_1426992254950_0002_m_000003_0 5 1>/home/hadoop-lzl/hadoop-2.6.0/logs/userlogs/application_1426992254950_0002/container_1426992254950_0002_01_000005/stdout 2>/home/hadoop-lzl/hadoop-2.6.0/logs/userlogs/application_1426992254950_0002/container_1426992254950_0002_01_000005/stderr
>
> Exception from container-launch.
> Container id: container_1426992254950_0002_01_000005
> Exit code: 1
> Stack trace: ExitCodeException exitCode=1:
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
> at org.apache.hadoop.util.Shell.run(Shell.java:455)
> at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:715)
> at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:211)
> at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
> at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
>
> Thanks,
> Fei
>
>
Re: Container beyond virtual memory limits
Posted by Gaurav Gupta <ga...@gmail.com>.
Increasing vmem:pmem ratio should help you out. Default value is 2:1,
change it to 5:1
On Sun, Mar 22, 2015 at 10:09 PM, Drake민영근 <dr...@nexr.com> wrote:
> Hi,
>
> See "6. Killing of Tasks Due to Virtual Memory Usage" in
> http://blog.cloudera.com/blog/2014/04/apache-hadoop-yarn-avoiding-6-time-consuming-gotchas/
>
>
> Drake 민영근 Ph.D
> kt NexR
>
> On Sun, Mar 22, 2015 at 12:43 PM, Fei Hu <hu...@gmail.com> wrote:
>
>> Hi,
>>
>> I just test my yarn installation, and run a Wordcount program. But it
>> always report the following error, who knows how to solve it? Thank you in
>> advance.
>>
>> Container [pid=7954,containerID=container_1426992254950_0002_01_000005]
>> is running beyond virtual memory limits. Current usage: 13.6 MB of 1 GB
>> physical memory used; 4.3 GB of 2.1 GB virtual memory used. Killing
>> container.
>> Dump of the process-tree for container_1426992254950_0002_01_000005 :
>> |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS)
>> SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
>> |- 7960 7954 7954 7954 (java) 5 0 4576591872 3199
>> /usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/jre/bin/java
>> -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN 1638
>> -Djava.io.tmpdir=/tmp/hadoop-root/nm-local-dir/usercache/root/appcache/application_1426992254950_0002/container_1426992254950_0002_01_000005/tmp
>> -Dlog4j.configuration=container-log4j.properties
>> -Dyarn.app.container.log.dir=/home/hadoop-lzl/hadoop-2.6.0/logs/userlogs/application_1426992254950_0002/container_1426992254950_0002_01_000005
>> -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA
>> org.apache.hadoop.mapred.YarnChild 199.26.254.140 36542
>> attempt_1426992254950_0002_m_000003_0 5
>> |- 7954 7949 7954 7954 (bash) 0 0 65421312 275 /bin/bash -c
>> /usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/jre/bin/java
>> -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN 1638
>> -Djava.io.tmpdir=/tmp/hadoop-root/nm-local-dir/usercache/root/appcache/application_1426992254950_0002/container_1426992254950_0002_01_000005/tmp
>> -Dlog4j.configuration=container-log4j.properties
>> -Dyarn.app.container.log.dir=/home/hadoop-lzl/hadoop-2.6.0/logs/userlogs/application_1426992254950_0002/container_1426992254950_0002_01_000005
>> -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA
>> org.apache.hadoop.mapred.YarnChild 199.26.254.140 36542
>> attempt_1426992254950_0002_m_000003_0 5
>> 1>/home/hadoop-lzl/hadoop-2.6.0/logs/userlogs/application_1426992254950_0002/container_1426992254950_0002_01_000005/stdout
>> 2>/home/hadoop-lzl/hadoop-2.6.0/logs/userlogs/application_1426992254950_0002/container_1426992254950_0002_01_000005/stderr
>>
>>
>> Exception from container-launch.
>> Container id: container_1426992254950_0002_01_000005
>> Exit code: 1
>> Stack trace: ExitCodeException exitCode=1:
>> at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
>> at org.apache.hadoop.util.Shell.run(Shell.java:455)
>> at
>> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:715)
>> at
>> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:211)
>> at
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
>> at
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
>> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>> at java.lang.Thread.run(Thread.java:679)
>>
>> Thanks,
>> Fei
>>
>
>
Re: Container beyond virtual memory limits
Posted by Fei Hu <hu...@gmail.com>.
Thank you for your help. It is useful.
Best,
Fei
> On Mar 23, 2015, at 1:09 AM, Drake민영근 <dr...@nexr.com> wrote:
>
> Hi,
>
> See "6. Killing of Tasks Due to Virtual Memory Usage" in http://blog.cloudera.com/blog/2014/04/apache-hadoop-yarn-avoiding-6-time-consuming-gotchas/ <http://blog.cloudera.com/blog/2014/04/apache-hadoop-yarn-avoiding-6-time-consuming-gotchas/>
>
> Drake 민영근 Ph.D
> kt NexR
>
> On Sun, Mar 22, 2015 at 12:43 PM, Fei Hu <hufei68@gmail.com <ma...@gmail.com>> wrote:
> Hi,
>
> I just test my yarn installation, and run a Wordcount program. But it always report the following error, who knows how to solve it? Thank you in advance.
>
> Container [pid=7954,containerID=container_1426992254950_0002_01_000005] is running beyond virtual memory limits. Current usage: 13.6 MB of 1 GB physical memory used; 4.3 GB of 2.1 GB virtual memory used. Killing container.
> Dump of the process-tree for container_1426992254950_0002_01_000005 :
> |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
> |- 7960 7954 7954 7954 (java) 5 0 4576591872 3199 /usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/jre/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN 1638 -Djava.io.tmpdir=/tmp/hadoop-root/nm-local-dir/usercache/root/appcache/application_1426992254950_0002/container_1426992254950_0002_01_000005/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/home/hadoop-lzl/hadoop-2.6.0/logs/userlogs/application_1426992254950_0002/container_1426992254950_0002_01_000005 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA org.apache.hadoop.mapred.YarnChild 199.26.254.140 36542 attempt_1426992254950_0002_m_000003_0 5
> |- 7954 7949 7954 7954 (bash) 0 0 65421312 275 /bin/bash -c /usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/jre/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN 1638 -Djava.io.tmpdir=/tmp/hadoop-root/nm-local-dir/usercache/root/appcache/application_1426992254950_0002/container_1426992254950_0002_01_000005/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/home/hadoop-lzl/hadoop-2.6.0/logs/userlogs/application_1426992254950_0002/container_1426992254950_0002_01_000005 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA org.apache.hadoop.mapred.YarnChild 199.26.254.140 36542 attempt_1426992254950_0002_m_000003_0 5 1>/home/hadoop-lzl/hadoop-2.6.0/logs/userlogs/application_1426992254950_0002/container_1426992254950_0002_01_000005/stdout 2>/home/hadoop-lzl/hadoop-2.6.0/logs/userlogs/application_1426992254950_0002/container_1426992254950_0002_01_000005/stderr
>
> Exception from container-launch.
> Container id: container_1426992254950_0002_01_000005
> Exit code: 1
> Stack trace: ExitCodeException exitCode=1:
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
> at org.apache.hadoop.util.Shell.run(Shell.java:455)
> at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:715)
> at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:211)
> at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
> at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
>
> Thanks,
> Fei
>
Re: Container beyond virtual memory limits
Posted by Fei Hu <hu...@gmail.com>.
Thank you for your help. It is useful.
Best,
Fei
> On Mar 23, 2015, at 1:09 AM, Drake민영근 <dr...@nexr.com> wrote:
>
> Hi,
>
> See "6. Killing of Tasks Due to Virtual Memory Usage" in http://blog.cloudera.com/blog/2014/04/apache-hadoop-yarn-avoiding-6-time-consuming-gotchas/ <http://blog.cloudera.com/blog/2014/04/apache-hadoop-yarn-avoiding-6-time-consuming-gotchas/>
>
> Drake 민영근 Ph.D
> kt NexR
>
> On Sun, Mar 22, 2015 at 12:43 PM, Fei Hu <hufei68@gmail.com <ma...@gmail.com>> wrote:
> Hi,
>
> I just test my yarn installation, and run a Wordcount program. But it always report the following error, who knows how to solve it? Thank you in advance.
>
> Container [pid=7954,containerID=container_1426992254950_0002_01_000005] is running beyond virtual memory limits. Current usage: 13.6 MB of 1 GB physical memory used; 4.3 GB of 2.1 GB virtual memory used. Killing container.
> Dump of the process-tree for container_1426992254950_0002_01_000005 :
> |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
> |- 7960 7954 7954 7954 (java) 5 0 4576591872 3199 /usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/jre/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN 1638 -Djava.io.tmpdir=/tmp/hadoop-root/nm-local-dir/usercache/root/appcache/application_1426992254950_0002/container_1426992254950_0002_01_000005/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/home/hadoop-lzl/hadoop-2.6.0/logs/userlogs/application_1426992254950_0002/container_1426992254950_0002_01_000005 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA org.apache.hadoop.mapred.YarnChild 199.26.254.140 36542 attempt_1426992254950_0002_m_000003_0 5
> |- 7954 7949 7954 7954 (bash) 0 0 65421312 275 /bin/bash -c /usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/jre/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN 1638 -Djava.io.tmpdir=/tmp/hadoop-root/nm-local-dir/usercache/root/appcache/application_1426992254950_0002/container_1426992254950_0002_01_000005/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/home/hadoop-lzl/hadoop-2.6.0/logs/userlogs/application_1426992254950_0002/container_1426992254950_0002_01_000005 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA org.apache.hadoop.mapred.YarnChild 199.26.254.140 36542 attempt_1426992254950_0002_m_000003_0 5 1>/home/hadoop-lzl/hadoop-2.6.0/logs/userlogs/application_1426992254950_0002/container_1426992254950_0002_01_000005/stdout 2>/home/hadoop-lzl/hadoop-2.6.0/logs/userlogs/application_1426992254950_0002/container_1426992254950_0002_01_000005/stderr
>
> Exception from container-launch.
> Container id: container_1426992254950_0002_01_000005
> Exit code: 1
> Stack trace: ExitCodeException exitCode=1:
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
> at org.apache.hadoop.util.Shell.run(Shell.java:455)
> at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:715)
> at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:211)
> at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
> at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
>
> Thanks,
> Fei
>
Re: Container beyond virtual memory limits
Posted by Gaurav Gupta <ga...@gmail.com>.
Increasing vmem:pmem ratio should help you out. Default value is 2:1,
change it to 5:1
On Sun, Mar 22, 2015 at 10:09 PM, Drake민영근 <dr...@nexr.com> wrote:
> Hi,
>
> See "6. Killing of Tasks Due to Virtual Memory Usage" in
> http://blog.cloudera.com/blog/2014/04/apache-hadoop-yarn-avoiding-6-time-consuming-gotchas/
>
>
> Drake 민영근 Ph.D
> kt NexR
>
> On Sun, Mar 22, 2015 at 12:43 PM, Fei Hu <hu...@gmail.com> wrote:
>
>> Hi,
>>
>> I just test my yarn installation, and run a Wordcount program. But it
>> always report the following error, who knows how to solve it? Thank you in
>> advance.
>>
>> Container [pid=7954,containerID=container_1426992254950_0002_01_000005]
>> is running beyond virtual memory limits. Current usage: 13.6 MB of 1 GB
>> physical memory used; 4.3 GB of 2.1 GB virtual memory used. Killing
>> container.
>> Dump of the process-tree for container_1426992254950_0002_01_000005 :
>> |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS)
>> SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
>> |- 7960 7954 7954 7954 (java) 5 0 4576591872 3199
>> /usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/jre/bin/java
>> -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN 1638
>> -Djava.io.tmpdir=/tmp/hadoop-root/nm-local-dir/usercache/root/appcache/application_1426992254950_0002/container_1426992254950_0002_01_000005/tmp
>> -Dlog4j.configuration=container-log4j.properties
>> -Dyarn.app.container.log.dir=/home/hadoop-lzl/hadoop-2.6.0/logs/userlogs/application_1426992254950_0002/container_1426992254950_0002_01_000005
>> -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA
>> org.apache.hadoop.mapred.YarnChild 199.26.254.140 36542
>> attempt_1426992254950_0002_m_000003_0 5
>> |- 7954 7949 7954 7954 (bash) 0 0 65421312 275 /bin/bash -c
>> /usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/jre/bin/java
>> -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN 1638
>> -Djava.io.tmpdir=/tmp/hadoop-root/nm-local-dir/usercache/root/appcache/application_1426992254950_0002/container_1426992254950_0002_01_000005/tmp
>> -Dlog4j.configuration=container-log4j.properties
>> -Dyarn.app.container.log.dir=/home/hadoop-lzl/hadoop-2.6.0/logs/userlogs/application_1426992254950_0002/container_1426992254950_0002_01_000005
>> -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA
>> org.apache.hadoop.mapred.YarnChild 199.26.254.140 36542
>> attempt_1426992254950_0002_m_000003_0 5
>> 1>/home/hadoop-lzl/hadoop-2.6.0/logs/userlogs/application_1426992254950_0002/container_1426992254950_0002_01_000005/stdout
>> 2>/home/hadoop-lzl/hadoop-2.6.0/logs/userlogs/application_1426992254950_0002/container_1426992254950_0002_01_000005/stderr
>>
>>
>> Exception from container-launch.
>> Container id: container_1426992254950_0002_01_000005
>> Exit code: 1
>> Stack trace: ExitCodeException exitCode=1:
>> at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
>> at org.apache.hadoop.util.Shell.run(Shell.java:455)
>> at
>> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:715)
>> at
>> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:211)
>> at
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
>> at
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
>> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>> at java.lang.Thread.run(Thread.java:679)
>>
>> Thanks,
>> Fei
>>
>
>
Re: Container beyond virtual memory limits
Posted by Gaurav Gupta <ga...@gmail.com>.
Increasing vmem:pmem ratio should help you out. Default value is 2:1,
change it to 5:1
On Sun, Mar 22, 2015 at 10:09 PM, Drake민영근 <dr...@nexr.com> wrote:
> Hi,
>
> See "6. Killing of Tasks Due to Virtual Memory Usage" in
> http://blog.cloudera.com/blog/2014/04/apache-hadoop-yarn-avoiding-6-time-consuming-gotchas/
>
>
> Drake 민영근 Ph.D
> kt NexR
>
> On Sun, Mar 22, 2015 at 12:43 PM, Fei Hu <hu...@gmail.com> wrote:
>
>> Hi,
>>
>> I just test my yarn installation, and run a Wordcount program. But it
>> always report the following error, who knows how to solve it? Thank you in
>> advance.
>>
>> Container [pid=7954,containerID=container_1426992254950_0002_01_000005]
>> is running beyond virtual memory limits. Current usage: 13.6 MB of 1 GB
>> physical memory used; 4.3 GB of 2.1 GB virtual memory used. Killing
>> container.
>> Dump of the process-tree for container_1426992254950_0002_01_000005 :
>> |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS)
>> SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
>> |- 7960 7954 7954 7954 (java) 5 0 4576591872 3199
>> /usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/jre/bin/java
>> -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN 1638
>> -Djava.io.tmpdir=/tmp/hadoop-root/nm-local-dir/usercache/root/appcache/application_1426992254950_0002/container_1426992254950_0002_01_000005/tmp
>> -Dlog4j.configuration=container-log4j.properties
>> -Dyarn.app.container.log.dir=/home/hadoop-lzl/hadoop-2.6.0/logs/userlogs/application_1426992254950_0002/container_1426992254950_0002_01_000005
>> -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA
>> org.apache.hadoop.mapred.YarnChild 199.26.254.140 36542
>> attempt_1426992254950_0002_m_000003_0 5
>> |- 7954 7949 7954 7954 (bash) 0 0 65421312 275 /bin/bash -c
>> /usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/jre/bin/java
>> -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN 1638
>> -Djava.io.tmpdir=/tmp/hadoop-root/nm-local-dir/usercache/root/appcache/application_1426992254950_0002/container_1426992254950_0002_01_000005/tmp
>> -Dlog4j.configuration=container-log4j.properties
>> -Dyarn.app.container.log.dir=/home/hadoop-lzl/hadoop-2.6.0/logs/userlogs/application_1426992254950_0002/container_1426992254950_0002_01_000005
>> -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA
>> org.apache.hadoop.mapred.YarnChild 199.26.254.140 36542
>> attempt_1426992254950_0002_m_000003_0 5
>> 1>/home/hadoop-lzl/hadoop-2.6.0/logs/userlogs/application_1426992254950_0002/container_1426992254950_0002_01_000005/stdout
>> 2>/home/hadoop-lzl/hadoop-2.6.0/logs/userlogs/application_1426992254950_0002/container_1426992254950_0002_01_000005/stderr
>>
>>
>> Exception from container-launch.
>> Container id: container_1426992254950_0002_01_000005
>> Exit code: 1
>> Stack trace: ExitCodeException exitCode=1:
>> at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
>> at org.apache.hadoop.util.Shell.run(Shell.java:455)
>> at
>> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:715)
>> at
>> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:211)
>> at
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
>> at
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
>> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>> at java.lang.Thread.run(Thread.java:679)
>>
>> Thanks,
>> Fei
>>
>
>
Re: Container beyond virtual memory limits
Posted by Gaurav Gupta <ga...@gmail.com>.
Increasing vmem:pmem ratio should help you out. Default value is 2:1,
change it to 5:1
On Sun, Mar 22, 2015 at 10:09 PM, Drake민영근 <dr...@nexr.com> wrote:
> Hi,
>
> See "6. Killing of Tasks Due to Virtual Memory Usage" in
> http://blog.cloudera.com/blog/2014/04/apache-hadoop-yarn-avoiding-6-time-consuming-gotchas/
>
>
> Drake 민영근 Ph.D
> kt NexR
>
> On Sun, Mar 22, 2015 at 12:43 PM, Fei Hu <hu...@gmail.com> wrote:
>
>> Hi,
>>
>> I just test my yarn installation, and run a Wordcount program. But it
>> always report the following error, who knows how to solve it? Thank you in
>> advance.
>>
>> Container [pid=7954,containerID=container_1426992254950_0002_01_000005]
>> is running beyond virtual memory limits. Current usage: 13.6 MB of 1 GB
>> physical memory used; 4.3 GB of 2.1 GB virtual memory used. Killing
>> container.
>> Dump of the process-tree for container_1426992254950_0002_01_000005 :
>> |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS)
>> SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
>> |- 7960 7954 7954 7954 (java) 5 0 4576591872 3199
>> /usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/jre/bin/java
>> -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN 1638
>> -Djava.io.tmpdir=/tmp/hadoop-root/nm-local-dir/usercache/root/appcache/application_1426992254950_0002/container_1426992254950_0002_01_000005/tmp
>> -Dlog4j.configuration=container-log4j.properties
>> -Dyarn.app.container.log.dir=/home/hadoop-lzl/hadoop-2.6.0/logs/userlogs/application_1426992254950_0002/container_1426992254950_0002_01_000005
>> -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA
>> org.apache.hadoop.mapred.YarnChild 199.26.254.140 36542
>> attempt_1426992254950_0002_m_000003_0 5
>> |- 7954 7949 7954 7954 (bash) 0 0 65421312 275 /bin/bash -c
>> /usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/jre/bin/java
>> -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN 1638
>> -Djava.io.tmpdir=/tmp/hadoop-root/nm-local-dir/usercache/root/appcache/application_1426992254950_0002/container_1426992254950_0002_01_000005/tmp
>> -Dlog4j.configuration=container-log4j.properties
>> -Dyarn.app.container.log.dir=/home/hadoop-lzl/hadoop-2.6.0/logs/userlogs/application_1426992254950_0002/container_1426992254950_0002_01_000005
>> -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA
>> org.apache.hadoop.mapred.YarnChild 199.26.254.140 36542
>> attempt_1426992254950_0002_m_000003_0 5
>> 1>/home/hadoop-lzl/hadoop-2.6.0/logs/userlogs/application_1426992254950_0002/container_1426992254950_0002_01_000005/stdout
>> 2>/home/hadoop-lzl/hadoop-2.6.0/logs/userlogs/application_1426992254950_0002/container_1426992254950_0002_01_000005/stderr
>>
>>
>> Exception from container-launch.
>> Container id: container_1426992254950_0002_01_000005
>> Exit code: 1
>> Stack trace: ExitCodeException exitCode=1:
>> at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
>> at org.apache.hadoop.util.Shell.run(Shell.java:455)
>> at
>> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:715)
>> at
>> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:211)
>> at
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
>> at
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
>> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>> at java.lang.Thread.run(Thread.java:679)
>>
>> Thanks,
>> Fei
>>
>
>
Re: Container beyond virtual memory limits
Posted by Fei Hu <hu...@gmail.com>.
Thank you for your help. It is useful.
Best,
Fei
> On Mar 23, 2015, at 1:09 AM, Drake민영근 <dr...@nexr.com> wrote:
>
> Hi,
>
> See "6. Killing of Tasks Due to Virtual Memory Usage" in http://blog.cloudera.com/blog/2014/04/apache-hadoop-yarn-avoiding-6-time-consuming-gotchas/ <http://blog.cloudera.com/blog/2014/04/apache-hadoop-yarn-avoiding-6-time-consuming-gotchas/>
>
> Drake 민영근 Ph.D
> kt NexR
>
> On Sun, Mar 22, 2015 at 12:43 PM, Fei Hu <hufei68@gmail.com <ma...@gmail.com>> wrote:
> Hi,
>
> I just test my yarn installation, and run a Wordcount program. But it always report the following error, who knows how to solve it? Thank you in advance.
>
> Container [pid=7954,containerID=container_1426992254950_0002_01_000005] is running beyond virtual memory limits. Current usage: 13.6 MB of 1 GB physical memory used; 4.3 GB of 2.1 GB virtual memory used. Killing container.
> Dump of the process-tree for container_1426992254950_0002_01_000005 :
> |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
> |- 7960 7954 7954 7954 (java) 5 0 4576591872 3199 /usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/jre/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN 1638 -Djava.io.tmpdir=/tmp/hadoop-root/nm-local-dir/usercache/root/appcache/application_1426992254950_0002/container_1426992254950_0002_01_000005/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/home/hadoop-lzl/hadoop-2.6.0/logs/userlogs/application_1426992254950_0002/container_1426992254950_0002_01_000005 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA org.apache.hadoop.mapred.YarnChild 199.26.254.140 36542 attempt_1426992254950_0002_m_000003_0 5
> |- 7954 7949 7954 7954 (bash) 0 0 65421312 275 /bin/bash -c /usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/jre/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN 1638 -Djava.io.tmpdir=/tmp/hadoop-root/nm-local-dir/usercache/root/appcache/application_1426992254950_0002/container_1426992254950_0002_01_000005/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/home/hadoop-lzl/hadoop-2.6.0/logs/userlogs/application_1426992254950_0002/container_1426992254950_0002_01_000005 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA org.apache.hadoop.mapred.YarnChild 199.26.254.140 36542 attempt_1426992254950_0002_m_000003_0 5 1>/home/hadoop-lzl/hadoop-2.6.0/logs/userlogs/application_1426992254950_0002/container_1426992254950_0002_01_000005/stdout 2>/home/hadoop-lzl/hadoop-2.6.0/logs/userlogs/application_1426992254950_0002/container_1426992254950_0002_01_000005/stderr
>
> Exception from container-launch.
> Container id: container_1426992254950_0002_01_000005
> Exit code: 1
> Stack trace: ExitCodeException exitCode=1:
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
> at org.apache.hadoop.util.Shell.run(Shell.java:455)
> at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:715)
> at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:211)
> at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
> at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
>
> Thanks,
> Fei
>
Re: Container beyond virtual memory limits
Posted by Fei Hu <hu...@gmail.com>.
Thank you for your help. It is useful.
Best,
Fei
> On Mar 23, 2015, at 1:09 AM, Drake민영근 <dr...@nexr.com> wrote:
>
> Hi,
>
> See "6. Killing of Tasks Due to Virtual Memory Usage" in http://blog.cloudera.com/blog/2014/04/apache-hadoop-yarn-avoiding-6-time-consuming-gotchas/ <http://blog.cloudera.com/blog/2014/04/apache-hadoop-yarn-avoiding-6-time-consuming-gotchas/>
>
> Drake 민영근 Ph.D
> kt NexR
>
> On Sun, Mar 22, 2015 at 12:43 PM, Fei Hu <hufei68@gmail.com <ma...@gmail.com>> wrote:
> Hi,
>
> I just test my yarn installation, and run a Wordcount program. But it always report the following error, who knows how to solve it? Thank you in advance.
>
> Container [pid=7954,containerID=container_1426992254950_0002_01_000005] is running beyond virtual memory limits. Current usage: 13.6 MB of 1 GB physical memory used; 4.3 GB of 2.1 GB virtual memory used. Killing container.
> Dump of the process-tree for container_1426992254950_0002_01_000005 :
> |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
> |- 7960 7954 7954 7954 (java) 5 0 4576591872 3199 /usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/jre/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN 1638 -Djava.io.tmpdir=/tmp/hadoop-root/nm-local-dir/usercache/root/appcache/application_1426992254950_0002/container_1426992254950_0002_01_000005/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/home/hadoop-lzl/hadoop-2.6.0/logs/userlogs/application_1426992254950_0002/container_1426992254950_0002_01_000005 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA org.apache.hadoop.mapred.YarnChild 199.26.254.140 36542 attempt_1426992254950_0002_m_000003_0 5
> |- 7954 7949 7954 7954 (bash) 0 0 65421312 275 /bin/bash -c /usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/jre/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN 1638 -Djava.io.tmpdir=/tmp/hadoop-root/nm-local-dir/usercache/root/appcache/application_1426992254950_0002/container_1426992254950_0002_01_000005/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/home/hadoop-lzl/hadoop-2.6.0/logs/userlogs/application_1426992254950_0002/container_1426992254950_0002_01_000005 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA org.apache.hadoop.mapred.YarnChild 199.26.254.140 36542 attempt_1426992254950_0002_m_000003_0 5 1>/home/hadoop-lzl/hadoop-2.6.0/logs/userlogs/application_1426992254950_0002/container_1426992254950_0002_01_000005/stdout 2>/home/hadoop-lzl/hadoop-2.6.0/logs/userlogs/application_1426992254950_0002/container_1426992254950_0002_01_000005/stderr
>
> Exception from container-launch.
> Container id: container_1426992254950_0002_01_000005
> Exit code: 1
> Stack trace: ExitCodeException exitCode=1:
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
> at org.apache.hadoop.util.Shell.run(Shell.java:455)
> at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:715)
> at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:211)
> at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
> at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
>
> Thanks,
> Fei
>
Re: Container beyond virtual memory limits
Posted by Drake민영근 <dr...@nexr.com>.
Hi,
See "6. Killing of Tasks Due to Virtual Memory Usage" in
http://blog.cloudera.com/blog/2014/04/apache-hadoop-yarn-avoiding-6-time-consuming-gotchas/
Drake 민영근 Ph.D
kt NexR
On Sun, Mar 22, 2015 at 12:43 PM, Fei Hu <hu...@gmail.com> wrote:
> Hi,
>
> I just test my yarn installation, and run a Wordcount program. But it
> always report the following error, who knows how to solve it? Thank you in
> advance.
>
> Container [pid=7954,containerID=container_1426992254950_0002_01_000005] is
> running beyond virtual memory limits. Current usage: 13.6 MB of 1 GB
> physical memory used; 4.3 GB of 2.1 GB virtual memory used. Killing
> container.
> Dump of the process-tree for container_1426992254950_0002_01_000005 :
> |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS)
> SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
> |- 7960 7954 7954 7954 (java) 5 0 4576591872 3199
> /usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/jre/bin/java
> -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN 1638
> -Djava.io.tmpdir=/tmp/hadoop-root/nm-local-dir/usercache/root/appcache/application_1426992254950_0002/container_1426992254950_0002_01_000005/tmp
> -Dlog4j.configuration=container-log4j.properties
> -Dyarn.app.container.log.dir=/home/hadoop-lzl/hadoop-2.6.0/logs/userlogs/application_1426992254950_0002/container_1426992254950_0002_01_000005
> -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA
> org.apache.hadoop.mapred.YarnChild 199.26.254.140 36542
> attempt_1426992254950_0002_m_000003_0 5
> |- 7954 7949 7954 7954 (bash) 0 0 65421312 275 /bin/bash -c
> /usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/jre/bin/java
> -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN 1638
> -Djava.io.tmpdir=/tmp/hadoop-root/nm-local-dir/usercache/root/appcache/application_1426992254950_0002/container_1426992254950_0002_01_000005/tmp
> -Dlog4j.configuration=container-log4j.properties
> -Dyarn.app.container.log.dir=/home/hadoop-lzl/hadoop-2.6.0/logs/userlogs/application_1426992254950_0002/container_1426992254950_0002_01_000005
> -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA
> org.apache.hadoop.mapred.YarnChild 199.26.254.140 36542
> attempt_1426992254950_0002_m_000003_0 5
> 1>/home/hadoop-lzl/hadoop-2.6.0/logs/userlogs/application_1426992254950_0002/container_1426992254950_0002_01_000005/stdout
> 2>/home/hadoop-lzl/hadoop-2.6.0/logs/userlogs/application_1426992254950_0002/container_1426992254950_0002_01_000005/stderr
>
>
> Exception from container-launch.
> Container id: container_1426992254950_0002_01_000005
> Exit code: 1
> Stack trace: ExitCodeException exitCode=1:
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
> at org.apache.hadoop.util.Shell.run(Shell.java:455)
> at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:715)
> at
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:211)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
>
> Thanks,
> Fei
>
Re: Container beyond virtual memory limits
Posted by Drake민영근 <dr...@nexr.com>.
Hi,
See "6. Killing of Tasks Due to Virtual Memory Usage" in
http://blog.cloudera.com/blog/2014/04/apache-hadoop-yarn-avoiding-6-time-consuming-gotchas/
Drake 민영근 Ph.D
kt NexR
On Sun, Mar 22, 2015 at 12:43 PM, Fei Hu <hu...@gmail.com> wrote:
> Hi,
>
> I just test my yarn installation, and run a Wordcount program. But it
> always report the following error, who knows how to solve it? Thank you in
> advance.
>
> Container [pid=7954,containerID=container_1426992254950_0002_01_000005] is
> running beyond virtual memory limits. Current usage: 13.6 MB of 1 GB
> physical memory used; 4.3 GB of 2.1 GB virtual memory used. Killing
> container.
> Dump of the process-tree for container_1426992254950_0002_01_000005 :
> |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS)
> SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
> |- 7960 7954 7954 7954 (java) 5 0 4576591872 3199
> /usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/jre/bin/java
> -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN 1638
> -Djava.io.tmpdir=/tmp/hadoop-root/nm-local-dir/usercache/root/appcache/application_1426992254950_0002/container_1426992254950_0002_01_000005/tmp
> -Dlog4j.configuration=container-log4j.properties
> -Dyarn.app.container.log.dir=/home/hadoop-lzl/hadoop-2.6.0/logs/userlogs/application_1426992254950_0002/container_1426992254950_0002_01_000005
> -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA
> org.apache.hadoop.mapred.YarnChild 199.26.254.140 36542
> attempt_1426992254950_0002_m_000003_0 5
> |- 7954 7949 7954 7954 (bash) 0 0 65421312 275 /bin/bash -c
> /usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/jre/bin/java
> -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN 1638
> -Djava.io.tmpdir=/tmp/hadoop-root/nm-local-dir/usercache/root/appcache/application_1426992254950_0002/container_1426992254950_0002_01_000005/tmp
> -Dlog4j.configuration=container-log4j.properties
> -Dyarn.app.container.log.dir=/home/hadoop-lzl/hadoop-2.6.0/logs/userlogs/application_1426992254950_0002/container_1426992254950_0002_01_000005
> -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA
> org.apache.hadoop.mapred.YarnChild 199.26.254.140 36542
> attempt_1426992254950_0002_m_000003_0 5
> 1>/home/hadoop-lzl/hadoop-2.6.0/logs/userlogs/application_1426992254950_0002/container_1426992254950_0002_01_000005/stdout
> 2>/home/hadoop-lzl/hadoop-2.6.0/logs/userlogs/application_1426992254950_0002/container_1426992254950_0002_01_000005/stderr
>
>
> Exception from container-launch.
> Container id: container_1426992254950_0002_01_000005
> Exit code: 1
> Stack trace: ExitCodeException exitCode=1:
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
> at org.apache.hadoop.util.Shell.run(Shell.java:455)
> at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:715)
> at
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:211)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
>
> Thanks,
> Fei
>
Re: Container beyond virtual memory limits
Posted by Drake민영근 <dr...@nexr.com>.
Hi,
See "6. Killing of Tasks Due to Virtual Memory Usage" in
http://blog.cloudera.com/blog/2014/04/apache-hadoop-yarn-avoiding-6-time-consuming-gotchas/
Drake 민영근 Ph.D
kt NexR
On Sun, Mar 22, 2015 at 12:43 PM, Fei Hu <hu...@gmail.com> wrote:
> Hi,
>
> I just test my yarn installation, and run a Wordcount program. But it
> always report the following error, who knows how to solve it? Thank you in
> advance.
>
> Container [pid=7954,containerID=container_1426992254950_0002_01_000005] is
> running beyond virtual memory limits. Current usage: 13.6 MB of 1 GB
> physical memory used; 4.3 GB of 2.1 GB virtual memory used. Killing
> container.
> Dump of the process-tree for container_1426992254950_0002_01_000005 :
> |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS)
> SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
> |- 7960 7954 7954 7954 (java) 5 0 4576591872 3199
> /usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/jre/bin/java
> -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN 1638
> -Djava.io.tmpdir=/tmp/hadoop-root/nm-local-dir/usercache/root/appcache/application_1426992254950_0002/container_1426992254950_0002_01_000005/tmp
> -Dlog4j.configuration=container-log4j.properties
> -Dyarn.app.container.log.dir=/home/hadoop-lzl/hadoop-2.6.0/logs/userlogs/application_1426992254950_0002/container_1426992254950_0002_01_000005
> -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA
> org.apache.hadoop.mapred.YarnChild 199.26.254.140 36542
> attempt_1426992254950_0002_m_000003_0 5
> |- 7954 7949 7954 7954 (bash) 0 0 65421312 275 /bin/bash -c
> /usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/jre/bin/java
> -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN 1638
> -Djava.io.tmpdir=/tmp/hadoop-root/nm-local-dir/usercache/root/appcache/application_1426992254950_0002/container_1426992254950_0002_01_000005/tmp
> -Dlog4j.configuration=container-log4j.properties
> -Dyarn.app.container.log.dir=/home/hadoop-lzl/hadoop-2.6.0/logs/userlogs/application_1426992254950_0002/container_1426992254950_0002_01_000005
> -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA
> org.apache.hadoop.mapred.YarnChild 199.26.254.140 36542
> attempt_1426992254950_0002_m_000003_0 5
> 1>/home/hadoop-lzl/hadoop-2.6.0/logs/userlogs/application_1426992254950_0002/container_1426992254950_0002_01_000005/stdout
> 2>/home/hadoop-lzl/hadoop-2.6.0/logs/userlogs/application_1426992254950_0002/container_1426992254950_0002_01_000005/stderr
>
>
> Exception from container-launch.
> Container id: container_1426992254950_0002_01_000005
> Exit code: 1
> Stack trace: ExitCodeException exitCode=1:
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
> at org.apache.hadoop.util.Shell.run(Shell.java:455)
> at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:715)
> at
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:211)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
>
> Thanks,
> Fei
>
Re: Container beyond virtual memory limits
Posted by Drake민영근 <dr...@nexr.com>.
Hi,
See "6. Killing of Tasks Due to Virtual Memory Usage" in
http://blog.cloudera.com/blog/2014/04/apache-hadoop-yarn-avoiding-6-time-consuming-gotchas/
Drake 민영근 Ph.D
kt NexR
On Sun, Mar 22, 2015 at 12:43 PM, Fei Hu <hu...@gmail.com> wrote:
> Hi,
>
> I just test my yarn installation, and run a Wordcount program. But it
> always report the following error, who knows how to solve it? Thank you in
> advance.
>
> Container [pid=7954,containerID=container_1426992254950_0002_01_000005] is
> running beyond virtual memory limits. Current usage: 13.6 MB of 1 GB
> physical memory used; 4.3 GB of 2.1 GB virtual memory used. Killing
> container.
> Dump of the process-tree for container_1426992254950_0002_01_000005 :
> |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS)
> SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
> |- 7960 7954 7954 7954 (java) 5 0 4576591872 3199
> /usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/jre/bin/java
> -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN 1638
> -Djava.io.tmpdir=/tmp/hadoop-root/nm-local-dir/usercache/root/appcache/application_1426992254950_0002/container_1426992254950_0002_01_000005/tmp
> -Dlog4j.configuration=container-log4j.properties
> -Dyarn.app.container.log.dir=/home/hadoop-lzl/hadoop-2.6.0/logs/userlogs/application_1426992254950_0002/container_1426992254950_0002_01_000005
> -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA
> org.apache.hadoop.mapred.YarnChild 199.26.254.140 36542
> attempt_1426992254950_0002_m_000003_0 5
> |- 7954 7949 7954 7954 (bash) 0 0 65421312 275 /bin/bash -c
> /usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/jre/bin/java
> -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN 1638
> -Djava.io.tmpdir=/tmp/hadoop-root/nm-local-dir/usercache/root/appcache/application_1426992254950_0002/container_1426992254950_0002_01_000005/tmp
> -Dlog4j.configuration=container-log4j.properties
> -Dyarn.app.container.log.dir=/home/hadoop-lzl/hadoop-2.6.0/logs/userlogs/application_1426992254950_0002/container_1426992254950_0002_01_000005
> -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA
> org.apache.hadoop.mapred.YarnChild 199.26.254.140 36542
> attempt_1426992254950_0002_m_000003_0 5
> 1>/home/hadoop-lzl/hadoop-2.6.0/logs/userlogs/application_1426992254950_0002/container_1426992254950_0002_01_000005/stdout
> 2>/home/hadoop-lzl/hadoop-2.6.0/logs/userlogs/application_1426992254950_0002/container_1426992254950_0002_01_000005/stderr
>
>
> Exception from container-launch.
> Container id: container_1426992254950_0002_01_000005
> Exit code: 1
> Stack trace: ExitCodeException exitCode=1:
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
> at org.apache.hadoop.util.Shell.run(Shell.java:455)
> at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:715)
> at
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:211)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
>
> Thanks,
> Fei
>