You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Arbi Akhina <ar...@gmail.com> on 2015/06/17 13:49:49 UTC
YARN container killed as running beyond memory limits
Hi, I've a YARN application that submits containers. In the
AplicationMaster logs I see that the container is killed. Here is the logs:
Jun 17, 2015 1:31:27 PM com.heavenize.modules.RMCallbackHandler
onContainersCompleted
INFO: container 'container_1434471275225_0007_01_000002' status is
ContainerStatus: [ContainerId: container_1434471275225_0007_01_000002,
State: COMPLETE, Diagnostics: Container
[pid=4069,containerID=container_1434471275225_0007_01_000002] is
running beyond virtual memory limits. Current usage: 576.2 MB of 2 GB
physical memory used; 4.2 GB of 4.2 GB virtual memory used. Killing
container.
Dump of the process-tree for container_1434471275225_0007_01_000002 :
|- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS)
SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES)
FULL_CMD_LINE
|- 4094 4093 4069 4069 (java) 2932 94 2916065280 122804
/usr/lib/jvm/java-7-openjdk-amd64/bin/java -Xms512m -Xmx2048m
-XX:MaxPermSize=250m -XX:+UseConcMarkSweepGC
-Dosmoze.path=/tmp/hadoop-hadoop/nm-local-dir/usercache/hadoop/appcache/container_1434471275225_0007_01_000002/Osmoze
-Dspring.profiles.active=webServer -jar
/tmp/hadoop-hadoop/nm-local-dir/usercache/hadoop/appcache/container_1434471275225_0007_01_000002/heavenize-modules.jar
|- 4093 4073 4069 4069 (sh) 0 0 4550656 164 /bin/sh
/tmp/hadoop-hadoop/nm-local-dir/usercache/hadoop/appcache/container_1434471275225_0007_01_000002/startup.sh
|- 4073 4069 4069 4069 (java) 249 34 1577267200 24239
/usr/lib/jvm/java-7-openjdk-amd64/bin/java
com.heavenize.yarn.task.ModulesManager -containerId
container_1434471275225_0007_01_000002 -port 5369 -exe
hdfs://hadoop-server/user/hadoop/heavenize/heavenize-modules.jar -conf
hdfs://hadoop-server/user/hadoop/heavenize/config.zip
|- 4069 1884 4069 4069 (bash) 0 0 12730368 304 /bin/bash -c
/usr/lib/jvm/java-7-openjdk-amd64/bin/java
com.heavenize.yarn.task.ModulesManager -containerId
container_1434471275225_0007_01_000002 -port 5369 -exe
hdfs://hadoop-server/user/hadoop/heavenize/heavenize-modules.jar -conf
hdfs://hadoop-server/user/hadoop/heavenize/config.zip 1>
/usr/local/hadoop/logs/userlogs/application_1434471275225_0007/container_1434471275225_0007_01_000002/stdout
2> /usr/local/hadoop/logs/userlogs/application_1434471275225_0007/container_1434471275225_0007_01_000002/stderr
I don't see any memory excess, any idea where this error comes from?
There is no errors in the container, it just stop logging as a result of
being killed.
Re: YARN container killed as running beyond memory limits
Posted by Gaurav Gupta <ga...@gmail.com>.
You can also change the default value of yarn.nodemanager.vmem-pmem-ratio
On Sat, Jun 20, 2015 at 12:39 AM, Drake민영근 <dr...@nexr.com> wrote:
> Hi,
>
> You should disable vmem check. See this:
> http://blog.cloudera.com/blog/2014/04/apache-hadoop-yarn-avoiding-6-time-consuming-gotchas/
>
>
> Thanks.
>
> 2015년 6월 17일 수요일, Naganarasimha G R (Naga)<ga...@huawei.com>님이
> 작성한 메시지:
>
> Hi,
>> From the logs its pretty clear its due to
>> *"Current usage: 576.2 MB of 2 GB physical memory used; 4.2 GB of 4.2 GB
>> virtual memory used. Killing container."*
>> Please increase the value yarn.nodemanager.vmem-pmem-ratio from the
>> default value 2 to something like 4 or 8 based on ur app and system.
>>
>> + Naga
>> ------------------------------
>> *From:* Arbi Akhina [arbi.akhina@gmail.com]
>> *Sent:* Wednesday, June 17, 2015 17:19
>> *To:* user@hadoop.apache.org
>> *Subject:* YARN container killed as running beyond memory limits
>>
>> Hi, I've a YARN application that submits containers. In the
>> AplicationMaster logs I see that the container is killed. Here is the logs:
>>
>> Jun 17, 2015 1:31:27 PM com.heavenize.modules.RMCallbackHandler onContainersCompleted
>> INFO: container 'container_1434471275225_0007_01_000002' status is ContainerStatus: [ContainerId: container_1434471275225_0007_01_000002, State: COMPLETE, Diagnostics: Container [pid=4069,containerID=container_1434471275225_0007_01_000002] is running beyond virtual memory limits. Current usage: 576.2 MB of 2 GB physical memory used; 4.2 GB of 4.2 GB virtual memory used. Killing container.
>> Dump of the process-tree for container_1434471275225_0007_01_000002 :
>> |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
>> |- 4094 4093 4069 4069 (java) 2932 94 2916065280 122804 /usr/lib/jvm/java-7-openjdk-amd64/bin/java -Xms512m -Xmx2048m -XX:MaxPermSize=250m -XX:+UseConcMarkSweepGC -Dosmoze.path=/tmp/hadoop-hadoop/nm-local-dir/usercache/hadoop/appcache/container_1434471275225_0007_01_000002/Osmoze -Dspring.profiles.active=webServer -jar /tmp/hadoop-hadoop/nm-local-dir/usercache/hadoop/appcache/container_1434471275225_0007_01_000002/heavenize-modules.jar
>> |- 4093 4073 4069 4069 (sh) 0 0 4550656 164 /bin/sh /tmp/hadoop-hadoop/nm-local-dir/usercache/hadoop/appcache/container_1434471275225_0007_01_000002/startup.sh
>> |- 4073 4069 4069 4069 (java) 249 34 1577267200 24239 /usr/lib/jvm/java-7-openjdk-amd64/bin/java com.heavenize.yarn.task.ModulesManager -containerId container_1434471275225_0007_01_000002 -port 5369 -exe hdfs://hadoop-server/user/hadoop/heavenize/heavenize-modules.jar -conf hdfs://hadoop-server/user/hadoop/heavenize/config.zip
>> |- 4069 1884 4069 4069 (bash) 0 0 12730368 304 /bin/bash -c /usr/lib/jvm/java-7-openjdk-amd64/bin/java com.heavenize.yarn.task.ModulesManager -containerId container_1434471275225_0007_01_000002 -port 5369 -exe hdfs://hadoop-server/user/hadoop/heavenize/heavenize-modules.jar -conf hdfs://hadoop-server/user/hadoop/heavenize/config.zip 1> /usr/local/hadoop/logs/userlogs/application_1434471275225_0007/container_1434471275225_0007_01_000002/stdout 2> /usr/local/hadoop/logs/userlogs/application_1434471275225_0007/container_1434471275225_0007_01_000002/stderr
>>
>>
>> I don't see any memory excess, any idea where this error comes from?
>> There is no errors in the container, it just stop logging as a result
>> of being killed.
>>
>
>
> --
> Drake 민영근 Ph.D
> kt NexR
>
>
Re: YARN container killed as running beyond memory limits
Posted by Gaurav Gupta <ga...@gmail.com>.
You can also change the default value of yarn.nodemanager.vmem-pmem-ratio
On Sat, Jun 20, 2015 at 12:39 AM, Drake민영근 <dr...@nexr.com> wrote:
> Hi,
>
> You should disable vmem check. See this:
> http://blog.cloudera.com/blog/2014/04/apache-hadoop-yarn-avoiding-6-time-consuming-gotchas/
>
>
> Thanks.
>
> 2015년 6월 17일 수요일, Naganarasimha G R (Naga)<ga...@huawei.com>님이
> 작성한 메시지:
>
> Hi,
>> From the logs its pretty clear its due to
>> *"Current usage: 576.2 MB of 2 GB physical memory used; 4.2 GB of 4.2 GB
>> virtual memory used. Killing container."*
>> Please increase the value yarn.nodemanager.vmem-pmem-ratio from the
>> default value 2 to something like 4 or 8 based on ur app and system.
>>
>> + Naga
>> ------------------------------
>> *From:* Arbi Akhina [arbi.akhina@gmail.com]
>> *Sent:* Wednesday, June 17, 2015 17:19
>> *To:* user@hadoop.apache.org
>> *Subject:* YARN container killed as running beyond memory limits
>>
>> Hi, I've a YARN application that submits containers. In the
>> AplicationMaster logs I see that the container is killed. Here is the logs:
>>
>> Jun 17, 2015 1:31:27 PM com.heavenize.modules.RMCallbackHandler onContainersCompleted
>> INFO: container 'container_1434471275225_0007_01_000002' status is ContainerStatus: [ContainerId: container_1434471275225_0007_01_000002, State: COMPLETE, Diagnostics: Container [pid=4069,containerID=container_1434471275225_0007_01_000002] is running beyond virtual memory limits. Current usage: 576.2 MB of 2 GB physical memory used; 4.2 GB of 4.2 GB virtual memory used. Killing container.
>> Dump of the process-tree for container_1434471275225_0007_01_000002 :
>> |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
>> |- 4094 4093 4069 4069 (java) 2932 94 2916065280 122804 /usr/lib/jvm/java-7-openjdk-amd64/bin/java -Xms512m -Xmx2048m -XX:MaxPermSize=250m -XX:+UseConcMarkSweepGC -Dosmoze.path=/tmp/hadoop-hadoop/nm-local-dir/usercache/hadoop/appcache/container_1434471275225_0007_01_000002/Osmoze -Dspring.profiles.active=webServer -jar /tmp/hadoop-hadoop/nm-local-dir/usercache/hadoop/appcache/container_1434471275225_0007_01_000002/heavenize-modules.jar
>> |- 4093 4073 4069 4069 (sh) 0 0 4550656 164 /bin/sh /tmp/hadoop-hadoop/nm-local-dir/usercache/hadoop/appcache/container_1434471275225_0007_01_000002/startup.sh
>> |- 4073 4069 4069 4069 (java) 249 34 1577267200 24239 /usr/lib/jvm/java-7-openjdk-amd64/bin/java com.heavenize.yarn.task.ModulesManager -containerId container_1434471275225_0007_01_000002 -port 5369 -exe hdfs://hadoop-server/user/hadoop/heavenize/heavenize-modules.jar -conf hdfs://hadoop-server/user/hadoop/heavenize/config.zip
>> |- 4069 1884 4069 4069 (bash) 0 0 12730368 304 /bin/bash -c /usr/lib/jvm/java-7-openjdk-amd64/bin/java com.heavenize.yarn.task.ModulesManager -containerId container_1434471275225_0007_01_000002 -port 5369 -exe hdfs://hadoop-server/user/hadoop/heavenize/heavenize-modules.jar -conf hdfs://hadoop-server/user/hadoop/heavenize/config.zip 1> /usr/local/hadoop/logs/userlogs/application_1434471275225_0007/container_1434471275225_0007_01_000002/stdout 2> /usr/local/hadoop/logs/userlogs/application_1434471275225_0007/container_1434471275225_0007_01_000002/stderr
>>
>>
>> I don't see any memory excess, any idea where this error comes from?
>> There is no errors in the container, it just stop logging as a result
>> of being killed.
>>
>
>
> --
> Drake 민영근 Ph.D
> kt NexR
>
>
Re: YARN container killed as running beyond memory limits
Posted by Gaurav Gupta <ga...@gmail.com>.
You can also change the default value of yarn.nodemanager.vmem-pmem-ratio
On Sat, Jun 20, 2015 at 12:39 AM, Drake민영근 <dr...@nexr.com> wrote:
> Hi,
>
> You should disable vmem check. See this:
> http://blog.cloudera.com/blog/2014/04/apache-hadoop-yarn-avoiding-6-time-consuming-gotchas/
>
>
> Thanks.
>
> 2015년 6월 17일 수요일, Naganarasimha G R (Naga)<ga...@huawei.com>님이
> 작성한 메시지:
>
> Hi,
>> From the logs its pretty clear its due to
>> *"Current usage: 576.2 MB of 2 GB physical memory used; 4.2 GB of 4.2 GB
>> virtual memory used. Killing container."*
>> Please increase the value yarn.nodemanager.vmem-pmem-ratio from the
>> default value 2 to something like 4 or 8 based on ur app and system.
>>
>> + Naga
>> ------------------------------
>> *From:* Arbi Akhina [arbi.akhina@gmail.com]
>> *Sent:* Wednesday, June 17, 2015 17:19
>> *To:* user@hadoop.apache.org
>> *Subject:* YARN container killed as running beyond memory limits
>>
>> Hi, I've a YARN application that submits containers. In the
>> AplicationMaster logs I see that the container is killed. Here is the logs:
>>
>> Jun 17, 2015 1:31:27 PM com.heavenize.modules.RMCallbackHandler onContainersCompleted
>> INFO: container 'container_1434471275225_0007_01_000002' status is ContainerStatus: [ContainerId: container_1434471275225_0007_01_000002, State: COMPLETE, Diagnostics: Container [pid=4069,containerID=container_1434471275225_0007_01_000002] is running beyond virtual memory limits. Current usage: 576.2 MB of 2 GB physical memory used; 4.2 GB of 4.2 GB virtual memory used. Killing container.
>> Dump of the process-tree for container_1434471275225_0007_01_000002 :
>> |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
>> |- 4094 4093 4069 4069 (java) 2932 94 2916065280 122804 /usr/lib/jvm/java-7-openjdk-amd64/bin/java -Xms512m -Xmx2048m -XX:MaxPermSize=250m -XX:+UseConcMarkSweepGC -Dosmoze.path=/tmp/hadoop-hadoop/nm-local-dir/usercache/hadoop/appcache/container_1434471275225_0007_01_000002/Osmoze -Dspring.profiles.active=webServer -jar /tmp/hadoop-hadoop/nm-local-dir/usercache/hadoop/appcache/container_1434471275225_0007_01_000002/heavenize-modules.jar
>> |- 4093 4073 4069 4069 (sh) 0 0 4550656 164 /bin/sh /tmp/hadoop-hadoop/nm-local-dir/usercache/hadoop/appcache/container_1434471275225_0007_01_000002/startup.sh
>> |- 4073 4069 4069 4069 (java) 249 34 1577267200 24239 /usr/lib/jvm/java-7-openjdk-amd64/bin/java com.heavenize.yarn.task.ModulesManager -containerId container_1434471275225_0007_01_000002 -port 5369 -exe hdfs://hadoop-server/user/hadoop/heavenize/heavenize-modules.jar -conf hdfs://hadoop-server/user/hadoop/heavenize/config.zip
>> |- 4069 1884 4069 4069 (bash) 0 0 12730368 304 /bin/bash -c /usr/lib/jvm/java-7-openjdk-amd64/bin/java com.heavenize.yarn.task.ModulesManager -containerId container_1434471275225_0007_01_000002 -port 5369 -exe hdfs://hadoop-server/user/hadoop/heavenize/heavenize-modules.jar -conf hdfs://hadoop-server/user/hadoop/heavenize/config.zip 1> /usr/local/hadoop/logs/userlogs/application_1434471275225_0007/container_1434471275225_0007_01_000002/stdout 2> /usr/local/hadoop/logs/userlogs/application_1434471275225_0007/container_1434471275225_0007_01_000002/stderr
>>
>>
>> I don't see any memory excess, any idea where this error comes from?
>> There is no errors in the container, it just stop logging as a result
>> of being killed.
>>
>
>
> --
> Drake 민영근 Ph.D
> kt NexR
>
>
Re: YARN container killed as running beyond memory limits
Posted by Gaurav Gupta <ga...@gmail.com>.
You can also change the default value of yarn.nodemanager.vmem-pmem-ratio
On Sat, Jun 20, 2015 at 12:39 AM, Drake민영근 <dr...@nexr.com> wrote:
> Hi,
>
> You should disable vmem check. See this:
> http://blog.cloudera.com/blog/2014/04/apache-hadoop-yarn-avoiding-6-time-consuming-gotchas/
>
>
> Thanks.
>
> 2015년 6월 17일 수요일, Naganarasimha G R (Naga)<ga...@huawei.com>님이
> 작성한 메시지:
>
> Hi,
>> From the logs its pretty clear its due to
>> *"Current usage: 576.2 MB of 2 GB physical memory used; 4.2 GB of 4.2 GB
>> virtual memory used. Killing container."*
>> Please increase the value yarn.nodemanager.vmem-pmem-ratio from the
>> default value 2 to something like 4 or 8 based on ur app and system.
>>
>> + Naga
>> ------------------------------
>> *From:* Arbi Akhina [arbi.akhina@gmail.com]
>> *Sent:* Wednesday, June 17, 2015 17:19
>> *To:* user@hadoop.apache.org
>> *Subject:* YARN container killed as running beyond memory limits
>>
>> Hi, I've a YARN application that submits containers. In the
>> AplicationMaster logs I see that the container is killed. Here is the logs:
>>
>> Jun 17, 2015 1:31:27 PM com.heavenize.modules.RMCallbackHandler onContainersCompleted
>> INFO: container 'container_1434471275225_0007_01_000002' status is ContainerStatus: [ContainerId: container_1434471275225_0007_01_000002, State: COMPLETE, Diagnostics: Container [pid=4069,containerID=container_1434471275225_0007_01_000002] is running beyond virtual memory limits. Current usage: 576.2 MB of 2 GB physical memory used; 4.2 GB of 4.2 GB virtual memory used. Killing container.
>> Dump of the process-tree for container_1434471275225_0007_01_000002 :
>> |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
>> |- 4094 4093 4069 4069 (java) 2932 94 2916065280 122804 /usr/lib/jvm/java-7-openjdk-amd64/bin/java -Xms512m -Xmx2048m -XX:MaxPermSize=250m -XX:+UseConcMarkSweepGC -Dosmoze.path=/tmp/hadoop-hadoop/nm-local-dir/usercache/hadoop/appcache/container_1434471275225_0007_01_000002/Osmoze -Dspring.profiles.active=webServer -jar /tmp/hadoop-hadoop/nm-local-dir/usercache/hadoop/appcache/container_1434471275225_0007_01_000002/heavenize-modules.jar
>> |- 4093 4073 4069 4069 (sh) 0 0 4550656 164 /bin/sh /tmp/hadoop-hadoop/nm-local-dir/usercache/hadoop/appcache/container_1434471275225_0007_01_000002/startup.sh
>> |- 4073 4069 4069 4069 (java) 249 34 1577267200 24239 /usr/lib/jvm/java-7-openjdk-amd64/bin/java com.heavenize.yarn.task.ModulesManager -containerId container_1434471275225_0007_01_000002 -port 5369 -exe hdfs://hadoop-server/user/hadoop/heavenize/heavenize-modules.jar -conf hdfs://hadoop-server/user/hadoop/heavenize/config.zip
>> |- 4069 1884 4069 4069 (bash) 0 0 12730368 304 /bin/bash -c /usr/lib/jvm/java-7-openjdk-amd64/bin/java com.heavenize.yarn.task.ModulesManager -containerId container_1434471275225_0007_01_000002 -port 5369 -exe hdfs://hadoop-server/user/hadoop/heavenize/heavenize-modules.jar -conf hdfs://hadoop-server/user/hadoop/heavenize/config.zip 1> /usr/local/hadoop/logs/userlogs/application_1434471275225_0007/container_1434471275225_0007_01_000002/stdout 2> /usr/local/hadoop/logs/userlogs/application_1434471275225_0007/container_1434471275225_0007_01_000002/stderr
>>
>>
>> I don't see any memory excess, any idea where this error comes from?
>> There is no errors in the container, it just stop logging as a result
>> of being killed.
>>
>
>
> --
> Drake 민영근 Ph.D
> kt NexR
>
>
Re: YARN container killed as running beyond memory limits
Posted by Drake민영근 <dr...@nexr.com>.
Hi,
You should disable vmem check. See this:
http://blog.cloudera.com/blog/2014/04/apache-hadoop-yarn-avoiding-6-time-consuming-gotchas/
Thanks.
2015년 6월 17일 수요일, Naganarasimha G R (Naga)<ga...@huawei.com>님이
작성한 메시지:
> Hi,
> From the logs its pretty clear its due to
> *"Current usage: 576.2 MB of 2 GB physical memory used; 4.2 GB of 4.2 GB
> virtual memory used. Killing container."*
> Please increase the value yarn.nodemanager.vmem-pmem-ratio from the
> default value 2 to something like 4 or 8 based on ur app and system.
>
> + Naga
> ------------------------------
> *From:* Arbi Akhina [arbi.akhina@gmail.com
> <javascript:_e(%7B%7D,'cvml','arbi.akhina@gmail.com');>]
> *Sent:* Wednesday, June 17, 2015 17:19
> *To:* user@hadoop.apache.org
> <javascript:_e(%7B%7D,'cvml','user@hadoop.apache.org');>
> *Subject:* YARN container killed as running beyond memory limits
>
> Hi, I've a YARN application that submits containers. In the
> AplicationMaster logs I see that the container is killed. Here is the logs:
>
> Jun 17, 2015 1:31:27 PM com.heavenize.modules.RMCallbackHandler onContainersCompleted
> INFO: container 'container_1434471275225_0007_01_000002' status is ContainerStatus: [ContainerId: container_1434471275225_0007_01_000002, State: COMPLETE, Diagnostics: Container [pid=4069,containerID=container_1434471275225_0007_01_000002] is running beyond virtual memory limits. Current usage: 576.2 MB of 2 GB physical memory used; 4.2 GB of 4.2 GB virtual memory used. Killing container.
> Dump of the process-tree for container_1434471275225_0007_01_000002 :
> |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
> |- 4094 4093 4069 4069 (java) 2932 94 2916065280 122804 /usr/lib/jvm/java-7-openjdk-amd64/bin/java -Xms512m -Xmx2048m -XX:MaxPermSize=250m -XX:+UseConcMarkSweepGC -Dosmoze.path=/tmp/hadoop-hadoop/nm-local-dir/usercache/hadoop/appcache/container_1434471275225_0007_01_000002/Osmoze -Dspring.profiles.active=webServer -jar /tmp/hadoop-hadoop/nm-local-dir/usercache/hadoop/appcache/container_1434471275225_0007_01_000002/heavenize-modules.jar
> |- 4093 4073 4069 4069 (sh) 0 0 4550656 164 /bin/sh /tmp/hadoop-hadoop/nm-local-dir/usercache/hadoop/appcache/container_1434471275225_0007_01_000002/startup.sh
> |- 4073 4069 4069 4069 (java) 249 34 1577267200 24239 /usr/lib/jvm/java-7-openjdk-amd64/bin/java com.heavenize.yarn.task.ModulesManager -containerId container_1434471275225_0007_01_000002 -port 5369 -exe hdfs://hadoop-server/user/hadoop/heavenize/heavenize-modules.jar -conf hdfs://hadoop-server/user/hadoop/heavenize/config.zip
> |- 4069 1884 4069 4069 (bash) 0 0 12730368 304 /bin/bash -c /usr/lib/jvm/java-7-openjdk-amd64/bin/java com.heavenize.yarn.task.ModulesManager -containerId container_1434471275225_0007_01_000002 -port 5369 -exe hdfs://hadoop-server/user/hadoop/heavenize/heavenize-modules.jar -conf hdfs://hadoop-server/user/hadoop/heavenize/config.zip 1> /usr/local/hadoop/logs/userlogs/application_1434471275225_0007/container_1434471275225_0007_01_000002/stdout 2> /usr/local/hadoop/logs/userlogs/application_1434471275225_0007/container_1434471275225_0007_01_000002/stderr
>
>
> I don't see any memory excess, any idea where this error comes from?
> There is no errors in the container, it just stop logging as a result of
> being killed.
>
--
Drake 민영근 Ph.D
kt NexR
Re: YARN container killed as running beyond memory limits
Posted by Drake민영근 <dr...@nexr.com>.
Hi,
You should disable vmem check. See this:
http://blog.cloudera.com/blog/2014/04/apache-hadoop-yarn-avoiding-6-time-consuming-gotchas/
Thanks.
2015년 6월 17일 수요일, Naganarasimha G R (Naga)<ga...@huawei.com>님이
작성한 메시지:
> Hi,
> From the logs its pretty clear its due to
> *"Current usage: 576.2 MB of 2 GB physical memory used; 4.2 GB of 4.2 GB
> virtual memory used. Killing container."*
> Please increase the value yarn.nodemanager.vmem-pmem-ratio from the
> default value 2 to something like 4 or 8 based on ur app and system.
>
> + Naga
> ------------------------------
> *From:* Arbi Akhina [arbi.akhina@gmail.com
> <javascript:_e(%7B%7D,'cvml','arbi.akhina@gmail.com');>]
> *Sent:* Wednesday, June 17, 2015 17:19
> *To:* user@hadoop.apache.org
> <javascript:_e(%7B%7D,'cvml','user@hadoop.apache.org');>
> *Subject:* YARN container killed as running beyond memory limits
>
> Hi, I've a YARN application that submits containers. In the
> AplicationMaster logs I see that the container is killed. Here is the logs:
>
> Jun 17, 2015 1:31:27 PM com.heavenize.modules.RMCallbackHandler onContainersCompleted
> INFO: container 'container_1434471275225_0007_01_000002' status is ContainerStatus: [ContainerId: container_1434471275225_0007_01_000002, State: COMPLETE, Diagnostics: Container [pid=4069,containerID=container_1434471275225_0007_01_000002] is running beyond virtual memory limits. Current usage: 576.2 MB of 2 GB physical memory used; 4.2 GB of 4.2 GB virtual memory used. Killing container.
> Dump of the process-tree for container_1434471275225_0007_01_000002 :
> |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
> |- 4094 4093 4069 4069 (java) 2932 94 2916065280 122804 /usr/lib/jvm/java-7-openjdk-amd64/bin/java -Xms512m -Xmx2048m -XX:MaxPermSize=250m -XX:+UseConcMarkSweepGC -Dosmoze.path=/tmp/hadoop-hadoop/nm-local-dir/usercache/hadoop/appcache/container_1434471275225_0007_01_000002/Osmoze -Dspring.profiles.active=webServer -jar /tmp/hadoop-hadoop/nm-local-dir/usercache/hadoop/appcache/container_1434471275225_0007_01_000002/heavenize-modules.jar
> |- 4093 4073 4069 4069 (sh) 0 0 4550656 164 /bin/sh /tmp/hadoop-hadoop/nm-local-dir/usercache/hadoop/appcache/container_1434471275225_0007_01_000002/startup.sh
> |- 4073 4069 4069 4069 (java) 249 34 1577267200 24239 /usr/lib/jvm/java-7-openjdk-amd64/bin/java com.heavenize.yarn.task.ModulesManager -containerId container_1434471275225_0007_01_000002 -port 5369 -exe hdfs://hadoop-server/user/hadoop/heavenize/heavenize-modules.jar -conf hdfs://hadoop-server/user/hadoop/heavenize/config.zip
> |- 4069 1884 4069 4069 (bash) 0 0 12730368 304 /bin/bash -c /usr/lib/jvm/java-7-openjdk-amd64/bin/java com.heavenize.yarn.task.ModulesManager -containerId container_1434471275225_0007_01_000002 -port 5369 -exe hdfs://hadoop-server/user/hadoop/heavenize/heavenize-modules.jar -conf hdfs://hadoop-server/user/hadoop/heavenize/config.zip 1> /usr/local/hadoop/logs/userlogs/application_1434471275225_0007/container_1434471275225_0007_01_000002/stdout 2> /usr/local/hadoop/logs/userlogs/application_1434471275225_0007/container_1434471275225_0007_01_000002/stderr
>
>
> I don't see any memory excess, any idea where this error comes from?
> There is no errors in the container, it just stop logging as a result of
> being killed.
>
--
Drake 민영근 Ph.D
kt NexR
Re: YARN container killed as running beyond memory limits
Posted by Drake민영근 <dr...@nexr.com>.
Hi,
You should disable vmem check. See this:
http://blog.cloudera.com/blog/2014/04/apache-hadoop-yarn-avoiding-6-time-consuming-gotchas/
Thanks.
2015년 6월 17일 수요일, Naganarasimha G R (Naga)<ga...@huawei.com>님이
작성한 메시지:
> Hi,
> From the logs its pretty clear its due to
> *"Current usage: 576.2 MB of 2 GB physical memory used; 4.2 GB of 4.2 GB
> virtual memory used. Killing container."*
> Please increase the value yarn.nodemanager.vmem-pmem-ratio from the
> default value 2 to something like 4 or 8 based on ur app and system.
>
> + Naga
> ------------------------------
> *From:* Arbi Akhina [arbi.akhina@gmail.com
> <javascript:_e(%7B%7D,'cvml','arbi.akhina@gmail.com');>]
> *Sent:* Wednesday, June 17, 2015 17:19
> *To:* user@hadoop.apache.org
> <javascript:_e(%7B%7D,'cvml','user@hadoop.apache.org');>
> *Subject:* YARN container killed as running beyond memory limits
>
> Hi, I've a YARN application that submits containers. In the
> AplicationMaster logs I see that the container is killed. Here is the logs:
>
> Jun 17, 2015 1:31:27 PM com.heavenize.modules.RMCallbackHandler onContainersCompleted
> INFO: container 'container_1434471275225_0007_01_000002' status is ContainerStatus: [ContainerId: container_1434471275225_0007_01_000002, State: COMPLETE, Diagnostics: Container [pid=4069,containerID=container_1434471275225_0007_01_000002] is running beyond virtual memory limits. Current usage: 576.2 MB of 2 GB physical memory used; 4.2 GB of 4.2 GB virtual memory used. Killing container.
> Dump of the process-tree for container_1434471275225_0007_01_000002 :
> |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
> |- 4094 4093 4069 4069 (java) 2932 94 2916065280 122804 /usr/lib/jvm/java-7-openjdk-amd64/bin/java -Xms512m -Xmx2048m -XX:MaxPermSize=250m -XX:+UseConcMarkSweepGC -Dosmoze.path=/tmp/hadoop-hadoop/nm-local-dir/usercache/hadoop/appcache/container_1434471275225_0007_01_000002/Osmoze -Dspring.profiles.active=webServer -jar /tmp/hadoop-hadoop/nm-local-dir/usercache/hadoop/appcache/container_1434471275225_0007_01_000002/heavenize-modules.jar
> |- 4093 4073 4069 4069 (sh) 0 0 4550656 164 /bin/sh /tmp/hadoop-hadoop/nm-local-dir/usercache/hadoop/appcache/container_1434471275225_0007_01_000002/startup.sh
> |- 4073 4069 4069 4069 (java) 249 34 1577267200 24239 /usr/lib/jvm/java-7-openjdk-amd64/bin/java com.heavenize.yarn.task.ModulesManager -containerId container_1434471275225_0007_01_000002 -port 5369 -exe hdfs://hadoop-server/user/hadoop/heavenize/heavenize-modules.jar -conf hdfs://hadoop-server/user/hadoop/heavenize/config.zip
> |- 4069 1884 4069 4069 (bash) 0 0 12730368 304 /bin/bash -c /usr/lib/jvm/java-7-openjdk-amd64/bin/java com.heavenize.yarn.task.ModulesManager -containerId container_1434471275225_0007_01_000002 -port 5369 -exe hdfs://hadoop-server/user/hadoop/heavenize/heavenize-modules.jar -conf hdfs://hadoop-server/user/hadoop/heavenize/config.zip 1> /usr/local/hadoop/logs/userlogs/application_1434471275225_0007/container_1434471275225_0007_01_000002/stdout 2> /usr/local/hadoop/logs/userlogs/application_1434471275225_0007/container_1434471275225_0007_01_000002/stderr
>
>
> I don't see any memory excess, any idea where this error comes from?
> There is no errors in the container, it just stop logging as a result of
> being killed.
>
--
Drake 민영근 Ph.D
kt NexR
Re: YARN container killed as running beyond memory limits
Posted by Drake민영근 <dr...@nexr.com>.
Hi,
You should disable vmem check. See this:
http://blog.cloudera.com/blog/2014/04/apache-hadoop-yarn-avoiding-6-time-consuming-gotchas/
Thanks.
2015년 6월 17일 수요일, Naganarasimha G R (Naga)<ga...@huawei.com>님이
작성한 메시지:
> Hi,
> From the logs its pretty clear its due to
> *"Current usage: 576.2 MB of 2 GB physical memory used; 4.2 GB of 4.2 GB
> virtual memory used. Killing container."*
> Please increase the value yarn.nodemanager.vmem-pmem-ratio from the
> default value 2 to something like 4 or 8 based on ur app and system.
>
> + Naga
> ------------------------------
> *From:* Arbi Akhina [arbi.akhina@gmail.com
> <javascript:_e(%7B%7D,'cvml','arbi.akhina@gmail.com');>]
> *Sent:* Wednesday, June 17, 2015 17:19
> *To:* user@hadoop.apache.org
> <javascript:_e(%7B%7D,'cvml','user@hadoop.apache.org');>
> *Subject:* YARN container killed as running beyond memory limits
>
> Hi, I've a YARN application that submits containers. In the
> AplicationMaster logs I see that the container is killed. Here is the logs:
>
> Jun 17, 2015 1:31:27 PM com.heavenize.modules.RMCallbackHandler onContainersCompleted
> INFO: container 'container_1434471275225_0007_01_000002' status is ContainerStatus: [ContainerId: container_1434471275225_0007_01_000002, State: COMPLETE, Diagnostics: Container [pid=4069,containerID=container_1434471275225_0007_01_000002] is running beyond virtual memory limits. Current usage: 576.2 MB of 2 GB physical memory used; 4.2 GB of 4.2 GB virtual memory used. Killing container.
> Dump of the process-tree for container_1434471275225_0007_01_000002 :
> |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
> |- 4094 4093 4069 4069 (java) 2932 94 2916065280 122804 /usr/lib/jvm/java-7-openjdk-amd64/bin/java -Xms512m -Xmx2048m -XX:MaxPermSize=250m -XX:+UseConcMarkSweepGC -Dosmoze.path=/tmp/hadoop-hadoop/nm-local-dir/usercache/hadoop/appcache/container_1434471275225_0007_01_000002/Osmoze -Dspring.profiles.active=webServer -jar /tmp/hadoop-hadoop/nm-local-dir/usercache/hadoop/appcache/container_1434471275225_0007_01_000002/heavenize-modules.jar
> |- 4093 4073 4069 4069 (sh) 0 0 4550656 164 /bin/sh /tmp/hadoop-hadoop/nm-local-dir/usercache/hadoop/appcache/container_1434471275225_0007_01_000002/startup.sh
> |- 4073 4069 4069 4069 (java) 249 34 1577267200 24239 /usr/lib/jvm/java-7-openjdk-amd64/bin/java com.heavenize.yarn.task.ModulesManager -containerId container_1434471275225_0007_01_000002 -port 5369 -exe hdfs://hadoop-server/user/hadoop/heavenize/heavenize-modules.jar -conf hdfs://hadoop-server/user/hadoop/heavenize/config.zip
> |- 4069 1884 4069 4069 (bash) 0 0 12730368 304 /bin/bash -c /usr/lib/jvm/java-7-openjdk-amd64/bin/java com.heavenize.yarn.task.ModulesManager -containerId container_1434471275225_0007_01_000002 -port 5369 -exe hdfs://hadoop-server/user/hadoop/heavenize/heavenize-modules.jar -conf hdfs://hadoop-server/user/hadoop/heavenize/config.zip 1> /usr/local/hadoop/logs/userlogs/application_1434471275225_0007/container_1434471275225_0007_01_000002/stdout 2> /usr/local/hadoop/logs/userlogs/application_1434471275225_0007/container_1434471275225_0007_01_000002/stderr
>
>
> I don't see any memory excess, any idea where this error comes from?
> There is no errors in the container, it just stop logging as a result of
> being killed.
>
--
Drake 민영근 Ph.D
kt NexR
RE: YARN container killed as running beyond memory limits
Posted by "Naganarasimha G R (Naga)" <ga...@huawei.com>.
Hi,
From the logs its pretty clear its due to
"Current usage: 576.2 MB of 2 GB physical memory used; 4.2 GB of 4.2 GB virtual memory used. Killing container."
Please increase the value yarn.nodemanager.vmem-pmem-ratio from the default value 2 to something like 4 or 8 based on ur app and system.
+ Naga
________________________________
From: Arbi Akhina [arbi.akhina@gmail.com]
Sent: Wednesday, June 17, 2015 17:19
To: user@hadoop.apache.org
Subject: YARN container killed as running beyond memory limits
Hi, I've a YARN application that submits containers. In the AplicationMaster logs I see that the container is killed. Here is the logs:
Jun 17, 2015 1:31:27 PM com.heavenize.modules.RMCallbackHandler onContainersCompleted
INFO: container 'container_1434471275225_0007_01_000002' status is ContainerStatus: [ContainerId: container_1434471275225_0007_01_000002, State: COMPLETE, Diagnostics: Container [pid=4069,containerID=container_1434471275225_0007_01_000002] is running beyond virtual memory limits. Current usage: 576.2 MB of 2 GB physical memory used; 4.2 GB of 4.2 GB virtual memory used. Killing container.
Dump of the process-tree for container_1434471275225_0007_01_000002 :
|- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
|- 4094 4093 4069 4069 (java) 2932 94 2916065280 122804 /usr/lib/jvm/java-7-openjdk-amd64/bin/java -Xms512m -Xmx2048m -XX:MaxPermSize=250m -XX:+UseConcMarkSweepGC -Dosmoze.path=/tmp/hadoop-hadoop/nm-local-dir/usercache/hadoop/appcache/container_1434471275225_0007_01_000002/Osmoze -Dspring.profiles.active=webServer -jar /tmp/hadoop-hadoop/nm-local-dir/usercache/hadoop/appcache/container_1434471275225_0007_01_000002/heavenize-modules.jar
|- 4093 4073 4069 4069 (sh) 0 0 4550656 164 /bin/sh /tmp/hadoop-hadoop/nm-local-dir/usercache/hadoop/appcache/container_1434471275225_0007_01_000002/startup.sh
|- 4073 4069 4069 4069 (java) 249 34 1577267200 24239 /usr/lib/jvm/java-7-openjdk-amd64/bin/java com.heavenize.yarn.task.ModulesManager -containerId container_1434471275225_0007_01_000002 -port 5369 -exe hdfs://hadoop-server/user/hadoop/heavenize/heavenize-modules.jar -conf hdfs://hadoop-server/user/hadoop/heavenize/config.zip
|- 4069 1884 4069 4069 (bash) 0 0 12730368 304 /bin/bash -c /usr/lib/jvm/java-7-openjdk-amd64/bin/java com.heavenize.yarn.task.ModulesManager -containerId container_1434471275225_0007_01_000002 -port 5369 -exe hdfs://hadoop-server/user/hadoop/heavenize/heavenize-modules.jar -conf hdfs://hadoop-server/user/hadoop/heavenize/config.zip 1> /usr/local/hadoop/logs/userlogs/application_1434471275225_0007/container_1434471275225_0007_01_000002/stdout 2> /usr/local/hadoop/logs/userlogs/application_1434471275225_0007/container_1434471275225_0007_01_000002/stderr
I don't see any memory excess, any idea where this error comes from?
There is no errors in the container, it just stop logging as a result of being killed.
RE: YARN container killed as running beyond memory limits
Posted by "Naganarasimha G R (Naga)" <ga...@huawei.com>.
Hi,
From the logs its pretty clear its due to
"Current usage: 576.2 MB of 2 GB physical memory used; 4.2 GB of 4.2 GB virtual memory used. Killing container."
Please increase the value yarn.nodemanager.vmem-pmem-ratio from the default value 2 to something like 4 or 8 based on ur app and system.
+ Naga
________________________________
From: Arbi Akhina [arbi.akhina@gmail.com]
Sent: Wednesday, June 17, 2015 17:19
To: user@hadoop.apache.org
Subject: YARN container killed as running beyond memory limits
Hi, I've a YARN application that submits containers. In the AplicationMaster logs I see that the container is killed. Here is the logs:
Jun 17, 2015 1:31:27 PM com.heavenize.modules.RMCallbackHandler onContainersCompleted
INFO: container 'container_1434471275225_0007_01_000002' status is ContainerStatus: [ContainerId: container_1434471275225_0007_01_000002, State: COMPLETE, Diagnostics: Container [pid=4069,containerID=container_1434471275225_0007_01_000002] is running beyond virtual memory limits. Current usage: 576.2 MB of 2 GB physical memory used; 4.2 GB of 4.2 GB virtual memory used. Killing container.
Dump of the process-tree for container_1434471275225_0007_01_000002 :
|- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
|- 4094 4093 4069 4069 (java) 2932 94 2916065280 122804 /usr/lib/jvm/java-7-openjdk-amd64/bin/java -Xms512m -Xmx2048m -XX:MaxPermSize=250m -XX:+UseConcMarkSweepGC -Dosmoze.path=/tmp/hadoop-hadoop/nm-local-dir/usercache/hadoop/appcache/container_1434471275225_0007_01_000002/Osmoze -Dspring.profiles.active=webServer -jar /tmp/hadoop-hadoop/nm-local-dir/usercache/hadoop/appcache/container_1434471275225_0007_01_000002/heavenize-modules.jar
|- 4093 4073 4069 4069 (sh) 0 0 4550656 164 /bin/sh /tmp/hadoop-hadoop/nm-local-dir/usercache/hadoop/appcache/container_1434471275225_0007_01_000002/startup.sh
|- 4073 4069 4069 4069 (java) 249 34 1577267200 24239 /usr/lib/jvm/java-7-openjdk-amd64/bin/java com.heavenize.yarn.task.ModulesManager -containerId container_1434471275225_0007_01_000002 -port 5369 -exe hdfs://hadoop-server/user/hadoop/heavenize/heavenize-modules.jar -conf hdfs://hadoop-server/user/hadoop/heavenize/config.zip
|- 4069 1884 4069 4069 (bash) 0 0 12730368 304 /bin/bash -c /usr/lib/jvm/java-7-openjdk-amd64/bin/java com.heavenize.yarn.task.ModulesManager -containerId container_1434471275225_0007_01_000002 -port 5369 -exe hdfs://hadoop-server/user/hadoop/heavenize/heavenize-modules.jar -conf hdfs://hadoop-server/user/hadoop/heavenize/config.zip 1> /usr/local/hadoop/logs/userlogs/application_1434471275225_0007/container_1434471275225_0007_01_000002/stdout 2> /usr/local/hadoop/logs/userlogs/application_1434471275225_0007/container_1434471275225_0007_01_000002/stderr
I don't see any memory excess, any idea where this error comes from?
There is no errors in the container, it just stop logging as a result of being killed.
RE: YARN container killed as running beyond memory limits
Posted by "Naganarasimha G R (Naga)" <ga...@huawei.com>.
Hi,
From the logs its pretty clear its due to
"Current usage: 576.2 MB of 2 GB physical memory used; 4.2 GB of 4.2 GB virtual memory used. Killing container."
Please increase the value yarn.nodemanager.vmem-pmem-ratio from the default value 2 to something like 4 or 8 based on ur app and system.
+ Naga
________________________________
From: Arbi Akhina [arbi.akhina@gmail.com]
Sent: Wednesday, June 17, 2015 17:19
To: user@hadoop.apache.org
Subject: YARN container killed as running beyond memory limits
Hi, I've a YARN application that submits containers. In the AplicationMaster logs I see that the container is killed. Here is the logs:
Jun 17, 2015 1:31:27 PM com.heavenize.modules.RMCallbackHandler onContainersCompleted
INFO: container 'container_1434471275225_0007_01_000002' status is ContainerStatus: [ContainerId: container_1434471275225_0007_01_000002, State: COMPLETE, Diagnostics: Container [pid=4069,containerID=container_1434471275225_0007_01_000002] is running beyond virtual memory limits. Current usage: 576.2 MB of 2 GB physical memory used; 4.2 GB of 4.2 GB virtual memory used. Killing container.
Dump of the process-tree for container_1434471275225_0007_01_000002 :
|- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
|- 4094 4093 4069 4069 (java) 2932 94 2916065280 122804 /usr/lib/jvm/java-7-openjdk-amd64/bin/java -Xms512m -Xmx2048m -XX:MaxPermSize=250m -XX:+UseConcMarkSweepGC -Dosmoze.path=/tmp/hadoop-hadoop/nm-local-dir/usercache/hadoop/appcache/container_1434471275225_0007_01_000002/Osmoze -Dspring.profiles.active=webServer -jar /tmp/hadoop-hadoop/nm-local-dir/usercache/hadoop/appcache/container_1434471275225_0007_01_000002/heavenize-modules.jar
|- 4093 4073 4069 4069 (sh) 0 0 4550656 164 /bin/sh /tmp/hadoop-hadoop/nm-local-dir/usercache/hadoop/appcache/container_1434471275225_0007_01_000002/startup.sh
|- 4073 4069 4069 4069 (java) 249 34 1577267200 24239 /usr/lib/jvm/java-7-openjdk-amd64/bin/java com.heavenize.yarn.task.ModulesManager -containerId container_1434471275225_0007_01_000002 -port 5369 -exe hdfs://hadoop-server/user/hadoop/heavenize/heavenize-modules.jar -conf hdfs://hadoop-server/user/hadoop/heavenize/config.zip
|- 4069 1884 4069 4069 (bash) 0 0 12730368 304 /bin/bash -c /usr/lib/jvm/java-7-openjdk-amd64/bin/java com.heavenize.yarn.task.ModulesManager -containerId container_1434471275225_0007_01_000002 -port 5369 -exe hdfs://hadoop-server/user/hadoop/heavenize/heavenize-modules.jar -conf hdfs://hadoop-server/user/hadoop/heavenize/config.zip 1> /usr/local/hadoop/logs/userlogs/application_1434471275225_0007/container_1434471275225_0007_01_000002/stdout 2> /usr/local/hadoop/logs/userlogs/application_1434471275225_0007/container_1434471275225_0007_01_000002/stderr
I don't see any memory excess, any idea where this error comes from?
There is no errors in the container, it just stop logging as a result of being killed.
RE: YARN container killed as running beyond memory limits
Posted by "Naganarasimha G R (Naga)" <ga...@huawei.com>.
Hi,
From the logs its pretty clear its due to
"Current usage: 576.2 MB of 2 GB physical memory used; 4.2 GB of 4.2 GB virtual memory used. Killing container."
Please increase the value yarn.nodemanager.vmem-pmem-ratio from the default value 2 to something like 4 or 8 based on ur app and system.
+ Naga
________________________________
From: Arbi Akhina [arbi.akhina@gmail.com]
Sent: Wednesday, June 17, 2015 17:19
To: user@hadoop.apache.org
Subject: YARN container killed as running beyond memory limits
Hi, I've a YARN application that submits containers. In the AplicationMaster logs I see that the container is killed. Here is the logs:
Jun 17, 2015 1:31:27 PM com.heavenize.modules.RMCallbackHandler onContainersCompleted
INFO: container 'container_1434471275225_0007_01_000002' status is ContainerStatus: [ContainerId: container_1434471275225_0007_01_000002, State: COMPLETE, Diagnostics: Container [pid=4069,containerID=container_1434471275225_0007_01_000002] is running beyond virtual memory limits. Current usage: 576.2 MB of 2 GB physical memory used; 4.2 GB of 4.2 GB virtual memory used. Killing container.
Dump of the process-tree for container_1434471275225_0007_01_000002 :
|- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
|- 4094 4093 4069 4069 (java) 2932 94 2916065280 122804 /usr/lib/jvm/java-7-openjdk-amd64/bin/java -Xms512m -Xmx2048m -XX:MaxPermSize=250m -XX:+UseConcMarkSweepGC -Dosmoze.path=/tmp/hadoop-hadoop/nm-local-dir/usercache/hadoop/appcache/container_1434471275225_0007_01_000002/Osmoze -Dspring.profiles.active=webServer -jar /tmp/hadoop-hadoop/nm-local-dir/usercache/hadoop/appcache/container_1434471275225_0007_01_000002/heavenize-modules.jar
|- 4093 4073 4069 4069 (sh) 0 0 4550656 164 /bin/sh /tmp/hadoop-hadoop/nm-local-dir/usercache/hadoop/appcache/container_1434471275225_0007_01_000002/startup.sh
|- 4073 4069 4069 4069 (java) 249 34 1577267200 24239 /usr/lib/jvm/java-7-openjdk-amd64/bin/java com.heavenize.yarn.task.ModulesManager -containerId container_1434471275225_0007_01_000002 -port 5369 -exe hdfs://hadoop-server/user/hadoop/heavenize/heavenize-modules.jar -conf hdfs://hadoop-server/user/hadoop/heavenize/config.zip
|- 4069 1884 4069 4069 (bash) 0 0 12730368 304 /bin/bash -c /usr/lib/jvm/java-7-openjdk-amd64/bin/java com.heavenize.yarn.task.ModulesManager -containerId container_1434471275225_0007_01_000002 -port 5369 -exe hdfs://hadoop-server/user/hadoop/heavenize/heavenize-modules.jar -conf hdfs://hadoop-server/user/hadoop/heavenize/config.zip 1> /usr/local/hadoop/logs/userlogs/application_1434471275225_0007/container_1434471275225_0007_01_000002/stdout 2> /usr/local/hadoop/logs/userlogs/application_1434471275225_0007/container_1434471275225_0007_01_000002/stderr
I don't see any memory excess, any idea where this error comes from?
There is no errors in the container, it just stop logging as a result of being killed.