Application failing due to physical container storage overflow (Nutch 2.3.1 + Hadoop 2.7.1 + Yarn)

["shubham.gupta" <sh...@orkash.com>]

Hey

The running nutch job fails due to the following error:


Container [pid=6179,containerID=container_1473334555047_0003_01_000015] 
is running beyond physical memory limits. Current usage: 4.1 GB of 4 GB 
physical memory used; 8.4 GB of 8.4 GB virtual memory used. Killing 
container.

The *configurations of mapred-site.xml* file used are:
     <configuration>
<property>
  <name>mapreduce.map.log.level</name>
  <value>ERROR</value>
</property>
<property>
  <name>mapreduce.reduce.log.level</name>
  <value>ERROR</value>
</property>


<property>
  <name>mapreduce.framework.name</name>
  <value>yarn</value>
</property>
<property>
<name>mapreduce.map.memory.mb</name>
<value>3572</value>
</property>
<property>
<name>mapreduce.reduce.memory.mb</name>
<value>4096</value>
</property>
<property>
<name>mapreduce.reduce.java.opts</name>
<value>-Xmx3765m</value>
</property>
<property>
<name>mapreduce.map.java.opts</name>
<value>-Xmx3265m</value>
</property>
<property>
     <name>yarn.app.mapreduce.am.resource.mb</name>
     <value>1228</value>
</property>
<property>
     <name>yarn.app.mapreduce.am.command-opts</name>
     <value>-Xmx983m</value>
</property>


The*properties of yarn-site.xml *are as follows:

<property>
     <name>yarn.scheduler.maximum-allocation-mb</name>
     <value>5120</value>
    <description>maximum memory allcated to containers.</description>
</property>
<property>
<name>yarn.scheduler.minimum-allocation-vcores</name>
     <value>1</value>
</property>
<property>
<name>yarn.scheduler.maximum-allocation-vcores</name>
     <value>4</value>
  </property>
<property>
    <name>yarn.nodemanager.resource.memory-mb</name>
    <value>12288</value>
<description>max memory allcated to nodemanager.</description>
</property>
<property>
  <name>yarn.nodemanager.vmem-pmem-ratio</name>
  <value>2.1</value>
</property>

Available memory is 8 GB RAM.

What are the best configurations that can be done to avoid this error.

-- 

Shubham Gupta