Hadoop 2.7.2 Yarn Memory Utiliziation

[George Liaw <ge...@gmail.com>]

Hi,
I'm playing around with setting up a Hadoop 2.7.2 cluster with some configs
that I inherited and I'm running into an issue where YARN isn't utilizing
all of the memory available on the nodemanagers and seems to be limited to
2-3 map tasks for an application for some reason. Can anyone shed some
light on this or let me know what else I should look into?

Attached screenshots below of what I'm seeing.

Relevant configs that I'm aware of:
*yarn-site.xml:*

<property>
  <name>yarn.scheduler.minimum-allocation-mb</name>
  <value>128</value>
</property>

<property>
  <name>yarn.scheduler.maximum-allocation-mb</name>
  <value>4096</value>
</property>

*mapred-site.xml:*

<property>
  <name>mapreduce.map.memory.mb</name>
  <value>4096</value>
</property>

<property>
  <name>mapreduce.reduce.memory.mb</name>
  <value>4096</value>
</property>

[image: Inline image 5][image: Inline image 4][image: Inline image 6]

Thanks,
George

-- 
George A. Liaw

RE: Hadoop 2.7.2 Yarn Memory Utiliziation

["Guttadauro, Jeff" <je...@here.com>]

I encountered under-utilization while moving in particular one of my MR jobs to YARN/Hadoop 2.7 a while back.  While it will probably depend on what your job is doing, in my case, I think the biggest improvement came for me when I increased the split size (from its default of 128M up to 2G) since it seemed like my mappers were finishing very quickly, so my containers were very short-lived, and I was suffering from the impact of allocation/deallocation of container resources.  I also bumped up memory on the resource manager and did a couple other things.

Here’s the thread:
https://mail-archives.apache.org/mod_mbox/hadoop-user/201605.mbox/%3CAM2PR04MB07084248711DC9138D805A9990420@AM2PR04MB0708.eurprd04.prod.outlook.com%3E

Hope this helps!  Good luck!
-Jeff

From: George Liaw [mailto:george.a.liaw@gmail.com]
Sent: Friday, October 07, 2016 2:32 PM
To: user@hadoop.apache.org
Subject: Hadoop 2.7.2 Yarn Memory Utiliziation

Hi,
I'm playing around with setting up a Hadoop 2.7.2 cluster with some configs that I inherited and I'm running into an issue where YARN isn't utilizing all of the memory available on the nodemanagers and seems to be limited to 2-3 map tasks for an application for some reason. Can anyone shed some light on this or let me know what else I should look into?

Attached screenshots below of what I'm seeing.

Relevant configs that I'm aware of:
yarn-site.xml:

<property>
  <name>yarn.scheduler.minimum-allocation-mb</name>
  <value>128</value>
</property>

<property>
  <name>yarn.scheduler.maximum-allocation-mb</name>
  <value>4096</value>
</property>
mapred-site.xml:

<property>
  <name>mapreduce.map.memory.mb</name>
  <value>4096</value>
</property>

<property>
  <name>mapreduce.reduce.memory.mb</name>
  <value>4096</value>
</property>
[Inline image 5][Inline image 4][Inline image 6]

Thanks,
George

--
George A. Liaw