Tasks failing with OutOfMemory after jvm upgrade to 1.6.0_18 or later

[Koji Noguchi <kn...@yahoo-inc.com>]

This is technically a java issue but I thought other hadoop users would find
it interesting.

When we upgraded from old jvm to a newer (32bit) jvm of 1.6.0_21, we started
seeing users' tasks having issues at random places

1. Tasks running 2-3 times slower
2. Tasks failing with OutOfMemory
3. Reducer failing with OutOfMemory while pulling map outpus

Our first response was to increase the heapsize but what actually was
happening was that NewRatio has changed from 8 to 2 after we upgraded jvm to
1.6.0_21.

http://forums.oracle.com/forums/thread.jspa?threadID=2167755
http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6887571

So with  -Xmx640m,  older jvm (1.6.0_17 or before)

old generation went up to

PS Old Generation
   capacity = 596574208 (568.9375MB)
   used     = 596574200 (568.9374923706055MB)
   free     = 8 (7.62939453125E-6MB)
   99.99999865901007% used

but newer jvm(1.6.0_21) only went up to

PS Old Generation
   capacity = 447414272 (426.6875MB)
   used     = 447414264 (426.68749237060547MB)
   free     = 8 (7.62939453125E-6MB)
   99.99999821194797% used


causing all sorts of issues.  To bring back the old behavior, we needed to
add -XX:NewRatio=8 to jvm param.

If you have MAPREDUCE-1207 in your version, you can set
<property>
<name>mapreduce.admin.map.child.java.opts</name>
<value>-XX:NewRatio=8</value>
</property>

<property>
<name>mapreduce.admin.reduce.child.java.opts</name>
<value>-XX:NewRatio=8</value>
</property>

Otherwise, you can set it on 'mapred.child.java.opts' and ask users to also
set it whenever they overwrite that conf.

Maybe this is a known issue to everyone but thought I would share it since
we spent so much time on this simple(?) issue  :)

You can check the NewRatio value by running jmap -histo on the jvm.

Koji