Tasks running out of memory and mapred.child.ulimit

[Steve Lewis <lo...@gmail.com>]

I have a small hadoop task which is running out of memory on a colleague's
cluster.
I looked at has mapred-site.xml and find

  <property>
    <name>mapred.child.java.opts</name>
    <value>-Xmx512M</value>
  </property>
  <property>
    <name>mapred.child.ulimit</name>
    <value>1572864</value>
  </property>

I set mapred.child.java.opts  <value>-Xmx1024M</value>

but in my cluster there is no value for mapred.child.ulimit
I do not usderstand this parameter and wonder
    what the default is?
    what value is appropriate for -Xmx1024M?
   what value is appropriate for -Xmx512M?

Are there other parameters I should look at

-- 
Steven M. Lewis PhD
4221 105th Ave NE
Kirkland, WA 98033
206-384-1340 (cell)
Skype lordjoe_com

Re: Tasks running out of memory and mapred.child.ulimit

[Joey Echeverria <jo...@cloudera.com>]

The ulimit should be set to 1.5 times the heap. One thing to note is the
unit is on KB.

-Joey
On Sep 30, 2011 1:24 PM, "Steve Lewis" <lo...@gmail.com> wrote:
> I have a small hadoop task which is running out of memory on a colleague's
> cluster.
> I looked at has mapred-site.xml and find
>
> <property>
> <name>mapred.child.java.opts</name>
> <value>-Xmx512M</value>
> </property>
> <property>
> <name>mapred.child.ulimit</name>
> <value>1572864</value>
> </property>
>
> I set mapred.child.java.opts <value>-Xmx1024M</value>
>
> but in my cluster there is no value for mapred.child.ulimit
> I do not usderstand this parameter and wonder
> what the default is?
> what value is appropriate for -Xmx1024M?
> what value is appropriate for -Xmx512M?
>
> Are there other parameters I should look at
>
> --
> Steven M. Lewis PhD
> 4221 105th Ave NE
> Kirkland, WA 98033
> 206-384-1340 (cell)
> Skype lordjoe_com