how to prevent JAVA HEAP OOM in shuffle process?

[ch huang <ju...@gmail.com>]

hi,maillist:
i recent get a problem,when i run MR job, it happened OOM in shuffle
process,the options about MR is default,not changed,which option should i
tuning? thanks

Re: how to prevent JAVA HEAP OOM in shuffle process?

[Adam Muise <am...@hortonworks.com>]

Setting map memory at command line with the *new* api:
hadoop jar hadoop-mapreduce-examples-2.2.0.2.0.6.0-76.jar wordcount
-Dmapreduce.map.java.opts=-Xmx1024m /user/hdfs/ades /tmp/wordcount


Setting map memory at command line with the *old* api:
hadoop jar hadoop-mapreduce-examples-2.2.0.2.0.6.0-76.jar wordcount
-Dmapred.map.child.java.opts=-Xmx1024m /user/hdfs/ades /tmp/wordcount

This is more for the user list, not hdfs-dev.


On Mon, Dec 2, 2013 at 7:07 AM, ch huang <ju...@gmail.com> wrote:

> hi,maillist:
> i recent get a problem,when i run MR job, it happened OOM in shuffle
> process,the options about MR is default,not changed,which option should i
> tuning? thanks
>



-- 
   * Adam Muise *       Solutions Engineer
------------------------------

    Phone:        416-417-4037
  Email:      amuise@hortonworks.com
  Website:   http://www.hortonworks.com/

      * Follow Us: *
<http://facebook.com/hortonworks/?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
<http://twitter.com/hortonworks?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
<http://www.linkedin.com/company/hortonworks?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>

 [image: photo]

  Latest From Our Blog:  SAP HANA + Hadoop: A Perfect Match
<http://hortonworks.com/blog/sap-hana-hadoop-a-perfect-match/?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.