You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Amandeep Khurana <am...@gmail.com> on 2009/02/07 00:39:54 UTC

Heap size error

I'm getting the following error while running my hadoop job:

09/02/06 15:33:03 INFO mapred.JobClient: Task Id :
attempt_200902061333_0004_r_000000_1, Status : FAILED
java.lang.OutOfMemoryError: Java heap space
        at java.util.Arrays.copyOf(Unknown Source)
        at java.lang.AbstractStringBuilder.expandCapacity(Unknown Source)
        at java.lang.AbstractStringBuilder.append(Unknown Source)
        at java.lang.StringBuffer.append(Unknown Source)
        at TableJoin$Reduce.reduce(TableJoin.java:61)
        at TableJoin$Reduce.reduce(TableJoin.java:1)
        at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:430)
        at org.apache.hadoop.mapred.Child.main(Child.java:155)

Any inputs?

Amandeep


Amandeep Khurana
Computer Science Graduate Student
University of California, Santa Cruz

Re: Heap size error

Posted by jason hadoop <ja...@gmail.com>.
The default task memory allocation size is set in the hadoop-default.xml
file for your configuration and is usually
The parameter is mapred.child.java.opts, and the value is generally
-Xmx200m.

You may alter this value in your JobConf object before you submit the job
and the individual tasks will use the altered value


If the variable that contains your JobConf object is named conf,
*
conf.set( "mapred.child.java.opts", "-Xmx512m");*

will override any existing value from your configuation with the value
"-Xmx512m", for job are are about to launch.


A way to do this that, in general, will preserve any values, with the sun
JDK would be to:
*
conf.set( "mapred.child.java.opts", conf.get("mapred.child.java.opts","") +
" -Xmx512m");*

The above line will append -Xmx512m to the current value of the
mapred.child.java.opts parameter, and use the value of "" if there is no
value set, or the value is null.

It of course may the that your application is using more memory than you
expect do to an incorrect assumption or programming error, and the above
will not be effective.


The hadoop script, in the bin directory of your installation, provides a way
to pass arguments to the
On Sat, Feb 7, 2009 at 5:54 AM, Rasit OZDAS <ra...@gmail.com> wrote:

> Hi, Amandeep,
> I've copied following lines from a site:
> ----------
> Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
>
> This can have two reasons:
>
>    * Your Java application has a memory leak. There are tools like
> YourKit Java Profiler that help you to identify such leaks.
>    * Your Java application really needs a lot of memory (more than
> 128 MB by default!). In this case the Java heap size can be increased
> using the following runtime parameters:
>
> java -Xms<initial heap size> -Xmx<maximum heap size>
>
> Defaults are:
>
> java -Xms32m -Xmx128m
>
> You can set this either in the Java Control Panel or on the command
> line, depending on the environment you run your application.
> ---------
>
> Hope this helps,
> Rasit
>
> 2009/2/7 Amandeep Khurana <am...@gmail.com>:
> > I'm getting the following error while running my hadoop job:
> >
> > 09/02/06 15:33:03 INFO mapred.JobClient: Task Id :
> > attempt_200902061333_0004_r_000000_1, Status : FAILED
> > java.lang.OutOfMemoryError: Java heap space
> >        at java.util.Arrays.copyOf(Unknown Source)
> >        at java.lang.AbstractStringBuilder.expandCapacity(Unknown Source)
> >        at java.lang.AbstractStringBuilder.append(Unknown Source)
> >        at java.lang.StringBuffer.append(Unknown Source)
> >        at TableJoin$Reduce.reduce(TableJoin.java:61)
> >        at TableJoin$Reduce.reduce(TableJoin.java:1)
> >        at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:430)
> >        at org.apache.hadoop.mapred.Child.main(Child.java:155)
> >
> > Any inputs?
> >
> > Amandeep
> >
> >
> > Amandeep Khurana
> > Computer Science Graduate Student
> > University of California, Santa Cruz
> >
>
>
>
> --
> M. Raşit ÖZDAŞ
>

Re: Heap size error

Posted by Rasit OZDAS <ra...@gmail.com>.
Hi, Amandeep,
I've copied following lines from a site:
----------
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space

This can have two reasons:

    * Your Java application has a memory leak. There are tools like
YourKit Java Profiler that help you to identify such leaks.
    * Your Java application really needs a lot of memory (more than
128 MB by default!). In this case the Java heap size can be increased
using the following runtime parameters:

java -Xms<initial heap size> -Xmx<maximum heap size>

Defaults are:

java -Xms32m -Xmx128m

You can set this either in the Java Control Panel or on the command
line, depending on the environment you run your application.
---------

Hope this helps,
Rasit

2009/2/7 Amandeep Khurana <am...@gmail.com>:
> I'm getting the following error while running my hadoop job:
>
> 09/02/06 15:33:03 INFO mapred.JobClient: Task Id :
> attempt_200902061333_0004_r_000000_1, Status : FAILED
> java.lang.OutOfMemoryError: Java heap space
>        at java.util.Arrays.copyOf(Unknown Source)
>        at java.lang.AbstractStringBuilder.expandCapacity(Unknown Source)
>        at java.lang.AbstractStringBuilder.append(Unknown Source)
>        at java.lang.StringBuffer.append(Unknown Source)
>        at TableJoin$Reduce.reduce(TableJoin.java:61)
>        at TableJoin$Reduce.reduce(TableJoin.java:1)
>        at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:430)
>        at org.apache.hadoop.mapred.Child.main(Child.java:155)
>
> Any inputs?
>
> Amandeep
>
>
> Amandeep Khurana
> Computer Science Graduate Student
> University of California, Santa Cruz
>



-- 
M. Raşit ÖZDAŞ