You are viewing a plain text version of this content. The canonical link for it is here.

Posted to common-user@hadoop.apache.org by Mori Bellamy <mb...@apple.com> on 2008/06/28 01:26:14 UTC

hadoop map reduce: additional heap space?

hey all,
i was wondering if theres a way to allocate more heap space for each  
mapper and reducer process that hadoop spawns. i'm getting this error:

java.lang.OutOfMemoryError: Java heap space
	at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:95)
	at java.io.DataOutputStream.write(DataOutputStream.java:90)
	at java.io.DataOutputStream.writeUTF(DataOutputStream.java:384)
	at java.io.DataOutputStream.writeUTF(DataOutputStream.java:306)
...

(i've written a custom class to use as the key/value pairs)

does anybody know how to allocate more heap space?? thanks!

Re: hadoop map reduce: additional heap space?

Posted by Arun C Murthy <ac...@yahoo-inc.com>.

On Jun 27, 2008, at 4:26 PM, Mori Bellamy wrote:

> hey all,
> i was wondering if theres a way to allocate more heap space for  
> each mapper and reducer process that hadoop spawns. i'm getting  
> this error:
>

Use the 'mapred.child.java.opts' parameter:
http://hadoop.apache.org/core/docs/current/mapred_tutorial.html#Task 
+Execution+%26+Environment

Arun

> java.lang.OutOfMemoryError: Java heap space
> 	at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:95)
> 	at java.io.DataOutputStream.write(DataOutputStream.java:90)
> 	at java.io.DataOutputStream.writeUTF(DataOutputStream.java:384)
> 	at java.io.DataOutputStream.writeUTF(DataOutputStream.java:306)
> ...
>
> (i've written a custom class to use as the key/value pairs)
>
> does anybody know how to allocate more heap space?? thanks!