You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by ma qiang <ma...@gmail.com> on 2008/02/25 08:48:13 UTC

about the length of Text

Hi all;
    A error happen in my program and system print as below:
        java.lang.OutOfMemoryError: Java heap space
	at java.util.Arrays.copyOf(Arrays.java:2786)
	at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:94)
	at java.io.DataOutputStream.write(DataOutputStream.java:90)
	at org.apache.hadoop.io.Text.write(Text.java:243)
	at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:349)
	at MyTest.map(UserMinHashMap.java:43)
	at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:192)
	at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1787)

and I think the error happen here: output.collect(new
Text(testKeyString),new Text(testValueString));
I'm not sure why? I guess the reason is the length of testValueString
is too large to construct Text or write into HD.   Who can tell why?

Thanks very much
Best Wishes!

Re: about the length of Text

Posted by Amar Kamat <am...@yahoo-inc.com>.
What is the heap size (see mapred.child.java.opts in your job.xml)? By the 
way what is the job configuration i.e what is the job type (sort etc), number of nodes 
etc. Can you provide more details? Anyways try increasing it which you can 
do it using
hadoop jar jar-file.jar program -Dmapred.child.java.opts=-Xmx512m <args>

Amar
On Mon, 25 Feb 2008, ma qiang wrote:

> Hi all;
>    A error happen in my program and system print as below:
>        java.lang.OutOfMemoryError: Java heap space
> 	at java.util.Arrays.copyOf(Arrays.java:2786)
> 	at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:94)
> 	at java.io.DataOutputStream.write(DataOutputStream.java:90)
> 	at org.apache.hadoop.io.Text.write(Text.java:243)
> 	at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:349)
> 	at MyTest.map(UserMinHashMap.java:43)
> 	at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
> 	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:192)
> 	at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1787)
>
> and I think the error happen here: output.collect(new
> Text(testKeyString),new Text(testValueString));
> I'm not sure why? I guess the reason is the length of testValueString
> is too large to construct Text or write into HD.   Who can tell why?
>
> Thanks very much
> Best Wishes!
>