You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by ma qiang <ma...@gmail.com> on 2008/02/25 08:48:13 UTC
about the length of Text
Hi all;
A error happen in my program and system print as below:
java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:2786)
at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:94)
at java.io.DataOutputStream.write(DataOutputStream.java:90)
at org.apache.hadoop.io.Text.write(Text.java:243)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:349)
at MyTest.map(UserMinHashMap.java:43)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:192)
at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1787)
and I think the error happen here: output.collect(new
Text(testKeyString),new Text(testValueString));
I'm not sure why? I guess the reason is the length of testValueString
is too large to construct Text or write into HD. Who can tell why?
Thanks very much
Best Wishes!
Re: about the length of Text
Posted by Amar Kamat <am...@yahoo-inc.com>.
What is the heap size (see mapred.child.java.opts in your job.xml)? By the
way what is the job configuration i.e what is the job type (sort etc), number of nodes
etc. Can you provide more details? Anyways try increasing it which you can
do it using
hadoop jar jar-file.jar program -Dmapred.child.java.opts=-Xmx512m <args>
Amar
On Mon, 25 Feb 2008, ma qiang wrote:
> Hi all;
> A error happen in my program and system print as below:
> java.lang.OutOfMemoryError: Java heap space
> at java.util.Arrays.copyOf(Arrays.java:2786)
> at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:94)
> at java.io.DataOutputStream.write(DataOutputStream.java:90)
> at org.apache.hadoop.io.Text.write(Text.java:243)
> at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:349)
> at MyTest.map(UserMinHashMap.java:43)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:192)
> at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1787)
>
> and I think the error happen here: output.collect(new
> Text(testKeyString),new Text(testValueString));
> I'm not sure why? I guess the reason is the length of testValueString
> is too large to construct Text or write into HD. Who can tell why?
>
> Thanks very much
> Best Wishes!
>