You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Hoot Thompson <ho...@ptpnow.com> on 2011/11/13 00:14:26 UTC
heap size problem durning mapreduce
I can't seem to get past this heapsize error. Relevant settings are as
follows:
# The maximum amount of heap to use, in MB. Default is 1000.
export HADOOP_HEAPSIZE=10000
<name>mapred.child.java.opts</name>
<value>-Xmx1048576m</value>
</property>
11/11/12 17:38:27 INFO hpc.Driver: Jar
Name: /home/hadoop/querry/HadoopTest.jar
0 [main] INFO nccs.hpc.Driver - Jar
Name: /home/hadoop/querry/HadoopTest.jar
0 [main] INFO nccs.hpc.Driver - Jar
Name: /home/hadoop/querry/HadoopTest.jar
11/11/12 17:38:27 WARN mapred.JobClient: Use GenericOptionsParser for
parsing the arguments. Applications should implement Tool for the same.
146 [main] WARN org.apache.hadoop.mapred.JobClient - Use
GenericOptionsParser for parsing the arguments. Applications should
implement Tool for the same.
146 [main] WARN org.apache.hadoop.mapred.JobClient - Use
GenericOptionsParser for parsing the arguments. Applications should
implement Tool for the same.
11/11/12 17:38:30 INFO input.FileInputFormat: Total input paths to
process : 2329
3330 [main] INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat
- Total input paths to process : 2329
3330 [main] INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat
- Total input paths to process : 2329
11/11/12 17:38:34 INFO mapred.JobClient: Running job:
job_201111121210_0001
7919 [main] INFO org.apache.hadoop.mapred.JobClient - Running job:
job_201111121210_0001
7919 [main] INFO org.apache.hadoop.mapred.JobClient - Running job:
job_201111121210_0001
11/11/12 17:38:35 INFO mapred.JobClient: map 0% reduce 0%
8922 [main] INFO org.apache.hadoop.mapred.JobClient - map 0% reduce 0%
8922 [main] INFO org.apache.hadoop.mapred.JobClient - map 0% reduce 0%
11/11/12 17:40:37 INFO mapred.JobClient: map 1% reduce 0%
130058 [main] INFO org.apache.hadoop.mapred.JobClient - map 1% reduce
0%
130058 [main] INFO org.apache.hadoop.mapred.JobClient - map 1% reduce
0%
11/11/12 17:42:17 INFO mapred.JobClient: map 2% reduce 0%
230162 [main] INFO org.apache.hadoop.mapred.JobClient - map 2% reduce
0%
230162 [main] INFO org.apache.hadoop.mapred.JobClient - map 2% reduce
0%
11/11/12 17:43:47 INFO mapred.JobClient: map 3% reduce 0%
320251 [main] INFO org.apache.hadoop.mapred.JobClient - map 3% reduce
0%
320251 [main] INFO org.apache.hadoop.mapred.JobClient - map 3% reduce
0%
11/11/12 17:45:26 INFO mapred.JobClient: map 4% reduce 0%
419346 [main] INFO org.apache.hadoop.mapred.JobClient - map 4% reduce
0%
419346 [main] INFO org.apache.hadoop.mapred.JobClient - map 4% reduce
0%
11/11/12 17:46:59 INFO mapred.JobClient: map 5% reduce 0%
512435 [main] INFO org.apache.hadoop.mapred.JobClient - map 5% reduce
0%
512435 [main] INFO org.apache.hadoop.mapred.JobClient - map 5% reduce
0%
11/11/12 17:48:26 INFO mapred.JobClient: Task Id :
attempt_201111121210_0001_r_000000_0, Status : FAILED
599520 [main] INFO org.apache.hadoop.mapred.JobClient - Task Id :
attempt_201111121210_0001_r_000000_0, Status : FAILED
599520 [main] INFO org.apache.hadoop.mapred.JobClient - Task Id :
attempt_201111121210_0001_r_000000_0, Status : FAILED
Error: java.lang.OutOfMemoryError: Java heap space
at org.apache.hadoop.mapred.ReduceTask$ReduceCopier
$MapOutputCopier.shuffleInMemory(ReduceTask.java:1685)
at org.apache.hadoop.mapred.ReduceTask$ReduceCopier
$MapOutputCopier.getMapOutput(ReduceTask.java:1545)
at org.apache.hadoop.mapred.ReduceTask$ReduceCopier
$MapOutputCopier.copyOutput(ReduceTask.java:1394)
at org.apache.hadoop.mapred.ReduceTask$ReduceCopier
$MapOutputCopier.run(ReduceTask.java:1326)
Re: heap size problem durning mapreduce
Posted by Brian Bockelman <bb...@cse.unl.edu>.
Erm - actually, the heap size you are specifying for the child is 1TB if I'm counting numbers correctly. Is it possible that Java is bombing because Linux isn't allowing you to overcommit that much memory?
Brian
On Nov 28, 2011, at 6:21 AM, Harsh J wrote:
> Hoot,
>
> Your settings of 10 GB per child process, and 10 GB per daemon process
> do not make sense.
>
> Take a read at these slides, will help you configure your heap setup
> properly, among other things:
> http://www.slideshare.net/cloudera/hadoop-troubleshooting-101-kate-ting-cloudera
>
> On Sun, Nov 13, 2011 at 4:44 AM, Hoot Thompson <ho...@ptpnow.com> wrote:
>> I can't seem to get past this heapsize error. Relevant settings are as
>> follows:
>>
>> # The maximum amount of heap to use, in MB. Default is 1000.
>> export HADOOP_HEAPSIZE=10000
>>
>> <name>mapred.child.java.opts</name>
>> <value>-Xmx1048576m</value>
>> </property>
>>
>>
>>
>> 11/11/12 17:38:27 INFO hpc.Driver: Jar
>> Name: /home/hadoop/querry/HadoopTest.jar
>> 0 [main] INFO nccs.hpc.Driver - Jar
>> Name: /home/hadoop/querry/HadoopTest.jar
>> 0 [main] INFO nccs.hpc.Driver - Jar
>> Name: /home/hadoop/querry/HadoopTest.jar
>> 11/11/12 17:38:27 WARN mapred.JobClient: Use GenericOptionsParser for
>> parsing the arguments. Applications should implement Tool for the same.
>> 146 [main] WARN org.apache.hadoop.mapred.JobClient - Use
>> GenericOptionsParser for parsing the arguments. Applications should
>> implement Tool for the same.
>> 146 [main] WARN org.apache.hadoop.mapred.JobClient - Use
>> GenericOptionsParser for parsing the arguments. Applications should
>> implement Tool for the same.
>> 11/11/12 17:38:30 INFO input.FileInputFormat: Total input paths to
>> process : 2329
>> 3330 [main] INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat
>> - Total input paths to process : 2329
>> 3330 [main] INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat
>> - Total input paths to process : 2329
>> 11/11/12 17:38:34 INFO mapred.JobClient: Running job:
>> job_201111121210_0001
>> 7919 [main] INFO org.apache.hadoop.mapred.JobClient - Running job:
>> job_201111121210_0001
>> 7919 [main] INFO org.apache.hadoop.mapred.JobClient - Running job:
>> job_201111121210_0001
>> 11/11/12 17:38:35 INFO mapred.JobClient: map 0% reduce 0%
>> 8922 [main] INFO org.apache.hadoop.mapred.JobClient - map 0% reduce 0%
>> 8922 [main] INFO org.apache.hadoop.mapred.JobClient - map 0% reduce 0%
>> 11/11/12 17:40:37 INFO mapred.JobClient: map 1% reduce 0%
>> 130058 [main] INFO org.apache.hadoop.mapred.JobClient - map 1% reduce
>> 0%
>> 130058 [main] INFO org.apache.hadoop.mapred.JobClient - map 1% reduce
>> 0%
>> 11/11/12 17:42:17 INFO mapred.JobClient: map 2% reduce 0%
>> 230162 [main] INFO org.apache.hadoop.mapred.JobClient - map 2% reduce
>> 0%
>> 230162 [main] INFO org.apache.hadoop.mapred.JobClient - map 2% reduce
>> 0%
>> 11/11/12 17:43:47 INFO mapred.JobClient: map 3% reduce 0%
>> 320251 [main] INFO org.apache.hadoop.mapred.JobClient - map 3% reduce
>> 0%
>> 320251 [main] INFO org.apache.hadoop.mapred.JobClient - map 3% reduce
>> 0%
>> 11/11/12 17:45:26 INFO mapred.JobClient: map 4% reduce 0%
>> 419346 [main] INFO org.apache.hadoop.mapred.JobClient - map 4% reduce
>> 0%
>> 419346 [main] INFO org.apache.hadoop.mapred.JobClient - map 4% reduce
>> 0%
>> 11/11/12 17:46:59 INFO mapred.JobClient: map 5% reduce 0%
>> 512435 [main] INFO org.apache.hadoop.mapred.JobClient - map 5% reduce
>> 0%
>> 512435 [main] INFO org.apache.hadoop.mapred.JobClient - map 5% reduce
>> 0%
>> 11/11/12 17:48:26 INFO mapred.JobClient: Task Id :
>> attempt_201111121210_0001_r_000000_0, Status : FAILED
>> 599520 [main] INFO org.apache.hadoop.mapred.JobClient - Task Id :
>> attempt_201111121210_0001_r_000000_0, Status : FAILED
>> 599520 [main] INFO org.apache.hadoop.mapred.JobClient - Task Id :
>> attempt_201111121210_0001_r_000000_0, Status : FAILED
>> Error: java.lang.OutOfMemoryError: Java heap space
>> at org.apache.hadoop.mapred.ReduceTask$ReduceCopier
>> $MapOutputCopier.shuffleInMemory(ReduceTask.java:1685)
>> at org.apache.hadoop.mapred.ReduceTask$ReduceCopier
>> $MapOutputCopier.getMapOutput(ReduceTask.java:1545)
>> at org.apache.hadoop.mapred.ReduceTask$ReduceCopier
>> $MapOutputCopier.copyOutput(ReduceTask.java:1394)
>> at org.apache.hadoop.mapred.ReduceTask$ReduceCopier
>> $MapOutputCopier.run(ReduceTask.java:1326)
>>
>>
>>
>
>
>
> --
> Harsh J
Re: heap size problem durning mapreduce
Posted by Harsh J <ha...@cloudera.com>.
Hoot,
Your settings of 10 GB per child process, and 10 GB per daemon process
do not make sense.
Take a read at these slides, will help you configure your heap setup
properly, among other things:
http://www.slideshare.net/cloudera/hadoop-troubleshooting-101-kate-ting-cloudera
On Sun, Nov 13, 2011 at 4:44 AM, Hoot Thompson <ho...@ptpnow.com> wrote:
> I can't seem to get past this heapsize error. Relevant settings are as
> follows:
>
> # The maximum amount of heap to use, in MB. Default is 1000.
> export HADOOP_HEAPSIZE=10000
>
> <name>mapred.child.java.opts</name>
> <value>-Xmx1048576m</value>
> </property>
>
>
>
> 11/11/12 17:38:27 INFO hpc.Driver: Jar
> Name: /home/hadoop/querry/HadoopTest.jar
> 0 [main] INFO nccs.hpc.Driver - Jar
> Name: /home/hadoop/querry/HadoopTest.jar
> 0 [main] INFO nccs.hpc.Driver - Jar
> Name: /home/hadoop/querry/HadoopTest.jar
> 11/11/12 17:38:27 WARN mapred.JobClient: Use GenericOptionsParser for
> parsing the arguments. Applications should implement Tool for the same.
> 146 [main] WARN org.apache.hadoop.mapred.JobClient - Use
> GenericOptionsParser for parsing the arguments. Applications should
> implement Tool for the same.
> 146 [main] WARN org.apache.hadoop.mapred.JobClient - Use
> GenericOptionsParser for parsing the arguments. Applications should
> implement Tool for the same.
> 11/11/12 17:38:30 INFO input.FileInputFormat: Total input paths to
> process : 2329
> 3330 [main] INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat
> - Total input paths to process : 2329
> 3330 [main] INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat
> - Total input paths to process : 2329
> 11/11/12 17:38:34 INFO mapred.JobClient: Running job:
> job_201111121210_0001
> 7919 [main] INFO org.apache.hadoop.mapred.JobClient - Running job:
> job_201111121210_0001
> 7919 [main] INFO org.apache.hadoop.mapred.JobClient - Running job:
> job_201111121210_0001
> 11/11/12 17:38:35 INFO mapred.JobClient: map 0% reduce 0%
> 8922 [main] INFO org.apache.hadoop.mapred.JobClient - map 0% reduce 0%
> 8922 [main] INFO org.apache.hadoop.mapred.JobClient - map 0% reduce 0%
> 11/11/12 17:40:37 INFO mapred.JobClient: map 1% reduce 0%
> 130058 [main] INFO org.apache.hadoop.mapred.JobClient - map 1% reduce
> 0%
> 130058 [main] INFO org.apache.hadoop.mapred.JobClient - map 1% reduce
> 0%
> 11/11/12 17:42:17 INFO mapred.JobClient: map 2% reduce 0%
> 230162 [main] INFO org.apache.hadoop.mapred.JobClient - map 2% reduce
> 0%
> 230162 [main] INFO org.apache.hadoop.mapred.JobClient - map 2% reduce
> 0%
> 11/11/12 17:43:47 INFO mapred.JobClient: map 3% reduce 0%
> 320251 [main] INFO org.apache.hadoop.mapred.JobClient - map 3% reduce
> 0%
> 320251 [main] INFO org.apache.hadoop.mapred.JobClient - map 3% reduce
> 0%
> 11/11/12 17:45:26 INFO mapred.JobClient: map 4% reduce 0%
> 419346 [main] INFO org.apache.hadoop.mapred.JobClient - map 4% reduce
> 0%
> 419346 [main] INFO org.apache.hadoop.mapred.JobClient - map 4% reduce
> 0%
> 11/11/12 17:46:59 INFO mapred.JobClient: map 5% reduce 0%
> 512435 [main] INFO org.apache.hadoop.mapred.JobClient - map 5% reduce
> 0%
> 512435 [main] INFO org.apache.hadoop.mapred.JobClient - map 5% reduce
> 0%
> 11/11/12 17:48:26 INFO mapred.JobClient: Task Id :
> attempt_201111121210_0001_r_000000_0, Status : FAILED
> 599520 [main] INFO org.apache.hadoop.mapred.JobClient - Task Id :
> attempt_201111121210_0001_r_000000_0, Status : FAILED
> 599520 [main] INFO org.apache.hadoop.mapred.JobClient - Task Id :
> attempt_201111121210_0001_r_000000_0, Status : FAILED
> Error: java.lang.OutOfMemoryError: Java heap space
> at org.apache.hadoop.mapred.ReduceTask$ReduceCopier
> $MapOutputCopier.shuffleInMemory(ReduceTask.java:1685)
> at org.apache.hadoop.mapred.ReduceTask$ReduceCopier
> $MapOutputCopier.getMapOutput(ReduceTask.java:1545)
> at org.apache.hadoop.mapred.ReduceTask$ReduceCopier
> $MapOutputCopier.copyOutput(ReduceTask.java:1394)
> at org.apache.hadoop.mapred.ReduceTask$ReduceCopier
> $MapOutputCopier.run(ReduceTask.java:1326)
>
>
>
--
Harsh J
Re: heap size problem durning mapreduce
Posted by Marcos Luis Ortiz Valmaseda <ma...@googlemail.com>.
Of course, a 32 bits OS can handle only 4 GB of memory (2^32), for
that reason, It's getting that error.
I personally recommend:
- Decreasy the value of HADOOP_HEAPSIZE
- Use a 64 bits Unix/Linux OS
Regards
2011/11/28 Wayne Wan <be...@gmail.com>:
> oh! seems your setting is out of the size that OS(32-bit) can manage!
>
> 2011/11/13 Hoot Thompson <ho...@ptpnow.com>
>
>> I can't seem to get past this heapsize error. Relevant settings are as
>> follows:
>>
>> # The maximum amount of heap to use, in MB. Default is 1000.
>> export HADOOP_HEAPSIZE=10000
>>
>> <name>mapred.child.java.opts</name>
>> <value>-Xmx1048576m</value>
>> </property>
>>
>>
>>
>> 11/11/12 17:38:27 INFO hpc.Driver: Jar
>> Name: /home/hadoop/querry/HadoopTest.jar
>> 0 [main] INFO nccs.hpc.Driver - Jar
>> Name: /home/hadoop/querry/HadoopTest.jar
>> 0 [main] INFO nccs.hpc.Driver - Jar
>> Name: /home/hadoop/querry/HadoopTest.jar
>> 11/11/12 17:38:27 WARN mapred.JobClient: Use GenericOptionsParser for
>> parsing the arguments. Applications should implement Tool for the same.
>> 146 [main] WARN org.apache.hadoop.mapred.JobClient - Use
>> GenericOptionsParser for parsing the arguments. Applications should
>> implement Tool for the same.
>> 146 [main] WARN org.apache.hadoop.mapred.JobClient - Use
>> GenericOptionsParser for parsing the arguments. Applications should
>> implement Tool for the same.
>> 11/11/12 17:38:30 INFO input.FileInputFormat: Total input paths to
>> process : 2329
>> 3330 [main] INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat
>> - Total input paths to process : 2329
>> 3330 [main] INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat
>> - Total input paths to process : 2329
>> 11/11/12 17:38:34 INFO mapred.JobClient: Running job:
>> job_201111121210_0001
>> 7919 [main] INFO org.apache.hadoop.mapred.JobClient - Running job:
>> job_201111121210_0001
>> 7919 [main] INFO org.apache.hadoop.mapred.JobClient - Running job:
>> job_201111121210_0001
>> 11/11/12 17:38:35 INFO mapred.JobClient: map 0% reduce 0%
>> 8922 [main] INFO org.apache.hadoop.mapred.JobClient - map 0% reduce 0%
>> 8922 [main] INFO org.apache.hadoop.mapred.JobClient - map 0% reduce 0%
>> 11/11/12 17:40:37 INFO mapred.JobClient: map 1% reduce 0%
>> 130058 [main] INFO org.apache.hadoop.mapred.JobClient - map 1% reduce
>> 0%
>> 130058 [main] INFO org.apache.hadoop.mapred.JobClient - map 1% reduce
>> 0%
>> 11/11/12 17:42:17 INFO mapred.JobClient: map 2% reduce 0%
>> 230162 [main] INFO org.apache.hadoop.mapred.JobClient - map 2% reduce
>> 0%
>> 230162 [main] INFO org.apache.hadoop.mapred.JobClient - map 2% reduce
>> 0%
>> 11/11/12 17:43:47 INFO mapred.JobClient: map 3% reduce 0%
>> 320251 [main] INFO org.apache.hadoop.mapred.JobClient - map 3% reduce
>> 0%
>> 320251 [main] INFO org.apache.hadoop.mapred.JobClient - map 3% reduce
>> 0%
>> 11/11/12 17:45:26 INFO mapred.JobClient: map 4% reduce 0%
>> 419346 [main] INFO org.apache.hadoop.mapred.JobClient - map 4% reduce
>> 0%
>> 419346 [main] INFO org.apache.hadoop.mapred.JobClient - map 4% reduce
>> 0%
>> 11/11/12 17:46:59 INFO mapred.JobClient: map 5% reduce 0%
>> 512435 [main] INFO org.apache.hadoop.mapred.JobClient - map 5% reduce
>> 0%
>> 512435 [main] INFO org.apache.hadoop.mapred.JobClient - map 5% reduce
>> 0%
>> 11/11/12 17:48:26 INFO mapred.JobClient: Task Id :
>> attempt_201111121210_0001_r_000000_0, Status : FAILED
>> 599520 [main] INFO org.apache.hadoop.mapred.JobClient - Task Id :
>> attempt_201111121210_0001_r_000000_0, Status : FAILED
>> 599520 [main] INFO org.apache.hadoop.mapred.JobClient - Task Id :
>> attempt_201111121210_0001_r_000000_0, Status : FAILED
>> Error: java.lang.OutOfMemoryError: Java heap space
>> at org.apache.hadoop.mapred.ReduceTask$ReduceCopier
>> $MapOutputCopier.shuffleInMemory(ReduceTask.java:1685)
>> at org.apache.hadoop.mapred.ReduceTask$ReduceCopier
>> $MapOutputCopier.getMapOutput(ReduceTask.java:1545)
>> at org.apache.hadoop.mapred.ReduceTask$ReduceCopier
>> $MapOutputCopier.copyOutput(ReduceTask.java:1394)
>> at org.apache.hadoop.mapred.ReduceTask$ReduceCopier
>> $MapOutputCopier.run(ReduceTask.java:1326)
>>
>>
>>
>
--
Marcos Luis Ortíz Valmaseda
Linux Infrastructure Engineer
Linux User # 418229
http://marcosluis2186.posterous.com
http://www.linkedin.com/in/marcosluis2186
Twitter: @marcosluis2186
Re: heap size problem durning mapreduce
Posted by Wayne Wan <be...@gmail.com>.
oh! seems your setting is out of the size that OS(32-bit) can manage!
2011/11/13 Hoot Thompson <ho...@ptpnow.com>
> I can't seem to get past this heapsize error. Relevant settings are as
> follows:
>
> # The maximum amount of heap to use, in MB. Default is 1000.
> export HADOOP_HEAPSIZE=10000
>
> <name>mapred.child.java.opts</name>
> <value>-Xmx1048576m</value>
> </property>
>
>
>
> 11/11/12 17:38:27 INFO hpc.Driver: Jar
> Name: /home/hadoop/querry/HadoopTest.jar
> 0 [main] INFO nccs.hpc.Driver - Jar
> Name: /home/hadoop/querry/HadoopTest.jar
> 0 [main] INFO nccs.hpc.Driver - Jar
> Name: /home/hadoop/querry/HadoopTest.jar
> 11/11/12 17:38:27 WARN mapred.JobClient: Use GenericOptionsParser for
> parsing the arguments. Applications should implement Tool for the same.
> 146 [main] WARN org.apache.hadoop.mapred.JobClient - Use
> GenericOptionsParser for parsing the arguments. Applications should
> implement Tool for the same.
> 146 [main] WARN org.apache.hadoop.mapred.JobClient - Use
> GenericOptionsParser for parsing the arguments. Applications should
> implement Tool for the same.
> 11/11/12 17:38:30 INFO input.FileInputFormat: Total input paths to
> process : 2329
> 3330 [main] INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat
> - Total input paths to process : 2329
> 3330 [main] INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat
> - Total input paths to process : 2329
> 11/11/12 17:38:34 INFO mapred.JobClient: Running job:
> job_201111121210_0001
> 7919 [main] INFO org.apache.hadoop.mapred.JobClient - Running job:
> job_201111121210_0001
> 7919 [main] INFO org.apache.hadoop.mapred.JobClient - Running job:
> job_201111121210_0001
> 11/11/12 17:38:35 INFO mapred.JobClient: map 0% reduce 0%
> 8922 [main] INFO org.apache.hadoop.mapred.JobClient - map 0% reduce 0%
> 8922 [main] INFO org.apache.hadoop.mapred.JobClient - map 0% reduce 0%
> 11/11/12 17:40:37 INFO mapred.JobClient: map 1% reduce 0%
> 130058 [main] INFO org.apache.hadoop.mapred.JobClient - map 1% reduce
> 0%
> 130058 [main] INFO org.apache.hadoop.mapred.JobClient - map 1% reduce
> 0%
> 11/11/12 17:42:17 INFO mapred.JobClient: map 2% reduce 0%
> 230162 [main] INFO org.apache.hadoop.mapred.JobClient - map 2% reduce
> 0%
> 230162 [main] INFO org.apache.hadoop.mapred.JobClient - map 2% reduce
> 0%
> 11/11/12 17:43:47 INFO mapred.JobClient: map 3% reduce 0%
> 320251 [main] INFO org.apache.hadoop.mapred.JobClient - map 3% reduce
> 0%
> 320251 [main] INFO org.apache.hadoop.mapred.JobClient - map 3% reduce
> 0%
> 11/11/12 17:45:26 INFO mapred.JobClient: map 4% reduce 0%
> 419346 [main] INFO org.apache.hadoop.mapred.JobClient - map 4% reduce
> 0%
> 419346 [main] INFO org.apache.hadoop.mapred.JobClient - map 4% reduce
> 0%
> 11/11/12 17:46:59 INFO mapred.JobClient: map 5% reduce 0%
> 512435 [main] INFO org.apache.hadoop.mapred.JobClient - map 5% reduce
> 0%
> 512435 [main] INFO org.apache.hadoop.mapred.JobClient - map 5% reduce
> 0%
> 11/11/12 17:48:26 INFO mapred.JobClient: Task Id :
> attempt_201111121210_0001_r_000000_0, Status : FAILED
> 599520 [main] INFO org.apache.hadoop.mapred.JobClient - Task Id :
> attempt_201111121210_0001_r_000000_0, Status : FAILED
> 599520 [main] INFO org.apache.hadoop.mapred.JobClient - Task Id :
> attempt_201111121210_0001_r_000000_0, Status : FAILED
> Error: java.lang.OutOfMemoryError: Java heap space
> at org.apache.hadoop.mapred.ReduceTask$ReduceCopier
> $MapOutputCopier.shuffleInMemory(ReduceTask.java:1685)
> at org.apache.hadoop.mapred.ReduceTask$ReduceCopier
> $MapOutputCopier.getMapOutput(ReduceTask.java:1545)
> at org.apache.hadoop.mapred.ReduceTask$ReduceCopier
> $MapOutputCopier.copyOutput(ReduceTask.java:1394)
> at org.apache.hadoop.mapred.ReduceTask$ReduceCopier
> $MapOutputCopier.run(ReduceTask.java:1326)
>
>
>