You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by Shing Hing Man <ma...@yahoo.com> on 2011/07/06 22:37:26 UTC
Pseudo mode :Reduce task failed when there are more than one reducers
Hi,
In Pseudo mode, when I have set no number of reducer to 2, I get the following error from a reduce task.
11/07/06 21:15:59 INFO mapreduce.Job: Task Id : attempt_201107062043_0002_r_000000_2, Status : FAILED
org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in shuffle in OnDiskMerger - Thread to merge on-disk map-outputs
at org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:124)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:362)
at org.apache.hadoop.mapred.Child$4.run(Child.java:217)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:742)
at org.apache.hadoop.mapred.Child.main(Child.java:211)
Caused by: java.lang.RuntimeException: java.io.EOFException
at org.apache.hadoop.io.WritableComparator.compare(WritableComparator.java:132)
at org.apache.hadoop.mapred.Merger$MergeQueue.lessThan(Merger.java:530)
at org.apache.hadoop.util.PriorityQueue.downHeap(PriorityQueue.java:141)
at org.apache.hadoop.util.PriorityQueue.adjustTop(PriorityQueue.java:108)
at org.apache.hadoop.mapred.Merger$MergeQueue.adjustPriorityQueue(Mer
But when I set number of reducer to 1, the job completes without the above error.
I am using Hadoop 0.21.0.
Thanks in advance for any assistance!
Shing
Re: Pseudo mode :Reduce task failed when there are more than one reducers
Posted by Shing Hing Man <ma...@yahoo.com>.
After setting mapred.child.java.opts=-Xmx512m (the default was 200m), I no longer get the exception.
Shing
________________________________
From: Shing Hing Man <ma...@yahoo.com>
To: "mapreduce-user@hadoop.apache.org" <ma...@hadoop.apache.org>
Sent: Wednesday, 6 July 2011, 21:37
Subject: Pseudo mode :Reduce task failed when there are more than one reducers
Hi,
In Pseudo mode, when I have set no number of reducer to 2, I get the following error from a reduce task.
11/07/06 21:15:59 INFO mapreduce.Job: Task Id : attempt_201107062043_0002_r_000000_2, Status : FAILED
org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in shuffle in OnDiskMerger - Thread to merge on-disk map-outputs
at org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:124)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:362)
at org.apache.hadoop.mapred.Child$4.run(Child.java:217)
at java.security.AccessController.doPrivileged(Native
Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:742)
at org.apache.hadoop.mapred.Child.main(Child.java:211)
Caused by: java.lang.RuntimeException: java.io.EOFException
at org.apache.hadoop.io.WritableComparator.compare(WritableComparator.java:132)
at org.apache.hadoop.mapred.Merger$MergeQueue.lessThan(Merger.java:530)
at org.apache.hadoop.util.PriorityQueue.downHeap(PriorityQueue.java:141)
at org.apache.hadoop.util.PriorityQueue.adjustTop(PriorityQueue.java:108)
at
org.apache.hadoop.mapred.Merger$MergeQueue.adjustPriorityQueue(Mer
But when I set number of reducer to 1, the job completes without the above error.
I am using Hadoop 0.21.0.
Thanks in advance for any assistance!
Shing