You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by Geoffry Roberts <ge...@gmail.com> on 2009/07/27 18:12:33 UTC
What does MAX_FAILED_UNIQUE_FETCHES mean?
All,
I am attempting to run my first map reduce job and I am getting the
following error. Does anyone know what it means?
Shuffle Error: Exceeded MAX_FAILED_UNIQUE_FETCHES; bailing-out.
Full text of the output follows:
09/07/27 07:06:54 WARN mapred.JobClient: Use GenericOptionsParser for
parsing the arguments. Applications should implement Tool for the same.
09/07/27 07:06:55 INFO mapred.FileInputFormat: Total input paths to process
: 1
09/07/27 07:06:56 INFO mapred.JobClient: Running job: job_200907270706_0001
09/07/27 07:06:57 INFO mapred.JobClient: map 0% reduce 0%
09/07/27 07:07:04 INFO mapred.JobClient: map 50% reduce 0%
09/07/27 07:07:08 INFO mapred.JobClient: map 100% reduce 0%
09/07/27 07:07:19 INFO mapred.JobClient: Task Id :
attempt_200907270706_0001_r_000000_0, Status : FAILED
Shuffle Error: Exceeded MAX_FAILED_UNIQUE_FETCHES; bailing-out.
09/07/27 07:07:19 WARN mapred.JobClient: Error reading task
outputhbase3.qn-niat.net
09/07/27 07:07:19 WARN mapred.JobClient: Error reading task
outputhbase3.qn-niat.net
09/07/27 07:07:29 INFO mapred.JobClient: Task Id :
attempt_200907270706_0001_r_000000_1, Status : FAILED
Shuffle Error: Exceeded MAX_FAILED_UNIQUE_FETCHES; bailing-out.
09/07/27 07:07:29 WARN mapred.JobClient: Error reading task
outputhbase3.qn-niat.net
09/07/27 07:07:29 WARN mapred.JobClient: Error reading task
outputhbase3.qn-niat.net
09/07/27 07:07:40 INFO mapred.JobClient: Task Id :
attempt_200907270706_0001_r_000000_2, Status : FAILED
Shuffle Error: Exceeded MAX_FAILED_UNIQUE_FETCHES; bailing-out.
09/07/27 07:07:40 WARN mapred.JobClient: Error reading task
outputhbase3.qn-niat.net
09/07/27 07:07:40 WARN mapred.JobClient: Error reading task
outputhbase3.qn-niat.net
java.io.IOException: Job failed!
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1232)
at qq.mob.mr.first.FirstMapRed.main(FirstMapRed.java:102)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:165)
at org.apache.hadoop.mapred.JobShell.run(JobShell.java:54)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
at org.apache.hadoop.mapred.JobShell.main(JobShell.java:68)
Re: What does MAX_FAILED_UNIQUE_FETCHES mean?
Posted by Jason Venner <ja...@gmail.com>.
I have seen this happen when there are inconsistent hostname to ip address
lookups across the cluster and a node running a reducer is not connecting to
the host that actually has the map output due to getting a different ip
address for the node name.
On Mon, Jul 27, 2009 at 9:46 AM, Geoffry Roberts
<ge...@gmail.com>wrote:
> Thanks for the response.
>
> Now how do I fix this? Is the problem most likely in my MR code? or in my
> hadoop configuration? or what?
>
>
> On Mon, Jul 27, 2009 at 9:33 AM, Harish Mallipeddi <
> harish.mallipeddi@gmail.com> wrote:
>
>>
>> On Mon, Jul 27, 2009 at 9:42 PM, Geoffry Roberts <
>> geoffry.roberts@gmail.com> wrote:
>>
>>> All,
>>>
>>> I am attempting to run my first map reduce job and I am getting the
>>> following error. Does anyone know what it means?
>>>
>>> Shuffle Error: Exceeded MAX_FAILED_UNIQUE_FETCHES; bailing-out.
>>>
>>>
>> After the maps are complete, reducers need to fetch the intermediate
>> map-outputs so they can reduce() them (this is part of the "shuffle" phase).
>> It seems like in your case, for some reason the reducers are unable to fetch
>> the map-ouputs from the corresponding TaskTracker nodes even after
>> MAX_FAILED_UNIQUE_FETCHES attempts. The TaskTrackers (actually a Jetty
>> webserver running on them) are responsible for serving these map-outputs.
>>
>> --
>> Harish Mallipeddi
>> http://blog.poundbang.in
>>
>
>
--
Pro Hadoop, a book to guide you from beginner to hadoop mastery,
http://www.amazon.com/dp/1430219424?tag=jewlerymall
www.prohadoopbook.com a community for Hadoop Professionals
Re: What does MAX_FAILED_UNIQUE_FETCHES mean?
Posted by Geoffry Roberts <ge...@gmail.com>.
Thanks for the response.
Now how do I fix this? Is the problem most likely in my MR code? or in my
hadoop configuration? or what?
On Mon, Jul 27, 2009 at 9:33 AM, Harish Mallipeddi <
harish.mallipeddi@gmail.com> wrote:
>
> On Mon, Jul 27, 2009 at 9:42 PM, Geoffry Roberts <
> geoffry.roberts@gmail.com> wrote:
>
>> All,
>>
>> I am attempting to run my first map reduce job and I am getting the
>> following error. Does anyone know what it means?
>>
>> Shuffle Error: Exceeded MAX_FAILED_UNIQUE_FETCHES; bailing-out.
>>
>>
> After the maps are complete, reducers need to fetch the intermediate
> map-outputs so they can reduce() them (this is part of the "shuffle" phase).
> It seems like in your case, for some reason the reducers are unable to fetch
> the map-ouputs from the corresponding TaskTracker nodes even after
> MAX_FAILED_UNIQUE_FETCHES attempts. The TaskTrackers (actually a Jetty
> webserver running on them) are responsible for serving these map-outputs.
>
> --
> Harish Mallipeddi
> http://blog.poundbang.in
>
Re: What does MAX_FAILED_UNIQUE_FETCHES mean?
Posted by Harish Mallipeddi <ha...@gmail.com>.
On Mon, Jul 27, 2009 at 9:42 PM, Geoffry Roberts
<ge...@gmail.com>wrote:
> All,
>
> I am attempting to run my first map reduce job and I am getting the
> following error. Does anyone know what it means?
>
> Shuffle Error: Exceeded MAX_FAILED_UNIQUE_FETCHES; bailing-out.
>
>
After the maps are complete, reducers need to fetch the intermediate
map-outputs so they can reduce() them (this is part of the "shuffle" phase).
It seems like in your case, for some reason the reducers are unable to fetch
the map-ouputs from the corresponding TaskTracker nodes even after
MAX_FAILED_UNIQUE_FETCHES attempts. The TaskTrackers (actually a Jetty
webserver running on them) are responsible for serving these map-outputs.
--
Harish Mallipeddi
http://blog.poundbang.in