You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Runping Qi (JIRA)" <ji...@apache.org> on 2006/04/20 23:11:05 UTC
[jira] Created: (HADOOP-156) Reducer threw IOEOFException
Reducer threw IOEOFException
-----------------------------
Key: HADOOP-156
URL: http://issues.apache.org/jira/browse/HADOOP-156
Project: Hadoop
Type: Bug
Reporter: Runping Qi
A job was running with all the map tasks completed.
The reducers were appending the intermediate files into the large intermediate file.
java.io.EOFException was thrown when the record reader tried to read the version number
during initialization. Here is the stack trace:
java.io.EOFException
at java.io.DataInputStream.readFully(DataInputStream.java:178)
at java.io.DataInputStream.readFully(DataInputStream.java:152)
at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:251)
at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:236)
at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:226)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:205)
at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:709)
Appearantly, the intermediate file was empty. I suspect that one map task
generated empty intermidiate files for all the reducers, since all the reducers
failed at the same place, and failed at the same place during retries.
Unfortunately, we cannot know which map task generated the empty files,
since the exception does not offer any clue.
One simple enhancement is that the record reader should catch IOException and re-throw it with additional
information, such as the file name.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-156) Reducer threw IOEOFException
Posted by "Sanjay Dahiya (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-156?page=comments#action_12435464 ]
Sanjay Dahiya commented on HADOOP-156:
--------------------------------------
ok, the EOFException in the case I posted earlier is not a critical issue. it happened after the job was aborted on job tracker, some task trackers cleaned up map outputs while reduce tasks were still running on others.
However there are genuine EOFExceptions as well which occur in sort phase on reduce tasks, which may be due to malformed map output. e.g. following
2006-09-15 08:04:13,900 INFO org.apache.hadoop.mapred.TaskTracker: task_0001_r_000511_0 0.33333334% reduce > sort
2006-09-15 08:04:13,996 INFO org.apache.hadoop.mapred.TaskRunner: task_0001_r_000511_0 06/09/15 08:04:13 WARN mapred.TaskTracker: Error running child
2006-09-15 08:04:13,996 INFO org.apache.hadoop.mapred.TaskRunner: task_0001_r_000511_0 java.io.EOFException
2006-09-15 08:04:13,996 INFO org.apache.hadoop.mapred.TaskRunner: task_0001_r_000511_0 at java.io.DataInputStream.readFully(DataInputStream.java:178)
2006-09-15 08:04:13,996 INFO org.apache.hadoop.mapred.TaskRunner: task_0001_r_000511_0 at java.io.DataInputStream.readFully(DataInputStream.java:152)
2006-09-15 08:04:13,996 INFO org.apache.hadoop.mapred.TaskRunner: task_0001_r_000511_0 at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:952)
2006-09-15 08:04:13,996 INFO org.apache.hadoop.mapred.TaskRunner: task_0001_r_000511_0 at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:937)
2006-09-15 08:04:13,996 INFO org.apache.hadoop.mapred.TaskRunner: task_0001_r_000511_0 at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:928)
2006-09-15 08:04:13,996 INFO org.apache.hadoop.mapred.TaskRunner: task_0001_r_000511_0 at org.apache.hadoop.io.SequenceFile$Sorter$SortPass.run(SequenceFile.java:1594)
2006-09-15 08:04:13,996 INFO org.apache.hadoop.mapred.TaskRunner: task_0001_r_000511_0 at org.apache.hadoop.io.SequenceFile$Sorter.sortPass(SequenceFile.java:1523)
2006-09-15 08:04:13,996 INFO org.apache.hadoop.mapred.TaskRunner: task_0001_r_000511_0 at org.apache.hadoop.io.SequenceFile$Sorter.sort(SequenceFile.java:1496)
2006-09-15 08:04:13,996 INFO org.apache.hadoop.mapred.TaskRunner: task_0001_r_000511_0 at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:240)
2006-09-15 08:04:13,996 INFO org.apache.hadoop.mapred.TaskRunner: task_0001_r_000511_0 at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1165)
> Reducer threw IOEOFException
> -----------------------------
>
> Key: HADOOP-156
> URL: http://issues.apache.org/jira/browse/HADOOP-156
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.3.0
> Reporter: Runping Qi
>
> A job was running with all the map tasks completed.
> The reducers were appending the intermediate files into the large intermediate file.
> java.io.EOFException was thrown when the record reader tried to read the version number
> during initialization. Here is the stack trace:
> java.io.EOFException
> at java.io.DataInputStream.readFully(DataInputStream.java:178)
> at java.io.DataInputStream.readFully(DataInputStream.java:152)
> at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:251)
> at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:236)
> at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:226)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:205)
> at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:709)
> Appearantly, the intermediate file was empty. I suspect that one map task
> generated empty intermidiate files for all the reducers, since all the reducers
> failed at the same place, and failed at the same place during retries.
> Unfortunately, we cannot know which map task generated the empty files,
> since the exception does not offer any clue.
> One simple enhancement is that the record reader should catch IOException and re-throw it with additional
> information, such as the file name.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HADOOP-156) Reducer threw IOEOFException
Posted by "Sameer Paranjpye (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-156?page=all ]
Sameer Paranjpye updated HADOOP-156:
------------------------------------
Fix Version: 0.4
Version: 0.3
Description:
A job was running with all the map tasks completed.
The reducers were appending the intermediate files into the large intermediate file.
java.io.EOFException was thrown when the record reader tried to read the version number
during initialization. Here is the stack trace:
java.io.EOFException
at java.io.DataInputStream.readFully(DataInputStream.java:178)
at java.io.DataInputStream.readFully(DataInputStream.java:152)
at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:251)
at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:236)
at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:226)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:205)
at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:709)
Appearantly, the intermediate file was empty. I suspect that one map task
generated empty intermidiate files for all the reducers, since all the reducers
failed at the same place, and failed at the same place during retries.
Unfortunately, we cannot know which map task generated the empty files,
since the exception does not offer any clue.
One simple enhancement is that the record reader should catch IOException and re-throw it with additional
information, such as the file name.
was:
A job was running with all the map tasks completed.
The reducers were appending the intermediate files into the large intermediate file.
java.io.EOFException was thrown when the record reader tried to read the version number
during initialization. Here is the stack trace:
java.io.EOFException
at java.io.DataInputStream.readFully(DataInputStream.java:178)
at java.io.DataInputStream.readFully(DataInputStream.java:152)
at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:251)
at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:236)
at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:226)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:205)
at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:709)
Appearantly, the intermediate file was empty. I suspect that one map task
generated empty intermidiate files for all the reducers, since all the reducers
failed at the same place, and failed at the same place during retries.
Unfortunately, we cannot know which map task generated the empty files,
since the exception does not offer any clue.
One simple enhancement is that the record reader should catch IOException and re-throw it with additional
information, such as the file name.
> Reducer threw IOEOFException
> -----------------------------
>
> Key: HADOOP-156
> URL: http://issues.apache.org/jira/browse/HADOOP-156
> Project: Hadoop
> Type: Bug
> Components: mapred
> Versions: 0.3
> Reporter: Runping Qi
> Fix For: 0.4
>
> A job was running with all the map tasks completed.
> The reducers were appending the intermediate files into the large intermediate file.
> java.io.EOFException was thrown when the record reader tried to read the version number
> during initialization. Here is the stack trace:
> java.io.EOFException
> at java.io.DataInputStream.readFully(DataInputStream.java:178)
> at java.io.DataInputStream.readFully(DataInputStream.java:152)
> at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:251)
> at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:236)
> at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:226)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:205)
> at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:709)
> Appearantly, the intermediate file was empty. I suspect that one map task
> generated empty intermidiate files for all the reducers, since all the reducers
> failed at the same place, and failed at the same place during retries.
> Unfortunately, we cannot know which map task generated the empty files,
> since the exception does not offer any clue.
> One simple enhancement is that the record reader should catch IOException and re-throw it with additional
> information, such as the file name.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-156) Reducer threw IOEOFException
Posted by "Sanjay Dahiya (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-156?page=comments#action_12435260 ]
Sanjay Dahiya commented on HADOOP-156:
--------------------------------------
I see the same problem with 0.6.2. But I suspect maps are producing correct data. From the logs it appears the map output was not empty for all reduces as some other reduce tasks read output and were successfully finished. For the map outputs which were not available when reduces asked for it, i can see consistently that task tracker assumes maps are done and deletes the files. after this reduces ask for the map data and the failures cascades.
Here is a common pattern -
2006-09-15 15:15:59,564 INFO org.apache.hadoop.mapred.TaskRunner: task_0001_m_000341_0 done; removing files.
For this task - before it got cleaned up a reduce task copied data successfully and finished.
2006-09-15 02:00:50,537 INFO org.apache.hadoop.mapred.TaskRunner: task_0001_r_000524_0 done copying task_0001_m_000341_0 output from kry2900.inktomisearch.com.
But after the cleanup another reduce task tries to copy and fails -
2006-09-15 15:16:01,794 WARN org.apache.hadoop.mapred.TaskTracker: Http server (getMapOutput.jsp): java.io.FileNotFoundException: /***/hadoop/mapred/local/task_0001_m_000341_0/part-94.out
On reduce task we get same EOFException.
> Reducer threw IOEOFException
> -----------------------------
>
> Key: HADOOP-156
> URL: http://issues.apache.org/jira/browse/HADOOP-156
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.3.0
> Reporter: Runping Qi
>
> A job was running with all the map tasks completed.
> The reducers were appending the intermediate files into the large intermediate file.
> java.io.EOFException was thrown when the record reader tried to read the version number
> during initialization. Here is the stack trace:
> java.io.EOFException
> at java.io.DataInputStream.readFully(DataInputStream.java:178)
> at java.io.DataInputStream.readFully(DataInputStream.java:152)
> at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:251)
> at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:236)
> at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:226)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:205)
> at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:709)
> Appearantly, the intermediate file was empty. I suspect that one map task
> generated empty intermidiate files for all the reducers, since all the reducers
> failed at the same place, and failed at the same place during retries.
> Unfortunately, we cannot know which map task generated the empty files,
> since the exception does not offer any clue.
> One simple enhancement is that the record reader should catch IOException and re-throw it with additional
> information, such as the file name.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HADOOP-156) Reducer threw IOEOFException
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-156?page=all ]
Doug Cutting updated HADOOP-156:
--------------------------------
Fix Version: 0.5.0
(was: 0.4.0)
> Reducer threw IOEOFException
> -----------------------------
>
> Key: HADOOP-156
> URL: http://issues.apache.org/jira/browse/HADOOP-156
> Project: Hadoop
> Type: Bug
> Components: mapred
> Versions: 0.3.0
> Reporter: Runping Qi
> Fix For: 0.5.0
>
> A job was running with all the map tasks completed.
> The reducers were appending the intermediate files into the large intermediate file.
> java.io.EOFException was thrown when the record reader tried to read the version number
> during initialization. Here is the stack trace:
> java.io.EOFException
> at java.io.DataInputStream.readFully(DataInputStream.java:178)
> at java.io.DataInputStream.readFully(DataInputStream.java:152)
> at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:251)
> at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:236)
> at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:226)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:205)
> at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:709)
> Appearantly, the intermediate file was empty. I suspect that one map task
> generated empty intermidiate files for all the reducers, since all the reducers
> failed at the same place, and failed at the same place during retries.
> Unfortunately, we cannot know which map task generated the empty files,
> since the exception does not offer any clue.
> One simple enhancement is that the record reader should catch IOException and re-throw it with additional
> information, such as the file name.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Updated: (HADOOP-156) Reducer threw IOEOFException
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-156?page=all ]
Doug Cutting updated HADOOP-156:
--------------------------------
Component: mapred
> Reducer threw IOEOFException
> -----------------------------
>
> Key: HADOOP-156
> URL: http://issues.apache.org/jira/browse/HADOOP-156
> Project: Hadoop
> Type: Bug
> Components: mapred
> Reporter: Runping Qi
>
> A job was running with all the map tasks completed.
> The reducers were appending the intermediate files into the large intermediate file.
> java.io.EOFException was thrown when the record reader tried to read the version number
> during initialization. Here is the stack trace:
> java.io.EOFException
> at java.io.DataInputStream.readFully(DataInputStream.java:178)
> at java.io.DataInputStream.readFully(DataInputStream.java:152)
> at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:251)
> at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:236)
> at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:226)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:205)
> at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:709)
> Appearantly, the intermediate file was empty. I suspect that one map task
> generated empty intermidiate files for all the reducers, since all the reducers
> failed at the same place, and failed at the same place during retries.
> Unfortunately, we cannot know which map task generated the empty files,
> since the exception does not offer any clue.
> One simple enhancement is that the record reader should catch IOException and re-throw it with additional
> information, such as the file name.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Updated: (HADOOP-156) Reducer threw IOEOFException
Posted by "Sanjay Dahiya (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-156?page=all ]
Sanjay Dahiya updated HADOOP-156:
---------------------------------
Comment: was deleted
> Reducer threw IOEOFException
> -----------------------------
>
> Key: HADOOP-156
> URL: http://issues.apache.org/jira/browse/HADOOP-156
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.3.0
> Reporter: Runping Qi
>
> A job was running with all the map tasks completed.
> The reducers were appending the intermediate files into the large intermediate file.
> java.io.EOFException was thrown when the record reader tried to read the version number
> during initialization. Here is the stack trace:
> java.io.EOFException
> at java.io.DataInputStream.readFully(DataInputStream.java:178)
> at java.io.DataInputStream.readFully(DataInputStream.java:152)
> at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:251)
> at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:236)
> at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:226)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:205)
> at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:709)
> Appearantly, the intermediate file was empty. I suspect that one map task
> generated empty intermidiate files for all the reducers, since all the reducers
> failed at the same place, and failed at the same place during retries.
> Unfortunately, we cannot know which map task generated the empty files,
> since the exception does not offer any clue.
> One simple enhancement is that the record reader should catch IOException and re-throw it with additional
> information, such as the file name.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-156) Reducer threw IOEOFException
Posted by "Johannes Zillmann (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-156?page=comments#action_12435198 ]
Johannes Zillmann commented on HADOOP-156:
------------------------------------------
i've tackled the same problem with hadoop 0.3.2 after setting mapred.reduce.tasks from 'nodes' to 'nodes * mapred.tasktracker.tasks.maximum(==2)' like sugested in the wiki.
java.io.EOFException at java.io.DataInputStream.readFully(DataInputStream.java:178) at java.io.DataInputStream.readFully(DataInputStream.java:152) at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:263) at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:247) at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:237) at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:203) at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:847)
> Reducer threw IOEOFException
> -----------------------------
>
> Key: HADOOP-156
> URL: http://issues.apache.org/jira/browse/HADOOP-156
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.3.0
> Reporter: Runping Qi
>
> A job was running with all the map tasks completed.
> The reducers were appending the intermediate files into the large intermediate file.
> java.io.EOFException was thrown when the record reader tried to read the version number
> during initialization. Here is the stack trace:
> java.io.EOFException
> at java.io.DataInputStream.readFully(DataInputStream.java:178)
> at java.io.DataInputStream.readFully(DataInputStream.java:152)
> at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:251)
> at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:236)
> at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:226)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:205)
> at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:709)
> Appearantly, the intermediate file was empty. I suspect that one map task
> generated empty intermidiate files for all the reducers, since all the reducers
> failed at the same place, and failed at the same place during retries.
> Unfortunately, we cannot know which map task generated the empty files,
> since the exception does not offer any clue.
> One simple enhancement is that the record reader should catch IOException and re-throw it with additional
> information, such as the file name.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-156) Reducer threw IOEOFException
Posted by "Sanjay Dahiya (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-156?page=comments#action_12435447 ]
Sanjay Dahiya commented on HADOOP-156:
--------------------------------------
Also the task tracker running the map task in question was *not* reported lost on jobtracker and hence reinitialized causing map output to cleanup.
> Reducer threw IOEOFException
> -----------------------------
>
> Key: HADOOP-156
> URL: http://issues.apache.org/jira/browse/HADOOP-156
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.3.0
> Reporter: Runping Qi
>
> A job was running with all the map tasks completed.
> The reducers were appending the intermediate files into the large intermediate file.
> java.io.EOFException was thrown when the record reader tried to read the version number
> during initialization. Here is the stack trace:
> java.io.EOFException
> at java.io.DataInputStream.readFully(DataInputStream.java:178)
> at java.io.DataInputStream.readFully(DataInputStream.java:152)
> at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:251)
> at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:236)
> at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:226)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:205)
> at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:709)
> Appearantly, the intermediate file was empty. I suspect that one map task
> generated empty intermidiate files for all the reducers, since all the reducers
> failed at the same place, and failed at the same place during retries.
> Unfortunately, we cannot know which map task generated the empty files,
> since the exception does not offer any clue.
> One simple enhancement is that the record reader should catch IOException and re-throw it with additional
> information, such as the file name.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-156) Reducer threw IOEOFException
Posted by "Sanjay Dahiya (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-156?page=comments#action_12435262 ]
Sanjay Dahiya commented on HADOOP-156:
--------------------------------------
I see the same problem with 0.6.2. But I suspect maps are producing correct data. From the logs it appears the map output was not empty for all reduces as some other reduce tasks read output and were successfully finished. For the map outputs which were not available when reduces asked for it, i can see consistently that task tracker assumes maps are done and deletes the files. after this reduces ask for the map data and the failures cascades.
Here is a common pattern -
2006-09-15 15:15:59,564 INFO org.apache.hadoop.mapred.TaskRunner: task_0001_m_000341_0 done; removing files.
For this task - before it got cleaned up a reduce task copied data successfully and finished.
2006-09-15 02:00:50,537 INFO org.apache.hadoop.mapred.TaskRunner: task_0001_r_000524_0 done copying task_0001_m_000341_0 output.
But after the cleanup another reduce task tries to copy and fails -
2006-09-15 15:16:01,794 WARN org.apache.hadoop.mapred.TaskTracker: Http server (getMapOutput.jsp): java.io.FileNotFoundException: /***/hadoop/mapred/local/task_0001_m_000341_0/part-94.out
On reduce task we get same EOFException.
> Reducer threw IOEOFException
> -----------------------------
>
> Key: HADOOP-156
> URL: http://issues.apache.org/jira/browse/HADOOP-156
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.3.0
> Reporter: Runping Qi
>
> A job was running with all the map tasks completed.
> The reducers were appending the intermediate files into the large intermediate file.
> java.io.EOFException was thrown when the record reader tried to read the version number
> during initialization. Here is the stack trace:
> java.io.EOFException
> at java.io.DataInputStream.readFully(DataInputStream.java:178)
> at java.io.DataInputStream.readFully(DataInputStream.java:152)
> at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:251)
> at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:236)
> at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:226)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:205)
> at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:709)
> Appearantly, the intermediate file was empty. I suspect that one map task
> generated empty intermidiate files for all the reducers, since all the reducers
> failed at the same place, and failed at the same place during retries.
> Unfortunately, we cannot know which map task generated the empty files,
> since the exception does not offer any clue.
> One simple enhancement is that the record reader should catch IOException and re-throw it with additional
> information, such as the file name.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira