You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Owen O'Malley (JIRA)" <ji...@apache.org> on 2006/09/20 18:28:22 UTC
[jira] Created: (HADOOP-552) getMapOutput doesn't reliably detect
errors and throw to the caller
getMapOutput doesn't reliably detect errors and throw to the caller
-------------------------------------------------------------------
Key: HADOOP-552
URL: http://issues.apache.org/jira/browse/HADOOP-552
Project: Hadoop
Issue Type: Bug
Affects Versions: 0.6.2
Reporter: Owen O'Malley
Assigned To: Owen O'Malley
Fix For: 0.7.0
getMapOutput does not reliabily detect errors in transmission, which usually ends up with an EOFException in the reduce's sort:
java.io.EOFException
at java.io.DataInputStream.readFully(DataInputStream.java:178)
at java.io.DataInputStream.readFully(DataInputStream.java:152)
at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:952)
at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:937)
at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:928)
at org.apache.hadoop.io.SequenceFile$Sorter$SortPass.run(SequenceFile.java:1594)
at org.apache.hadoop.io.SequenceFile$Sorter.sortPass(SequenceFile.java:1523)
at org.apache.hadoop.io.SequenceFile$Sorter.sort(SequenceFile.java:1496)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:240)
at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1173)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HADOOP-552) getMapOutput doesn't reliably detect
errors and throw to the caller
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-552?page=all ]
Doug Cutting updated HADOOP-552:
--------------------------------
Status: Resolved (was: Patch Available)
Resolution: Fixed
I just committed this. Thanks, Owen!
> getMapOutput doesn't reliably detect errors and throw to the caller
> -------------------------------------------------------------------
>
> Key: HADOOP-552
> URL: http://issues.apache.org/jira/browse/HADOOP-552
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.6.2
> Reporter: Owen O'Malley
> Assigned To: Owen O'Malley
> Fix For: 0.7.0
>
> Attachments: size-check.patch
>
>
> getMapOutput does not reliabily detect errors in transmission, which usually ends up with an EOFException in the reduce's sort:
> java.io.EOFException
> at java.io.DataInputStream.readFully(DataInputStream.java:178)
> at java.io.DataInputStream.readFully(DataInputStream.java:152)
> at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:952)
> at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:937)
> at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:928)
> at org.apache.hadoop.io.SequenceFile$Sorter$SortPass.run(SequenceFile.java:1594)
> at org.apache.hadoop.io.SequenceFile$Sorter.sortPass(SequenceFile.java:1523)
> at org.apache.hadoop.io.SequenceFile$Sorter.sort(SequenceFile.java:1496)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:240)
> at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1173)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HADOOP-552) getMapOutput doesn't reliably detect
errors and throw to the caller
Posted by "Sameer Paranjpye (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-552?page=all ]
Sameer Paranjpye updated HADOOP-552:
------------------------------------
Component/s: mapred
> getMapOutput doesn't reliably detect errors and throw to the caller
> -------------------------------------------------------------------
>
> Key: HADOOP-552
> URL: http://issues.apache.org/jira/browse/HADOOP-552
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.6.2
> Reporter: Owen O'Malley
> Assigned To: Owen O'Malley
> Fix For: 0.7.0
>
>
> getMapOutput does not reliabily detect errors in transmission, which usually ends up with an EOFException in the reduce's sort:
> java.io.EOFException
> at java.io.DataInputStream.readFully(DataInputStream.java:178)
> at java.io.DataInputStream.readFully(DataInputStream.java:152)
> at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:952)
> at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:937)
> at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:928)
> at org.apache.hadoop.io.SequenceFile$Sorter$SortPass.run(SequenceFile.java:1594)
> at org.apache.hadoop.io.SequenceFile$Sorter.sortPass(SequenceFile.java:1523)
> at org.apache.hadoop.io.SequenceFile$Sorter.sort(SequenceFile.java:1496)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:240)
> at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1173)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HADOOP-552) getMapOutput doesn't reliably detect
errors and throw to the caller
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-552?page=all ]
Owen O'Malley updated HADOOP-552:
---------------------------------
Status: Patch Available (was: Open)
> getMapOutput doesn't reliably detect errors and throw to the caller
> -------------------------------------------------------------------
>
> Key: HADOOP-552
> URL: http://issues.apache.org/jira/browse/HADOOP-552
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.6.2
> Reporter: Owen O'Malley
> Assigned To: Owen O'Malley
> Fix For: 0.7.0
>
> Attachments: size-check.patch
>
>
> getMapOutput does not reliabily detect errors in transmission, which usually ends up with an EOFException in the reduce's sort:
> java.io.EOFException
> at java.io.DataInputStream.readFully(DataInputStream.java:178)
> at java.io.DataInputStream.readFully(DataInputStream.java:152)
> at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:952)
> at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:937)
> at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:928)
> at org.apache.hadoop.io.SequenceFile$Sorter$SortPass.run(SequenceFile.java:1594)
> at org.apache.hadoop.io.SequenceFile$Sorter.sortPass(SequenceFile.java:1523)
> at org.apache.hadoop.io.SequenceFile$Sorter.sort(SequenceFile.java:1496)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:240)
> at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1173)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HADOOP-552) getMapOutput doesn't reliably detect
errors and throw to the caller
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-552?page=all ]
Owen O'Malley updated HADOOP-552:
---------------------------------
Attachment: size-check.patch
Re-write of the MapOutputLocation.getFile to handle errors better including checking the content-length and deleting the partial file if something goes wrong.
> getMapOutput doesn't reliably detect errors and throw to the caller
> -------------------------------------------------------------------
>
> Key: HADOOP-552
> URL: http://issues.apache.org/jira/browse/HADOOP-552
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.6.2
> Reporter: Owen O'Malley
> Assigned To: Owen O'Malley
> Fix For: 0.7.0
>
> Attachments: size-check.patch
>
>
> getMapOutput does not reliabily detect errors in transmission, which usually ends up with an EOFException in the reduce's sort:
> java.io.EOFException
> at java.io.DataInputStream.readFully(DataInputStream.java:178)
> at java.io.DataInputStream.readFully(DataInputStream.java:152)
> at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:952)
> at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:937)
> at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:928)
> at org.apache.hadoop.io.SequenceFile$Sorter$SortPass.run(SequenceFile.java:1594)
> at org.apache.hadoop.io.SequenceFile$Sorter.sortPass(SequenceFile.java:1523)
> at org.apache.hadoop.io.SequenceFile$Sorter.sort(SequenceFile.java:1496)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:240)
> at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1173)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira