You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Devaraj Das (JIRA)" <ji...@apache.org> on 2008/09/22 06:35:44 UTC
[jira] Updated: (HADOOP-4163) If a reducer failed at shuffling
stage, the task should fail, not just logging an exception
[ https://issues.apache.org/jira/browse/HADOOP-4163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Devaraj Das updated HADOOP-4163:
--------------------------------
Description:
I saw a reducer stuck at the shuffling stage, with the following exception logged in the log file:
2008-08-30 00:16:23,265 ERROR org.apache.hadoop.mapred.ReduceTask: Map output copy failure: org.apache.hadoop.fs.FSError: java.io.IOException: No space left on device
at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.write(RawLocalFileSystem.java:199)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
at java.io.FilterOutputStream.close(FilterOutputStream.java:140)
at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:59)
at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:79)
at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.close(ChecksumFileSystem.java:332)
at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:59)
at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:79)
at org.apache.hadoop.mapred.MapOutputLocation.getFile(MapOutputLocation.java:185)
at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.copyOutput(ReduceTask.java:815)
at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.run(ReduceTask.java:764)
Caused by: java.io.IOException: No space left on device
at java.io.FileOutputStream.writeBytes(Native Method)
at java.io.FileOutputStream.write(FileOutputStream.java:260)
at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.write(RawLocalFileSystem.java:197)
... 11 more
2008-08-30 00:16:23,320 WARN org.apache.hadoop.mapred.TaskTracker: Error running child
java.io.IOException: task_200808291851_0001_r_000023_0The reduce copier failed
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:329)
at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2122)
The task should have died.
was:
I saw a reducer stuck at the shuffling stage, with the following exception logged in the log file:
2008-08-30 00:16:23,265 ERROR org.apache.hadoop.mapred.ReduceTask: Map output copy failure: org.apache.hadoop.fs.FSError: java.io.IOException: No space left on device
at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.write(RawLocalFileSystem.java:199)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
at java.io.FilterOutputStream.close(FilterOutputStream.java:140)
at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:59)
at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:79)
at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.close(ChecksumFileSystem.java:332)
at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:59)
at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:79)
at org.apache.hadoop.mapred.MapOutputLocation.getFile(MapOutputLocation.java:185)
at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.copyOutput(ReduceTask.java:815)
at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.run(ReduceTask.java:764)
Caused by: java.io.IOException: No space left on device
at java.io.FileOutputStream.writeBytes(Native Method)
at java.io.FileOutputStream.write(FileOutputStream.java:260)
at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.write(RawLocalFileSystem.java:197)
... 11 more
2008-08-30 00:16:23,320 WARN org.apache.hadoop.mapred.TaskTracker: Error running child
java.io.IOException: task_200808291851_0001_r_000023_0The reduce copier failed
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:329)
at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2122)
The task should have died.
Priority: Blocker (was: Major)
Fix Version/s: 0.19.0
Assignee: Amareshwari Sriramadasu
Marking this as a blocker until we get to the root cause..
> If a reducer failed at shuffling stage, the task should fail, not just logging an exception
> -------------------------------------------------------------------------------------------
>
> Key: HADOOP-4163
> URL: https://issues.apache.org/jira/browse/HADOOP-4163
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.17.1
> Reporter: Runping Qi
> Assignee: Amareshwari Sriramadasu
> Priority: Blocker
> Fix For: 0.19.0
>
>
> I saw a reducer stuck at the shuffling stage, with the following exception logged in the log file:
> 2008-08-30 00:16:23,265 ERROR org.apache.hadoop.mapred.ReduceTask: Map output copy failure: org.apache.hadoop.fs.FSError: java.io.IOException: No space left on device
> at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.write(RawLocalFileSystem.java:199)
> at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
> at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
> at java.io.FilterOutputStream.close(FilterOutputStream.java:140)
> at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:59)
> at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:79)
> at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.close(ChecksumFileSystem.java:332)
> at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:59)
> at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:79)
> at org.apache.hadoop.mapred.MapOutputLocation.getFile(MapOutputLocation.java:185)
> at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.copyOutput(ReduceTask.java:815)
> at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.run(ReduceTask.java:764)
> Caused by: java.io.IOException: No space left on device
> at java.io.FileOutputStream.writeBytes(Native Method)
> at java.io.FileOutputStream.write(FileOutputStream.java:260)
> at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.write(RawLocalFileSystem.java:197)
> ... 11 more
> 2008-08-30 00:16:23,320 WARN org.apache.hadoop.mapred.TaskTracker: Error running child
> java.io.IOException: task_200808291851_0001_r_000023_0The reduce copier failed
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:329)
> at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2122)
> The task should have died.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.