You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Arun C Murthy <ac...@yahoo-inc.com> on 2008/09/11 20:08:53 UTC
Re: Reduce task failed: org.apache.hadoop.fs.FSError: java.io.IOException
On Sep 11, 2008, at 9:10 AM, pvvpr wrote:
> Hello,
> Never came across this error before. Upgraded to 0.18.0 this
> morning and
> ran a nutch fetch job. Got this exception in both the reduce
> attempts of
> a task and they failed. All other reducers seemed to work fine, except
> for one task.
>
> Any ideas what could be the problem?
>
Looks like data on disk got corrupted ( org.apache.hadoop.fs.FSError:
java.io.IOException: Input/output error); could you check the disks?
Arun
> - Prasad Pingali.
> IIIT, Hyderabad.
>
> 2008-09-11 06:31:19,837 INFO org.apache.hadoop.mapred.ReduceTask:
> attempt_200809101353_0021_r_000004_0: Got 1 new map-outputs & number
> of
> known map outputs is 1
> 2008-09-11 06:31:19,837 INFO org.apache.hadoop.mapred.ReduceTask:
> attempt_200809101353_0021_r_000004_0 Scheduled 1 of 1 known outputs (0
> slow hosts and 0 dup hosts)
> 2008-09-11 06:31:20,133 INFO org.apache.hadoop.mapred.ReduceTask:
> Shuffling 8230686 bytes (8230686 raw bytes) into RAM from
> attempt_200809101353_0021_m_000095_0
> 2008-09-11 06:31:22,332 INFO org.apache.hadoop.mapred.ReduceTask: Read
> 8230686 bytes from map-output for attempt_200809101353_0021_m_000095_0
> 2008-09-11 06:31:22,333 INFO org.apache.hadoop.mapred.ReduceTask:
> Rec #1
> from attempt_200809101353_0021_m_000095_0 -> (33, 134) from machine10
> 2008-09-11 06:31:28,837 INFO org.apache.hadoop.mapred.ReduceTask:
> attempt_200809101353_0021_r_000004_0: Got 1 new map-outputs & number
> of
> known map outputs is 1
> 2008-09-11 06:31:28,838 INFO org.apache.hadoop.mapred.ReduceTask:
> attempt_200809101353_0021_r_000004_0 Scheduled 1 of 1 known outputs (0
> slow hosts and 0 dup hosts)
> 2008-09-11 06:31:29,585 INFO org.apache.hadoop.mapred.ReduceTask:
> Shuffling 21454877 bytes (21454877 raw bytes) into Local-FS from
> attempt_200809101353_0021_m_000074_0
> 2008-09-11 06:31:37,831 INFO org.apache.hadoop.mapred.ReduceTask: Read
> 21454877 bytes from map-output for
> attempt_200809101353_0021_m_000074_0
> 2008-09-11 06:31:37,832 INFO org.apache.hadoop.mapred.ReduceTask:
> attempt_200809101353_0021_r_000004_0We have 19 map outputs on disk.
> Triggering merge of 10 files
> 2008-09-11 06:31:38,033 INFO org.apache.hadoop.mapred.Merger:
> Merging 10
> sorted segments
> 2008-09-11 06:31:43,838 INFO org.apache.hadoop.mapred.ReduceTask:
> attempt_200809101353_0021_r_000004_0 Need another 9 map output(s)
> where 0
> is already in progress
> 2008-09-11 06:31:43,838 INFO org.apache.hadoop.mapred.ReduceTask:
> attempt_200809101353_0021_r_000004_0: Got 0 new map-outputs & number
> of
> known map outputs is 0
> 2008-09-11 06:31:43,838 INFO org.apache.hadoop.mapred.ReduceTask:
> attempt_200809101353_0021_r_000004_0 Scheduled 0 of 0 known outputs (0
> slow hosts and 0 dup hosts)
> 2008-09-11 06:32:03,095 INFO org.apache.hadoop.mapred.Merger: Down
> to the
> last merge-pass, with 10 segments left of total size: 262561859 bytes
> 2008-09-11 06:32:51,044 WARN org.apache.hadoop.mapred.ReduceTask:
> attempt_200809101353_0021_r_000004_0 Merging of the local FS files
> threw
> an exception: org.apache.hadoop.fs.FSError: java.io.IOException:
> Input/output error
> at
> org.apache.hadoop.fs.RawLocalFileSystem
> $LocalFSFileInputStream.read(RawLocalFileSystem.java:149)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:
> 256)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:
> 317)
> at java.io.DataInputStream.read(DataInputStream.java:132)
> at
> org.apache.hadoop.fs.FSInputChecker.readFully(FSInputChecker.java:380)
> at
> org.apache.hadoop.fs.ChecksumFileSystem
> $ChecksumFSInputChecker.readChunk(ChecksumFileSystem.java:208)
> at
> org
> .apache
> .hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.java:236)
> at
> org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:191)
> at
> org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:159)
> at java.io.DataInputStream.read(DataInputStream.java:132)
> at org.apache.hadoop.mapred.IFile$Reader.readData(IFile.java:
> 263)
> at org.apache.hadoop.mapred.IFile$Reader.rejigData(IFile.java:
> 293)
> at
> org.apache.hadoop.mapred.IFile$Reader.readNextBlock(IFile.java:277)
> at org.apache.hadoop.mapred.IFile$Reader.next(IFile.java:339)
> at org.apache.hadoop.mapred.Merger$Segment.next(Merger.java:
> 134)
> at
> org.apache.hadoop.mapred.Merger
> $MergeQueue.adjustPriorityQueue(Merger.java:225)
> at org.apache.hadoop.mapred.Merger
> $MergeQueue.next(Merger.java:242)
> at org.apache.hadoop.mapred.Merger.writeFile(Merger.java:83)
> at
> org.apache.hadoop.mapred.ReduceTask$ReduceCopier
> $LocalFSMerger.run(ReduceTask.java:2021)
> Caused by: java.io.IOException: Input/output error
> at java.io.FileInputStream.readBytes(Native Method)
> at java.io.FileInputStream.read(FileInputStream.java:199)
> at
> org.apache.hadoop.fs.RawLocalFileSystem
> $TrackingFileInputStream.read(RawLocalFileSystem.java:90)
> at
> org.apache.hadoop.fs.RawLocalFileSystem
> $LocalFSFileInputStream.read(RawLocalFileSystem.java:143)
>
>
Re: Reduce task failed: org.apache.hadoop.fs.FSError: java.io.IOException
Posted by Prasad Pingali <pv...@research.iiit.ac.in>.
Thanks Arun,
Yeah I think it was a disk problem. Redoing the task went through fine.
Since disk corruptions may be common, doesn't hadoop pick a replicated block?
- Prasad.
On Thursday 11 September 2008 11:38:53 pm Arun C Murthy wrote:
> On Sep 11, 2008, at 9:10 AM, pvvpr wrote:
> > Hello,
> > Never came across this error before. Upgraded to 0.18.0 this
> > morning and
> > ran a nutch fetch job. Got this exception in both the reduce
> > attempts of
> > a task and they failed. All other reducers seemed to work fine, except
> > for one task.
> >
> > Any ideas what could be the problem?
>
> Looks like data on disk got corrupted ( org.apache.hadoop.fs.FSError:
> java.io.IOException: Input/output error); could you check the disks?
>
> Arun
>
> > - Prasad Pingali.
> > IIIT, Hyderabad.
> >
> > 2008-09-11 06:31:19,837 INFO org.apache.hadoop.mapred.ReduceTask:
> > attempt_200809101353_0021_r_000004_0: Got 1 new map-outputs & number
> > of
> > known map outputs is 1
> > 2008-09-11 06:31:19,837 INFO org.apache.hadoop.mapred.ReduceTask:
> > attempt_200809101353_0021_r_000004_0 Scheduled 1 of 1 known outputs (0
> > slow hosts and 0 dup hosts)
> > 2008-09-11 06:31:20,133 INFO org.apache.hadoop.mapred.ReduceTask:
> > Shuffling 8230686 bytes (8230686 raw bytes) into RAM from
> > attempt_200809101353_0021_m_000095_0
> > 2008-09-11 06:31:22,332 INFO org.apache.hadoop.mapred.ReduceTask: Read
> > 8230686 bytes from map-output for attempt_200809101353_0021_m_000095_0
> > 2008-09-11 06:31:22,333 INFO org.apache.hadoop.mapred.ReduceTask:
> > Rec #1
> > from attempt_200809101353_0021_m_000095_0 -> (33, 134) from machine10
> > 2008-09-11 06:31:28,837 INFO org.apache.hadoop.mapred.ReduceTask:
> > attempt_200809101353_0021_r_000004_0: Got 1 new map-outputs & number
> > of
> > known map outputs is 1
> > 2008-09-11 06:31:28,838 INFO org.apache.hadoop.mapred.ReduceTask:
> > attempt_200809101353_0021_r_000004_0 Scheduled 1 of 1 known outputs (0
> > slow hosts and 0 dup hosts)
> > 2008-09-11 06:31:29,585 INFO org.apache.hadoop.mapred.ReduceTask:
> > Shuffling 21454877 bytes (21454877 raw bytes) into Local-FS from
> > attempt_200809101353_0021_m_000074_0
> > 2008-09-11 06:31:37,831 INFO org.apache.hadoop.mapred.ReduceTask: Read
> > 21454877 bytes from map-output for
> > attempt_200809101353_0021_m_000074_0
> > 2008-09-11 06:31:37,832 INFO org.apache.hadoop.mapred.ReduceTask:
> > attempt_200809101353_0021_r_000004_0We have 19 map outputs on disk.
> > Triggering merge of 10 files
> > 2008-09-11 06:31:38,033 INFO org.apache.hadoop.mapred.Merger:
> > Merging 10
> > sorted segments
> > 2008-09-11 06:31:43,838 INFO org.apache.hadoop.mapred.ReduceTask:
> > attempt_200809101353_0021_r_000004_0 Need another 9 map output(s)
> > where 0
> > is already in progress
> > 2008-09-11 06:31:43,838 INFO org.apache.hadoop.mapred.ReduceTask:
> > attempt_200809101353_0021_r_000004_0: Got 0 new map-outputs & number
> > of
> > known map outputs is 0
> > 2008-09-11 06:31:43,838 INFO org.apache.hadoop.mapred.ReduceTask:
> > attempt_200809101353_0021_r_000004_0 Scheduled 0 of 0 known outputs (0
> > slow hosts and 0 dup hosts)
> > 2008-09-11 06:32:03,095 INFO org.apache.hadoop.mapred.Merger: Down
> > to the
> > last merge-pass, with 10 segments left of total size: 262561859 bytes
> > 2008-09-11 06:32:51,044 WARN org.apache.hadoop.mapred.ReduceTask:
> > attempt_200809101353_0021_r_000004_0 Merging of the local FS files
> > threw
> > an exception: org.apache.hadoop.fs.FSError: java.io.IOException:
> > Input/output error
> > at
> > org.apache.hadoop.fs.RawLocalFileSystem
> > $LocalFSFileInputStream.read(RawLocalFileSystem.java:149)
> > at java.io.BufferedInputStream.read1(BufferedInputStream.java:
> > 256)
> > at java.io.BufferedInputStream.read(BufferedInputStream.java:
> > 317)
> > at java.io.DataInputStream.read(DataInputStream.java:132)
> > at
> > org.apache.hadoop.fs.FSInputChecker.readFully(FSInputChecker.java:380)
> > at
> > org.apache.hadoop.fs.ChecksumFileSystem
> > $ChecksumFSInputChecker.readChunk(ChecksumFileSystem.java:208)
> > at
> > org
> > .apache
> > .hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.java:236)
> > at
> > org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:191)
> > at
> > org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:159)
> > at java.io.DataInputStream.read(DataInputStream.java:132)
> > at org.apache.hadoop.mapred.IFile$Reader.readData(IFile.java:
> > 263)
> > at org.apache.hadoop.mapred.IFile$Reader.rejigData(IFile.java:
> > 293)
> > at
> > org.apache.hadoop.mapred.IFile$Reader.readNextBlock(IFile.java:277)
> > at org.apache.hadoop.mapred.IFile$Reader.next(IFile.java:339)
> > at org.apache.hadoop.mapred.Merger$Segment.next(Merger.java:
> > 134)
> > at
> > org.apache.hadoop.mapred.Merger
> > $MergeQueue.adjustPriorityQueue(Merger.java:225)
> > at org.apache.hadoop.mapred.Merger
> > $MergeQueue.next(Merger.java:242)
> > at org.apache.hadoop.mapred.Merger.writeFile(Merger.java:83)
> > at
> > org.apache.hadoop.mapred.ReduceTask$ReduceCopier
> > $LocalFSMerger.run(ReduceTask.java:2021)
> > Caused by: java.io.IOException: Input/output error
> > at java.io.FileInputStream.readBytes(Native Method)
> > at java.io.FileInputStream.read(FileInputStream.java:199)
> > at
> > org.apache.hadoop.fs.RawLocalFileSystem
> > $TrackingFileInputStream.read(RawLocalFileSystem.java:90)
> > at
> > org.apache.hadoop.fs.RawLocalFileSystem
> > $LocalFSFileInputStream.read(RawLocalFileSystem.java:143)