You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by gziyang <gz...@gmail.com> on 2009/01/04 06:39:12 UTC

EOFException causes failure.

Hello,

I would like to run a mapreduce program on hadoop 0.19.0. The program is designed to iterate for 1000 times. However, in a certain iteration, the program fails due to an EOFException, but the jobs complete for all the previous iterations. Sometimes the program fails in the 5th iteration, while in other times, the program might fail until the 9th iteration. It seems a little random.

A similar situation has been discussed for hadoop version 0.2, but it is said the bug has been fixed in version 0.3. Anybody knows that what is wrong with my hadoop 0.19.0?

...
09/01/04 12:54:37 INFO mapred.JobClient:  map 60% reduce 0%
09/01/04 12:54:41 INFO mapred.JobClient:  map 66% reduce 0%
09/01/04 12:54:41 INFO mapred.JobClient: Task Id : attempt_200901041244_0007_m_000011_0, Status : FAILED
java.io.EOFException
        at java.io.DataInputStream.readFully(DataInputStream.java:180)
        at java.io.DataInputStream.readFully(DataInputStream.java:152)
        at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1450)
        at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1428)
        at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1417)
        at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1412)
        at org.apache.hadoop.mapred.SequenceFileRecordReader.<init>(SequenceFileRecordReader.java:43)
        at org.apache.hadoop.mapred.SequenceFileInputFormat.getRecordReader(SequenceFileInputFormat.java:58)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:321)
        at org.apache.hadoop.mapred.Child.main(Child.java:155)
09/01/04 12:54:43 INFO mapred.JobClient:  map 70% reduce 1%
09/01/04 12:54:45 INFO mapred.JobClient:  map 78% reduce 1%
09/01/04 12:54:46 INFO mapred.JobClient:  map 81% reduce 2%
09/01/04 12:54:48 INFO mapred.JobClient:  map 83% reduce 2%
09/01/04 12:54:48 INFO mapred.JobClient: Task Id : attempt_200901041244_0007_m_000011_1, Status : FAILED
java.io.EOFException
        at java.io.DataInputStream.readFully(DataInputStream.java:180)
        at java.io.DataInputStream.readFully(DataInputStream.java:152)
        at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1450)
        at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1428)
        at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1417)
        at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1412)
        at org.apache.hadoop.mapred.SequenceFileRecordReader.<init>(SequenceFileRecordReader.java:43)
        at org.apache.hadoop.mapred.SequenceFileInputFormat.getRecordReader(SequenceFileInputFormat.java:58)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:321)
        at org.apache.hadoop.mapred.Child.main(Child.java:155)
09/01/04 12:54:50 INFO mapred.JobClient:  map 83% reduce 3%
09/01/04 12:54:51 INFO mapred.JobClient:  map 86% reduce 4%
09/01/04 12:54:53 INFO mapred.JobClient:  map 86% reduce 6%
09/01/04 12:54:53 INFO mapred.JobClient: Task Id : attempt_200901041244_0007_m_000011_2, Status : FAILED
java.io.EOFException
        at java.io.DataInputStream.readFully(DataInputStream.java:180)
        at java.io.DataInputStream.readFully(DataInputStream.java:152)
        at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1450)
        at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1428)
        at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1417)
        at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1412)
        at org.apache.hadoop.mapred.SequenceFileRecordReader.<init>(SequenceFileRecordReader.java:43)
        at org.apache.hadoop.mapred.SequenceFileInputFormat.getRecordReader(SequenceFileInputFormat.java:58)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:321)
        at org.apache.hadoop.mapred.Child.main(Child.java:155)
09/01/04 12:54:55 INFO mapred.JobClient:  map 86% reduce 9%
09/01/04 12:54:56 INFO mapred.JobClient:  map 88% reduce 10%
09/01/04 12:54:58 INFO mapred.JobClient:  map 88% reduce 15%
java.io.IOException: Job failed!

Thanks a lot!

2009-01-04 



gziyang

Re: EOFException causes failure.

Posted by Aaron Kimball <aa...@cloudera.com>.
It seems to be having a problem deserializing data from a SequenceFile. It's
the same task failing over and over again -- the same slice of a particular
SequenceFile generated by the previous iteration. This suggests that maybe
you are generating incorrect SequenceFiles in the previous iteration.

Are you using custom implementations of Writable? Maybe your readFields()
and write() methods do not line up with one another correctly? Or that
there's a bug where you claim to have a variable-length entry of K bytes,
but are actually writing fewer than K?

Good luck,
- Aaron

On Sat, Jan 3, 2009 at 9:39 PM, gziyang <gz...@gmail.com> wrote:

> Hello,
>
> I would like to run a mapreduce program on hadoop 0.19.0. The program is
> designed to iterate for 1000 times. However, in a certain iteration, the
> program fails due to an EOFException, but the jobs complete for all the
> previous iterations. Sometimes the program fails in the 5th iteration, while
> in other times, the program might fail until the 9th iteration. It seems a
> little random.
>
> A similar situation has been discussed for hadoop version 0.2, but it is
> said the bug has been fixed in version 0.3. Anybody knows that what is wrong
> with my hadoop 0.19.0?
>
> ...
> 09/01/04 12:54:37 INFO mapred.JobClient:  map 60% reduce 0%
> 09/01/04 12:54:41 INFO mapred.JobClient:  map 66% reduce 0%
> 09/01/04 12:54:41 INFO mapred.JobClient: Task Id :
> attempt_200901041244_0007_m_000011_0, Status : FAILED
> java.io.EOFException
>        at java.io.DataInputStream.readFully(DataInputStream.java:180)
>        at java.io.DataInputStream.readFully(DataInputStream.java:152)
>        at
> org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1450)
>        at
> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1428)
>        at
> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1417)
>        at
> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1412)
>        at
> org.apache.hadoop.mapred.SequenceFileRecordReader.<init>(SequenceFileRecordReader.java:43)
>        at
> org.apache.hadoop.mapred.SequenceFileInputFormat.getRecordReader(SequenceFileInputFormat.java:58)
>        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:321)
>        at org.apache.hadoop.mapred.Child.main(Child.java:155)
> 09/01/04 12:54:43 INFO mapred.JobClient:  map 70% reduce 1%
> 09/01/04 12:54:45 INFO mapred.JobClient:  map 78% reduce 1%
> 09/01/04 12:54:46 INFO mapred.JobClient:  map 81% reduce 2%
> 09/01/04 12:54:48 INFO mapred.JobClient:  map 83% reduce 2%
> 09/01/04 12:54:48 INFO mapred.JobClient: Task Id :
> attempt_200901041244_0007_m_000011_1, Status : FAILED
> java.io.EOFException
>        at java.io.DataInputStream.readFully(DataInputStream.java:180)
>        at java.io.DataInputStream.readFully(DataInputStream.java:152)
>        at
> org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1450)
>        at
> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1428)
>        at
> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1417)
>        at
> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1412)
>        at
> org.apache.hadoop.mapred.SequenceFileRecordReader.<init>(SequenceFileRecordReader.java:43)
>        at
> org.apache.hadoop.mapred.SequenceFileInputFormat.getRecordReader(SequenceFileInputFormat.java:58)
>        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:321)
>        at org.apache.hadoop.mapred.Child.main(Child.java:155)
> 09/01/04 12:54:50 INFO mapred.JobClient:  map 83% reduce 3%
> 09/01/04 12:54:51 INFO mapred.JobClient:  map 86% reduce 4%
> 09/01/04 12:54:53 INFO mapred.JobClient:  map 86% reduce 6%
> 09/01/04 12:54:53 INFO mapred.JobClient: Task Id :
> attempt_200901041244_0007_m_000011_2, Status : FAILED
> java.io.EOFException
>        at java.io.DataInputStream.readFully(DataInputStream.java:180)
>        at java.io.DataInputStream.readFully(DataInputStream.java:152)
>        at
> org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1450)
>        at
> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1428)
>        at
> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1417)
>        at
> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1412)
>        at
> org.apache.hadoop.mapred.SequenceFileRecordReader.<init>(SequenceFileRecordReader.java:43)
>        at
> org.apache.hadoop.mapred.SequenceFileInputFormat.getRecordReader(SequenceFileInputFormat.java:58)
>        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:321)
>        at org.apache.hadoop.mapred.Child.main(Child.java:155)
> 09/01/04 12:54:55 INFO mapred.JobClient:  map 86% reduce 9%
> 09/01/04 12:54:56 INFO mapred.JobClient:  map 88% reduce 10%
> 09/01/04 12:54:58 INFO mapred.JobClient:  map 88% reduce 15%
> java.io.IOException: Job failed!
>
> Thanks a lot!
>
> 2009-01-04
>
>
>
> gziyang
>