You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Oleg Ruchovets <or...@gmail.com> on 2012/07/24 10:19:56 UTC

Unexpected end of input stream (GZ)

Hi , I got such exception running hadoop job:

java.io.EOFException: Unexpected end of input stream at
org.apache.hadoop.io.compress.DecompressorStream.getCompressedData(DecompressorStream.java:99)
at
org.apache.hadoop.io.compress.DecompressorStream.decompress(DecompressorStream.java:87)
at
org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:75)
at java.io.InputStream.read(InputStream.java:85) at
org.apache.hadoop.util.LineReader.readDefaultLine(LineReader.java:205) at
org.apache.hadoop.util.LineReader.readLine(LineReader.java:169) at
org.apache.hadoop.mapreduce.lib.input.LineRecordReader.nextKeyValue(LineRecordReader.java:114)
at
org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:456)
at org.apache.hadoop.mapreduce.MapContext.nextKeyValue(MapContext.java:67)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143) at
org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:647) at
org.apache.hadoop.mapred.MapTask.run(MapTask.java:323) at
org.apache.hadoop.mapred.Child$4.run(Child.

As I understood some of my files are corrupted ( I am working with GZ
format).

I resolve the issue using conf.set("mapred.max.map.failures.percent" , "1"),

But I don't know what file cause the problem.

Question:
 How can I get a filename which is corrupted.

Thanks in advance
Oleg.

Re: Unexpected end of input stream (GZ)

Posted by Bejoy Ks <be...@gmail.com>.
Hi Oleg

>From the job tracker page, you can get to the failed tasks and see
which was the file split processed by that task. The split information
is available under the status column for each task.

The file split information is not available on job history.


Regrads
Bejoy KS

On Tue, Jul 24, 2012 at 1:49 PM, Oleg Ruchovets <or...@gmail.com> wrote:
> Hi , I got such exception running hadoop job:
>
> java.io.EOFException: Unexpected end of input stream at
> org.apache.hadoop.io.compress.DecompressorStream.getCompressedData(DecompressorStream.java:99)
> at
> org.apache.hadoop.io.compress.DecompressorStream.decompress(DecompressorStream.java:87)
> at
> org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:75)
> at java.io.InputStream.read(InputStream.java:85) at
> org.apache.hadoop.util.LineReader.readDefaultLine(LineReader.java:205) at
> org.apache.hadoop.util.LineReader.readLine(LineReader.java:169) at
> org.apache.hadoop.mapreduce.lib.input.LineRecordReader.nextKeyValue(LineRecordReader.java:114)
> at
> org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:456)
> at org.apache.hadoop.mapreduce.MapContext.nextKeyValue(MapContext.java:67)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143) at
> org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:647) at
> org.apache.hadoop.mapred.MapTask.run(MapTask.java:323) at
> org.apache.hadoop.mapred.Child$4.run(Child.
>
> As I understood some of my files are corrupted ( I am working with GZ
> format).
>
> I resolve the issue using conf.set("mapred.max.map.failures.percent" , "1"),
>
> But I don't know what file cause the problem.
>
> Question:
>  How can I get a filename which is corrupted.
>
> Thanks in advance
> Oleg.