You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nutch.apache.org by Sebastian Nagel <wa...@googlemail.com> on 2013/09/17 15:39:24 UTC

IOException reading segments with current trunk

Hi,

recently I got some IO exceptions when reading older segments
with recent trunk builds. Did anyone make similar observations?

According to the stack it seems possible that NUTCH-1622
causes segments' parse_data to be incompatible between versions?

Thanks,
Sebastian

java.io.IOException: IO error in map input file
file:.../segments/20130623204752/parse_data/part-00000/data
...
Caused by: java.io.EOFException
        at java.io.DataInputStream.readFully(DataInputStream.java:197)
        at java.io.DataInputStream.readUTF(DataInputStream.java:609)
        at java.io.DataInputStream.readUTF(DataInputStream.java:564)
        at org.apache.hadoop.io.AbstractMapWritable.readFields(AbstractMapWritable.java:199)
        at org.apache.hadoop.io.MapWritable.readFields(MapWritable.java:146)
        at org.apache.nutch.parse.Outlink.readFields(Outlink.java:54)
        at org.apache.nutch.parse.Outlink.read(Outlink.java:84)
        at org.apache.nutch.parse.ParseData.readFields(ParseData.java:133)
        at
org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:67)
        at
org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:40)

Re: IOException reading segments with current trunk

Posted by Sebastian Nagel <wa...@googlemail.com>.
Thanks, good to know.

We should add a warning to release notes and CHANGES.txt.
Ideally, of course, reading segments should be backward compatible.

Sebastian


2013/9/17 Markus Jelsma <ma...@openindex.io>

> Yes, we've got trouble with it too, similar exception, but we're did not
> sync our nutch with that issue IIRC.
>
> Will check further this week.
>
> -----Original message-----
> > From:Sebastian Nagel <wa...@googlemail.com>
> > Sent: Tuesday 17th September 2013 15:40
> > To: dev@nutch.apache.org
> > Subject: IOException reading segments with current trunk
> >
> > Hi,
> >
> > recently I got some IO exceptions when reading older segments
> > with recent trunk builds. Did anyone make similar observations?
> >
> > According to the stack it seems possible that NUTCH-1622
> > causes segments' parse_data to be incompatible between versions?
> >
> > Thanks,
> > Sebastian
> >
> > java.io.IOException: IO error in map input file
> > file:.../segments/20130623204752/parse_data/part-00000/data
> > ...
> > Caused by: java.io.EOFException
> >         at java.io.DataInputStream.readFully(DataInputStream.java:197)
> >         at java.io.DataInputStream.readUTF(DataInputStream.java:609)
> >         at java.io.DataInputStream.readUTF(DataInputStream.java:564)
> >         at
> org.apache.hadoop.io.AbstractMapWritable.readFields(AbstractMapWritable.java:199)
> >         at
> org.apache.hadoop.io.MapWritable.readFields(MapWritable.java:146)
> >         at org.apache.nutch.parse.Outlink.readFields(Outlink.java:54)
> >         at org.apache.nutch.parse.Outlink.read(Outlink.java:84)
> >         at
> org.apache.nutch.parse.ParseData.readFields(ParseData.java:133)
> >         at
> >
> org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:67)
> >         at
> >
> org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:40)
> >
>

RE: IOException reading segments with current trunk

Posted by Markus Jelsma <ma...@openindex.io>.
Yes, we've got trouble with it too, similar exception, but we're did not sync our nutch with that issue IIRC. 

Will check further this week.
 
-----Original message-----
> From:Sebastian Nagel <wa...@googlemail.com>
> Sent: Tuesday 17th September 2013 15:40
> To: dev@nutch.apache.org
> Subject: IOException reading segments with current trunk
> 
> Hi,
> 
> recently I got some IO exceptions when reading older segments
> with recent trunk builds. Did anyone make similar observations?
> 
> According to the stack it seems possible that NUTCH-1622
> causes segments' parse_data to be incompatible between versions?
> 
> Thanks,
> Sebastian
> 
> java.io.IOException: IO error in map input file
> file:.../segments/20130623204752/parse_data/part-00000/data
> ...
> Caused by: java.io.EOFException
>         at java.io.DataInputStream.readFully(DataInputStream.java:197)
>         at java.io.DataInputStream.readUTF(DataInputStream.java:609)
>         at java.io.DataInputStream.readUTF(DataInputStream.java:564)
>         at org.apache.hadoop.io.AbstractMapWritable.readFields(AbstractMapWritable.java:199)
>         at org.apache.hadoop.io.MapWritable.readFields(MapWritable.java:146)
>         at org.apache.nutch.parse.Outlink.readFields(Outlink.java:54)
>         at org.apache.nutch.parse.Outlink.read(Outlink.java:84)
>         at org.apache.nutch.parse.ParseData.readFields(ParseData.java:133)
>         at
> org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:67)
>         at
> org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:40)
>