You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by Patricio Galeas <pg...@yahoo.de> on 2011/10/29 00:09:37 UTC
error by merging segments
Hello,
when I try to merge segment using ...
bin/nutch mergesegs /user/nutch/crawl_al/MERGEDsegments -dir /user/nutch/my_crawl/segments -filter -slice 50000
.... I get the following error. What I'm doing wrong?
Thanks
Patricio
java.io.EOFException
at java.io.DataInputStream.readByte(DataInputStream.java:250)
at org.apache.hadoop.io.WritableUtils.readVLong(WritableUtils.java:298)
at org.apache.hadoop.io.WritableUtils.readVInt(WritableUtils.java:319)
at org.apache.hadoop.io.Text.readString(Text.java:400)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.createBlockOutputStream(DFSClient.java:2901)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2826)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2102)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2288)
java.io.EOFException
at java.io.DataInputStream.readByte(DataInputStream.java:250)
at org.apache.hadoop.io.WritableUtils.readVLong(WritableUtils.java:298)
at org.apache.hadoop.io.WritableUtils.readVInt(WritableUtils.java:319)
at org.apache.hadoop.io.Text.readString(Text.java:400)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.createBlockOutputStream(DFSClient.java:2901)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2826)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2102)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2288)
java.io.EOFException
at java.io.DataInputStream.readByte(DataInputStream.java:250)
at org.apache.hadoop.io.WritableUtils.readVLong(WritableUtils.java:298)
at org.apache.hadoop.io.WritableUtils.readVInt(WritableUtils.java:319)
at org.apache.hadoop.io.Text.readString(Text.java:400)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.createBlockOutputStream(DFSClient.java:2901)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2826)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2102)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2288)
attempt_201110152026_0003_r_000001_1: log4j:WARN No appenders could be found for logger (org.apache.hadoop.hdfs.DFSClient).
attempt_201110152026_0003_r_000001_1: log4j:WARN Please initialize the log4j system properly.
java.io.EOFException
at java.io.DataInputStream.readByte(DataInputStream.java:250)
at org.apache.hadoop.io.WritableUtils.readVLong(WritableUtils.java:298)
at org.apache.hadoop.io.WritableUtils.readVInt(WritableUtils.java:319)
at org.apache.hadoop.io.Text.readString(Text.java:400)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.createBlockOutputStream(DFSClient.java:2901)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2826)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2102)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2288)
attempt_201110152026_0003_r_000000_1: log4j:WARN No appenders could be found for logger (org.apache.hadoop.hdfs.DFSClient).
attempt_201110152026_0003_r_000000_1: log4j:WARN Please initialize the log4j system properly.
java.io.EOFException
at java.io.DataInputStream.readByte(DataInputStream.java:250)
at org.apache.hadoop.io.WritableUtils.readVLong(WritableUtils.java:298)
at org.apache.hadoop.io.WritableUtils.readVInt(WritableUtils.java:319)
at org.apache.hadoop.io.Text.readString(Text.java:400)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.createBlockOutputStream(DFSClient.java:2901)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2826)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2102)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2288)
java.io.EOFException
at java.io.DataInputStream.readByte(DataInputStream.java:250)
at org.apache.hadoop.io.WritableUtils.readVLong(WritableUtils.java:298)
at org.apache.hadoop.io.WritableUtils.readVInt(WritableUtils.java:319)
at org.apache.hadoop.io.Text.readString(Text.java:400)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.createBlockOutputStream(DFSClient.java:2901)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2826)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2102)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2288)
Exception in thread "main" java.io.IOException: Job failed!
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1252)
at org.apache.nutch.segment.SegmentMerger.merge(SegmentMerger.java:638)
at org.apache.nutch.segment.SegmentMerger.main(SegmentMerger.java:683
Re: error by merging segments
Posted by lewis john mcgibbney <le...@gmail.com>.
Can you state which version of Nutch your using please. Also what kind of
setup are you using?
Was everything else working OK up until the stage when you attempted to
mergesegs? It appears from the log output that we are not specifying the
correct input paths. Can you try merging just two directories into one
larger directory. Also please try without -filter and -slice arguements.
On Mon, Oct 31, 2011 at 5:05 PM, Patricio Galeas <pg...@yahoo.de> wrote:
> Hello Lewis,
>
> thanks for the response, but in both cases i got an error.
>
> FIRST CASE:
> ERROR: No input segments.
>
> SECOND CASE:
> Exception in thread "main" java.io.IOException: No input paths specified
> in job
> at
> org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:152)
> at
> org.apache.hadoop.mapred.SequenceFileInputFormat.listStatus(SequenceFileInputFormat.java:44)
> at
> org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:201)
> at org.apache.hadoop.mapred.JobClient.writeOldSplits(JobClient.java:810)
> at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:781)
> at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:730)
> at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1249)
> at org.apache.nutch.segment.SegmentMerger.merge(SegmentMerger.java:638)
> at org.apache.nutch.segment.SegmentMerger.main(SegmentMerger.java:683)
>
> Regards,
> Patricio
>
> ------------------------------
> *Von:* lewis john mcgibbney <le...@gmail.com>
> *An:* user@nutch.apache.org; Patricio Galeas <pg...@yahoo.de>
> *Cc:* nutch-user <nu...@lucene.apache.org>
> *Gesendet:* 12:10 Montag, 31.Oktober 2011
> *Betreff:* Re: error by merging segments
>
> Hi Patricio,
>
> Try this
>
> bin/nutch mergesegs /user/nutch/crawl_al/MERGEDsegments -dir
> /user/nutch/my_crawl/segments/* -filter -slice 50000
>
> or
>
> bin/nutch mergesegs /user/nutch/crawl_al/MERGEDsegments -dir
> /user/nutch/my_crawl/segments/seg1 /user/nutch/my_crawl/segments/seg2
> /user/nutch/my_crawl/segments/seg3, etc -filter -slice 50000
>
> If this works then we need to edit the wiki to accomodate the '/*' which
> is required to refer to ALL segments in any given directory.
>
> HTH
>
> On Fri, Oct 28, 2011 at 11:09 PM, Patricio Galeas <pg...@yahoo.de>wrote:
>
> Hello,
>
> when I try to merge segment using ...
>
> bin/nutch mergesegs /user/nutch/crawl_al/MERGEDsegments -dir
> /user/nutch/my_crawl/segments -filter -slice 50000
>
>
> .... I get the following error. What I'm doing wrong?
>
> Thanks
> Patricio
>
> java.io.EOFException
> at java.io.DataInputStream.readByte(DataInputStream.java:250)
> at org.apache.hadoop.io.WritableUtils.readVLong(WritableUtils.java:298)
> at org.apache.hadoop.io.WritableUtils.readVInt(WritableUtils.java:319)
> at org.apache.hadoop.io.Text.readString(Text.java:400)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.createBlockOutputStream(DFSClient.java:2901)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2826)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2102)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2288)
>
> java.io.EOFException
> at java.io.DataInputStream.readByte(DataInputStream.java:250)
> at org.apache.hadoop.io.WritableUtils.readVLong(WritableUtils.java:298)
> at org.apache.hadoop.io.WritableUtils.readVInt(WritableUtils.java:319)
> at org.apache.hadoop.io.Text.readString(Text.java:400)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.createBlockOutputStream(DFSClient.java:2901)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2826)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2102)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2288)
>
> java.io.EOFException
> at java.io.DataInputStream.readByte(DataInputStream.java:250)
> at org.apache.hadoop.io.WritableUtils.readVLong(WritableUtils.java:298)
> at org.apache.hadoop.io.WritableUtils.readVInt(WritableUtils.java:319)
> at org.apache.hadoop.io.Text.readString(Text.java:400)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.createBlockOutputStream(DFSClient.java:2901)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2826)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2102)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2288)
>
> attempt_201110152026_0003_r_000001_1: log4j:WARN No appenders could be
> found for logger (org.apache.hadoop.hdfs.DFSClient).
> attempt_201110152026_0003_r_000001_1: log4j:WARN Please initialize the
> log4j system properly.
> java.io.EOFException
> at java.io.DataInputStream.readByte(DataInputStream.java:250)
> at org.apache.hadoop.io.WritableUtils.readVLong(WritableUtils.java:298)
> at org.apache.hadoop.io.WritableUtils.readVInt(WritableUtils.java:319)
> at org.apache.hadoop.io.Text.readString(Text.java:400)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.createBlockOutputStream(DFSClient.java:2901)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2826)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2102)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2288)
>
> attempt_201110152026_0003_r_000000_1: log4j:WARN No appenders could be
> found for logger (org.apache.hadoop.hdfs.DFSClient).
> attempt_201110152026_0003_r_000000_1: log4j:WARN Please initialize the
> log4j system properly.
> java.io.EOFException
> at java.io.DataInputStream.readByte(DataInputStream.java:250)
> at org.apache.hadoop.io.WritableUtils.readVLong(WritableUtils.java:298)
> at org.apache.hadoop.io.WritableUtils.readVInt(WritableUtils.java:319)
> at org.apache.hadoop.io.Text.readString(Text.java:400)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.createBlockOutputStream(DFSClient.java:2901)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2826)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2102)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2288)
>
> java.io.EOFException
> at java.io.DataInputStream.readByte(DataInputStream.java:250)
> at org.apache.hadoop.io.WritableUtils.readVLong(WritableUtils.java:298)
> at org.apache.hadoop.io.WritableUtils.readVInt(WritableUtils.java:319)
> at org.apache.hadoop.io.Text.readString(Text.java:400)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.createBlockOutputStream(DFSClient.java:2901)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2826)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2102)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2288)
>
> Exception in thread "main" java.io.IOException: Job failed!
> at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1252)
> at org.apache.nutch.segment.SegmentMerger.merge(SegmentMerger.java:638)
> at org.apache.nutch.segment.SegmentMerger.main(SegmentMerger.java:683
>
>
>
>
> --
> *Lewis*
>
>
>
>
--
*Lewis*
Re: error by merging segments
Posted by Patricio Galeas <pg...@yahoo.de>.
Hello Lewis,
thanks for the response, but in both cases i got an error.
FIRST CASE:
ERROR: No input segments.
SECOND CASE:
Exception in thread "main" java.io.IOException: No input paths specified in job
at org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:152)
at org.apache.hadoop.mapred.SequenceFileInputFormat.listStatus(SequenceFileInputFormat.java:44)
at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:201)
at org.apache.hadoop.mapred.JobClient.writeOldSplits(JobClient.java:810)
at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:781)
at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:730)
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1249)
at org.apache.nutch.segment.SegmentMerger.merge(SegmentMerger.java:638)
at org.apache.nutch.segment.SegmentMerger.main(SegmentMerger.java:683)
Regards,
Patricio
________________________________
Von: lewis john mcgibbney <le...@gmail.com>
An: user@nutch.apache.org; Patricio Galeas <pg...@yahoo.de>
Cc: nutch-user <nu...@lucene.apache.org>
Gesendet: 12:10 Montag, 31.Oktober 2011
Betreff: Re: error by merging segments
Hi Patricio,
Try this
bin/nutch mergesegs /user/nutch/crawl_al/MERGEDsegments -dir /user/nutch/my_crawl/segments/* -filter -slice 50000
or
bin/nutch mergesegs /user/nutch/crawl_al/MERGEDsegments -dir /user/nutch/my_crawl/segments/seg1 /user/nutch/my_crawl/segments/seg2 /user/nutch/my_crawl/segments/seg3, etc -filter -slice 50000
If this works then we need to edit the wiki to accomodate the '/*' which is required to refer to ALL segments in any given directory.
HTH
On Fri, Oct 28, 2011 at 11:09 PM, Patricio Galeas <pg...@yahoo.de> wrote:
Hello,
>
>when I try to merge segment using ...
>
>bin/nutch mergesegs /user/nutch/crawl_al/MERGEDsegments -dir /user/nutch/my_crawl/segments -filter -slice 50000
>
>
>.... I get the following error. What I'm doing wrong?
>
>Thanks
>Patricio
>
>java.io.EOFException
>at java.io.DataInputStream.readByte(DataInputStream.java:250)
>at org.apache.hadoop.io.WritableUtils.readVLong(WritableUtils.java:298)
>at org.apache.hadoop.io.WritableUtils.readVInt(WritableUtils.java:319)
>at org.apache.hadoop.io.Text.readString(Text.java:400)
>at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.createBlockOutputStream(DFSClient.java:2901)
>at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2826)
>at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2102)
>at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2288)
>
>java.io.EOFException
>at java.io.DataInputStream.readByte(DataInputStream.java:250)
>at org.apache.hadoop.io.WritableUtils.readVLong(WritableUtils.java:298)
>at org.apache.hadoop.io.WritableUtils.readVInt(WritableUtils.java:319)
>at org.apache.hadoop.io.Text.readString(Text.java:400)
>at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.createBlockOutputStream(DFSClient.java:2901)
>at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2826)
>at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2102)
>at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2288)
>
>java.io.EOFException
>at java.io.DataInputStream.readByte(DataInputStream.java:250)
>at org.apache.hadoop.io.WritableUtils.readVLong(WritableUtils.java:298)
>at org.apache.hadoop.io.WritableUtils.readVInt(WritableUtils.java:319)
>at org.apache.hadoop.io.Text.readString(Text.java:400)
>at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.createBlockOutputStream(DFSClient.java:2901)
>at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2826)
>at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2102)
>at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2288)
>
>attempt_201110152026_0003_r_000001_1: log4j:WARN No appenders could be found for logger (org.apache.hadoop.hdfs.DFSClient).
>attempt_201110152026_0003_r_000001_1: log4j:WARN Please initialize the log4j system properly.
>java.io.EOFException
>at java.io.DataInputStream.readByte(DataInputStream.java:250)
>at org.apache.hadoop.io.WritableUtils.readVLong(WritableUtils.java:298)
>at org.apache.hadoop.io.WritableUtils.readVInt(WritableUtils.java:319)
>at org.apache.hadoop.io.Text.readString(Text.java:400)
>at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.createBlockOutputStream(DFSClient.java:2901)
>at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2826)
>at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2102)
>at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2288)
>
>attempt_201110152026_0003_r_000000_1: log4j:WARN No appenders could be found for logger (org.apache.hadoop.hdfs.DFSClient).
>attempt_201110152026_0003_r_000000_1: log4j:WARN Please initialize the log4j system properly.
>java.io.EOFException
>at java.io.DataInputStream.readByte(DataInputStream.java:250)
>at org.apache.hadoop.io.WritableUtils.readVLong(WritableUtils.java:298)
>at org.apache.hadoop.io.WritableUtils.readVInt(WritableUtils.java:319)
>at org.apache.hadoop.io.Text.readString(Text.java:400)
>at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.createBlockOutputStream(DFSClient.java:2901)
>at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2826)
>at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2102)
>at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2288)
>
>java.io.EOFException
>at java.io.DataInputStream.readByte(DataInputStream.java:250)
>at org.apache.hadoop.io.WritableUtils.readVLong(WritableUtils.java:298)
>at org.apache.hadoop.io.WritableUtils.readVInt(WritableUtils.java:319)
>at org.apache.hadoop.io.Text.readString(Text.java:400)
>at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.createBlockOutputStream(DFSClient.java:2901)
>at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2826)
>at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2102)
>at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2288)
>
>Exception in thread "main" java.io.IOException: Job failed!
>at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1252)
>at org.apache.nutch.segment.SegmentMerger.merge(SegmentMerger.java:638)
>at org.apache.nutch.segment.SegmentMerger.main(SegmentMerger.java:683
>
--
Lewis
Re: error by merging segments
Posted by lewis john mcgibbney <le...@gmail.com>.
Hi Patricio,
Try this
bin/nutch mergesegs /user/nutch/crawl_al/MERGEDsegments -dir
/user/nutch/my_crawl/segments/* -filter -slice 50000
or
bin/nutch mergesegs /user/nutch/crawl_al/MERGEDsegments -dir
/user/nutch/my_crawl/segments/seg1 /user/nutch/my_crawl/segments/seg2
/user/nutch/my_crawl/segments/seg3, etc -filter -slice 50000
If this works then we need to edit the wiki to accomodate the '/*' which is
required to refer to ALL segments in any given directory.
HTH
On Fri, Oct 28, 2011 at 11:09 PM, Patricio Galeas <pg...@yahoo.de> wrote:
> Hello,
>
> when I try to merge segment using ...
>
> bin/nutch mergesegs /user/nutch/crawl_al/MERGEDsegments -dir
> /user/nutch/my_crawl/segments -filter -slice 50000
>
>
> .... I get the following error. What I'm doing wrong?
>
> Thanks
> Patricio
>
> java.io.EOFException
> at java.io.DataInputStream.readByte(DataInputStream.java:250)
> at org.apache.hadoop.io.WritableUtils.readVLong(WritableUtils.java:298)
> at org.apache.hadoop.io.WritableUtils.readVInt(WritableUtils.java:319)
> at org.apache.hadoop.io.Text.readString(Text.java:400)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.createBlockOutputStream(DFSClient.java:2901)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2826)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2102)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2288)
>
> java.io.EOFException
> at java.io.DataInputStream.readByte(DataInputStream.java:250)
> at org.apache.hadoop.io.WritableUtils.readVLong(WritableUtils.java:298)
> at org.apache.hadoop.io.WritableUtils.readVInt(WritableUtils.java:319)
> at org.apache.hadoop.io.Text.readString(Text.java:400)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.createBlockOutputStream(DFSClient.java:2901)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2826)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2102)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2288)
>
> java.io.EOFException
> at java.io.DataInputStream.readByte(DataInputStream.java:250)
> at org.apache.hadoop.io.WritableUtils.readVLong(WritableUtils.java:298)
> at org.apache.hadoop.io.WritableUtils.readVInt(WritableUtils.java:319)
> at org.apache.hadoop.io.Text.readString(Text.java:400)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.createBlockOutputStream(DFSClient.java:2901)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2826)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2102)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2288)
>
> attempt_201110152026_0003_r_000001_1: log4j:WARN No appenders could be
> found for logger (org.apache.hadoop.hdfs.DFSClient).
> attempt_201110152026_0003_r_000001_1: log4j:WARN Please initialize the
> log4j system properly.
> java.io.EOFException
> at java.io.DataInputStream.readByte(DataInputStream.java:250)
> at org.apache.hadoop.io.WritableUtils.readVLong(WritableUtils.java:298)
> at org.apache.hadoop.io.WritableUtils.readVInt(WritableUtils.java:319)
> at org.apache.hadoop.io.Text.readString(Text.java:400)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.createBlockOutputStream(DFSClient.java:2901)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2826)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2102)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2288)
>
> attempt_201110152026_0003_r_000000_1: log4j:WARN No appenders could be
> found for logger (org.apache.hadoop.hdfs.DFSClient).
> attempt_201110152026_0003_r_000000_1: log4j:WARN Please initialize the
> log4j system properly.
> java.io.EOFException
> at java.io.DataInputStream.readByte(DataInputStream.java:250)
> at org.apache.hadoop.io.WritableUtils.readVLong(WritableUtils.java:298)
> at org.apache.hadoop.io.WritableUtils.readVInt(WritableUtils.java:319)
> at org.apache.hadoop.io.Text.readString(Text.java:400)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.createBlockOutputStream(DFSClient.java:2901)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2826)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2102)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2288)
>
> java.io.EOFException
> at java.io.DataInputStream.readByte(DataInputStream.java:250)
> at org.apache.hadoop.io.WritableUtils.readVLong(WritableUtils.java:298)
> at org.apache.hadoop.io.WritableUtils.readVInt(WritableUtils.java:319)
> at org.apache.hadoop.io.Text.readString(Text.java:400)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.createBlockOutputStream(DFSClient.java:2901)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2826)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2102)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2288)
>
> Exception in thread "main" java.io.IOException: Job failed!
> at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1252)
> at org.apache.nutch.segment.SegmentMerger.merge(SegmentMerger.java:638)
> at org.apache.nutch.segment.SegmentMerger.main(SegmentMerger.java:683
>
--
*Lewis*