You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by Pankaj Mathur <ag...@hotmail.com> on 2006/10/10 05:00:35 UTC
Problem with readseg
Hi,
I am running 'readseg' to get data for a particular URL and getting the
following exception. (see attached trace)
Everything else like search etc. is working fine. However, I have been
unable to understand this error.
Any help is highly appreciated
thanks
-Sameer
----------------------------------------------------------
This is the trace:
$ bin/nutch readseg -get segments/20061008154327
http://www.bartleby.com/100/
SegmentReader: get 'http://www.bartleby.com/100/'
06/10/09 19:31:53 INFO segment.SegmentReader: SegmentReader: get
'http://www.bar
tleby.com/100/'
java.lang.ArrayIndexOutOfBoundsException: 0
06/10/09 19:31:53 WARN segment.SegmentReader:
java.lang.ArrayIndexOutOfBoundsExc
eption: 0
at
org.apache.nutch.segment.SegmentReader.getMapRecords(SegmentReader.java:352)
06/10/09 19:31:53 WARN segment.SegmentReader: at
org.apache.nutch.segment.Segmen
tReader.getMapRecords(SegmentReader.java:352)
at org.apache.nutch.segment.SegmentReader.access$000(SegmentReader.java:40)
06/10/09 19:31:53 WARN segment.SegmentReader: at
org.apache.nutch.segment.Segmen
tReader.access$000(SegmentReader.java:40)
at org.apache.nutch.segment.SegmentReader$1.run(SegmentReader.java:265)
06/10/09 19:31:53 WARN segment.SegmentReader: at
org.apache.nutch.segment.Segmen
tReader$1.run(SegmentReader.java:265)
java.lang.ArrayIndexOutOfBoundsException: 0
06/10/09 19:31:53 WARN segment.SegmentReader:
java.lang.ArrayIndexOutOfBoundsExc
eption: 0
at
org.apache.nutch.segment.SegmentReader.getMapRecords(SegmentReader.java:352)
06/10/09 19:31:53 WARN segment.SegmentReader: at
org.apache.nutch.segment.Segmen
tReader.getMapRecords(SegmentReader.java:352)
at org.apache.nutch.segment.SegmentReader.access$000(SegmentReader.java:40)
06/10/09 19:31:53 WARN segment.SegmentReader: at
org.apache.nutch.segment.Segmen
tReader.access$000(SegmentReader.java:40)
at org.apache.nutch.segment.SegmentReader$2.run(SegmentReader.java:275)
06/10/09 19:31:53 WARN segment.SegmentReader: at
org.apache.nutch.segment.Segmen
tReader$2.run(SegmentReader.java:275)
java.lang.ArrayIndexOutOfBoundsException: 0
06/10/09 19:31:53 WARN segment.SegmentReader:
java.lang.ArrayIndexOutOfBoundsExc
eption: 0
at
org.apache.nutch.segment.SegmentReader.getSeqRecords(SegmentReader.java:369)
06/10/09 19:31:53 WARN segment.SegmentReader: at
org.apache.nutch.segment.Segmen
tReader.getSeqRecords(SegmentReader.java:369)
at org.apache.nutch.segment.SegmentReader.access$100(SegmentReader.java:40)
06/10/09 19:31:53 WARN segment.SegmentReader: at
org.apache.nutch.segment.Segmen
tReader.access$100(SegmentReader.java:40)
at org.apache.nutch.segment.SegmentReader$3.run(SegmentReader.java:285)
06/10/09 19:31:53 WARN segment.SegmentReader: at
org.apache.nutch.segment.Segmen
tReader$3.run(SegmentReader.java:285)
java.lang.ArrayIndexOutOfBoundsException: 0
06/10/09 19:31:53 WARN segment.SegmentReader:
java.lang.ArrayIndexOutOfBoundsExc
eption: 0
at
org.apache.nutch.segment.SegmentReader.getSeqRecords(SegmentReader.java:369)
06/10/09 19:31:53 WARN segment.SegmentReader: at
org.apache.nutch.segment.Segmen
tReader.getSeqRecords(SegmentReader.java:369)
at org.apache.nutch.segment.SegmentReader.access$100(SegmentReader.java:40)
06/10/09 19:31:53 WARN segment.SegmentReader: at
org.apache.nutch.segment.Segmen
tReader.access$100(SegmentReader.java:40)
at org.apache.nutch.segment.SegmentReader$4.run(SegmentReader.java:295)
06/10/09 19:31:53 WARN segment.SegmentReader: at
org.apache.nutch.segment.Segmen
tReader$4.run(SegmentReader.java:295)
java.lang.ArrayIndexOutOfBoundsException: 0
06/10/09 19:31:53 WARN segment.SegmentReader:
java.lang.ArrayIndexOutOfBoundsExc
eption: 0
at
org.apache.nutch.segment.SegmentReader.getMapRecords(SegmentReader.java:352)
06/10/09 19:31:53 WARN segment.SegmentReader: at
org.apache.nutch.segment.Segmen
tReader.getMapRecords(SegmentReader.java:352)
at org.apache.nutch.segment.SegmentReader.access$000(SegmentReader.java:40)
06/10/09 19:31:53 WARN segment.SegmentReader: at
org.apache.nutch.segment.Segmen
tReader.access$000(SegmentReader.java:40)
at org.apache.nutch.segment.SegmentReader$5.run(SegmentReader.java:305)
06/10/09 19:31:53 WARN segment.SegmentReader: at
org.apache.nutch.segment.Segmen
tReader$5.run(SegmentReader.java:305)
java.lang.ArrayIndexOutOfBoundsException: 0
06/10/09 19:31:53 WARN segment.SegmentReader:
java.lang.ArrayIndexOutOfBoundsExc
eption: 0
at
org.apache.nutch.segment.SegmentReader.getMapRecords(SegmentReader.java:352)
06/10/09 19:31:53 WARN segment.SegmentReader: at
org.apache.nutch.segment.Segmen
tReader.getMapRecords(SegmentReader.java:352)
at org.apache.nutch.segment.SegmentReader.access$000(SegmentReader.java:40)
06/10/09 19:31:53 WARN segment.SegmentReader: at
org.apache.nutch.segment.Segmen
tReader.access$000(SegmentReader.java:40)
at org.apache.nutch.segment.SegmentReader$6.run(SegmentReader.java:315)
06/10/09 19:31:53 WARN segment.SegmentReader: at
org.apache.nutch.segment.Segmen
tReader$6.run(SegmentReader.java:315)
Re: Re: How to read content of a particular url from the crawldb?
Posted by Albert Chern <al...@gmail.com>.
That's a good point. I think the Lucene index might be the only place
where that information is stored. If you really needed it I guess you
could build your own mapping of URL to segment. However, I am not
that familiar with Nutch so I will let someone with more experience
answer this.
On 10/15/06, shjiang <ji...@souchang.com> wrote:
> from the version 0.8 ,"bin/nutch segread" has been replaced by
> "bin/nutch readseg"
> I try the command :bin/nutch readseg -get
> ./crawl/segments/20061013144233/ http://www.nokia.com.cn/
> I can get the entire content of the url.
> The problem is that there are sevaral segments directories under
> ./crawl/segments/ ,how can i know the content of the specified url in
> which segment.
> > You could do it from the command line using bin/nutch segread, or you
> > could do it in Java by opening map file readers on the directories
> > called "content" found in each segment.
> >
> > On 10/15/06, shjiang <ji...@souchang.com> wrote:
> >> I cannot find any api that support this function to read the content of
> >> a specified url from the crawldb.
> >>
> >
> >
>
>
Re: How to read content of a particular url from the crawldb?
Posted by shjiang <ji...@souchang.com>.
from the version 0.8 ,"bin/nutch segread" has been replaced by
"bin/nutch readseg"
I try the command :bin/nutch readseg -get
./crawl/segments/20061013144233/ http://www.nokia.com.cn/
I can get the entire content of the url.
The problem is that there are sevaral segments directories under
./crawl/segments/ ,how can i know the content of the specified url in
which segment.
> You could do it from the command line using bin/nutch segread, or you
> could do it in Java by opening map file readers on the directories
> called "content" found in each segment.
>
> On 10/15/06, shjiang <ji...@souchang.com> wrote:
>> I cannot find any api that support this function to read the content of
>> a specified url from the crawldb.
>>
>
>
Re: How to read content of a particular url from the crawldb?
Posted by Albert Chern <al...@gmail.com>.
You could do it from the command line using bin/nutch segread, or you
could do it in Java by opening map file readers on the directories
called "content" found in each segment.
On 10/15/06, shjiang <ji...@souchang.com> wrote:
> I cannot find any api that support this function to read the content of
> a specified url from the crawldb.
>
How to read content of a particular url from the crawldb?
Posted by shjiang <ji...@souchang.com>.
I cannot find any api that support this function to read the content of
a specified url from the crawldb.