You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nutch.apache.org by Nils Hoeller <ni...@arcor.de> on 2005/08/14 17:00:07 UTC
Indexing the whole WebDB or get Pages out of WebDB that are Indexed
Hi,
is there a way to index the whole WebDB,
which means the normal sites that have been indexed
+ the sites that are one depth deeper and so
beeing only stored in the WebDB
or
if I have a WebDBReader reader
and read for example all pages
(reader.pages), is there
a way to get only pages that have been indexed?
Thanks Nils
Re: Indexing the whole WebDB or get Pages out of WebDB that are Indexed
Posted by Doug Cutting <cu...@nutch.org>.
Nils Hoeller wrote:
> is there a way to index the whole WebDB,
> which means the normal sites that have been indexed
> + the sites that are one depth deeper and so
> beeing only stored in the WebDB
This is supposed to be possible, but I think no one has tried this in a
while and fear it may no longer work.
If you specify '-refetchonly' when you generate your fetchlist then it
should generate a fetchlist with fetch=false entries for all of the
previously unfetched pages. Then the fetcher should pass these through
to the output with null content, and the indexer should index the url
and incoming anchor texts.
But glancing at the current code it looks like IndexSegment.java does
not index entries with ProtocolStatus.NOTFETCHING.
If you desire this behavior, please file a bug report. Also, please try
to patch IndexSegment.java so that it does index these entries. If this
works, please attach your patch to the bug report.
Doug