You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nutch.apache.org by Nils Hoeller <ni...@arcor.de> on 2005/08/14 17:00:07 UTC

Indexing the whole WebDB or get Pages out of WebDB that are Indexed

Hi,

is there a way to index the whole WebDB,
which means the normal sites  that have been indexed
+ the sites that are one depth deeper and so 
beeing only stored in the WebDB

or

if I have a WebDBReader reader 

and read for example all pages
(reader.pages), is there 
a way to get only pages that have been indexed?


Thanks Nils


Re: Indexing the whole WebDB or get Pages out of WebDB that are Indexed

Posted by Doug Cutting <cu...@nutch.org>.
Nils Hoeller wrote:
> is there a way to index the whole WebDB,
> which means the normal sites  that have been indexed
> + the sites that are one depth deeper and so 
> beeing only stored in the WebDB

This is supposed to be possible, but I think no one has tried this in a 
while and fear it may no longer work.

If you specify '-refetchonly' when you generate your fetchlist then it 
should generate a fetchlist with fetch=false entries for all of the 
previously unfetched pages.  Then the fetcher should pass these through 
to the output with null content, and the indexer should index the url 
and incoming anchor texts.

But glancing at the current code it looks like IndexSegment.java does 
not index entries with ProtocolStatus.NOTFETCHING.

If you desire this behavior, please file a bug report.  Also, please try 
to patch IndexSegment.java so that it does index these entries.  If this 
works, please attach your patch to the bug report.

Doug