Indexing Process

[Jeff Maki <cr...@gmail.com>]

Hello everyone,

I'm not going to post my config files as not to spam you all, but I
have a general question: I'm trying to index the pages of a website
(obviously), and I've created a special page with a link to all the
pages I want to index. I then pointed nutch to this special link page.
I set max_outlinks appropriately, and I do see all the page URLs I
expect go by in the log for the fetching stage.

When nutch gets to indexing, however, not all the documents appear in
the log--it looks as if not all of the fetched pages are being
indexed. Searching for terms I know are on the missing pages also
turns up nothing--they're not in the index!?

Can anybody tell me what factors affect the indexing stage? I want to
have nutch index *all* documents it fetches. How can I do this?

Any tips/ideas/things to configure?

Thanks in advance,

-Jeff

Re: Indexing Process

[Carl Cerecke <ca...@nzs.com>]

Look in nutch-default.xml

The properties db.max.outlinks.per.page and http.content.limit might 
need to have their values increased.

Cheers,
Carl.


Jeff Maki wrote:
> Hello everyone,
> 
> I'm not going to post my config files as not to spam you all, but I
> have a general question: I'm trying to index the pages of a website
> (obviously), and I've created a special page with a link to all the
> pages I want to index. I then pointed nutch to this special link page.
> I set max_outlinks appropriately, and I do see all the page URLs I
> expect go by in the log for the fetching stage.
> 
> When nutch gets to indexing, however, not all the documents appear in
> the log--it looks as if not all of the fetched pages are being
> indexed. Searching for terms I know are on the missing pages also
> turns up nothing--they're not in the index!?
> 
> Can anybody tell me what factors affect the indexing stage? I want to
> have nutch index *all* documents it fetches. How can I do this?
> 
> Any tips/ideas/things to configure?
> 
> Thanks in advance,
> 
> -Jeff
> 
> _____________________________________________________________________
> 
> This has been cleaned & processed by www.rocketspam.co.nz
> _____________________________________________________________________
>