Trying to fetch and parse many that remain untouched first crawl?

[Steve LaBarbera <st...@gmail.com>]

To all nutch users,

I'm trying to crawl farther into all of the websites but what I'm continuing
to see is that a very small number of pages get actually fetched the very
first time I crawl. If I call the scan continuously after completion
multiple times it does begin to pick up the unfetched pages, but I'd prefer
a single scan that get better initial coverage. Even when I put the number
of rounds as high as 50 I still get a result of for example 1 pages fetched
15 unfetched. I'm using the site http://www.beyondbeautifuljlo.com as my
current bench testing (don't ask me why I'm using this site just ended up
running into it and it looked small enough to better understand coverage via
manual check) but it happens with pretty much every crawl I try. I'm
thinking it's actually the score of the pages that are being found that are
the problem because when I dump the links I see the one page fetched had a
score of 1.0. While all the rest in the list are 0.0. Is there a way to
lower that threshold to fetch things with lower scores? I looked in the
config file but couldn't really find anything that look to help. I have read
that I can plugin to the scoring and maybe overwrite it but wanted to first
see if there was an easier way. 

Thanks in advance,
Steve

Re: Trying to fetch and parse many that remain untouched first crawl?

[Siddharth Shah <ia...@gmail.com>]

Hi Steve,
               I had similar issues while doing a crawl on couple of
websites. This is what I discovered

1. Remove -[?*!@=] from conf/regex-urlfilter.txt If there are pages/seed
URLs with special character they will get filtered out
2. Adjust db.max.outlinks.per.page to be negative (e.g. -1)  I think
default value is 100 (What this means for a parsed pages it will only
process/follow 100 out going links)
3. Be sure to set file.content.limit and http.content.limit to be negative
(e.g. -1) else by default it will only fetch and store about 64KB.

Let me know if this helps.

Thank you,
Sidharth

On Wed, Mar 18, 2015 at 8:26 AM, Steve LaBarbera <
steve.labarbera.one@gmail.com> wrote:

> To all nutch users,
>
> I'm trying to crawl farther into all of the websites but what I'm
> continuing
> to see is that a very small number of pages get actually fetched the very
> first time I crawl. If I call the scan continuously after completion
> multiple times it does begin to pick up the unfetched pages, but I'd prefer
> a single scan that get better initial coverage. Even when I put the number
> of rounds as high as 50 I still get a result of for example 1 pages fetched
> 15 unfetched. I'm using the site http://www.beyondbeautifuljlo.com as my
> current bench testing (don't ask me why I'm using this site just ended up
> running into it and it looked small enough to better understand coverage
> via
> manual check) but it happens with pretty much every crawl I try. I'm
> thinking it's actually the score of the pages that are being found that are
> the problem because when I dump the links I see the one page fetched had a
> score of 1.0. While all the rest in the list are 0.0. Is there a way to
> lower that threshold to fetch things with lower scores? I looked in the
> config file but couldn't really find anything that look to help. I have
> read
> that I can plugin to the scoring and maybe overwrite it but wanted to first
> see if there was an easier way.
>
> Thanks in advance,
> Steve
>
>
>