CrawlDb and Generator time growing unnaturally

[Peter Harrington <pe...@gmail.com>]

I run Nutch1.3 crawl with topN = 5000, and depth=20.
For the first two crawl cycles the Generator and CrawlDb Update phases
take ~1hour.  Around the 3rd cycle this increases to 3.5 hours, then
around the 9th cycle these two phases take over 12 hours.  I have
plotted out this time and it's not growing naturally as in linearly or
exponsntially or anything like that.  There are distinct digital steps
in the Generator and CrawlDb time.  I expect these phases to take
longer as I have more links but not like this.
After the crawling is complete I started crawling again and the
Generator and CrawlDb time go back to taking ~1 hours.  It seems that
I can keep these times at 1hour if I do not use a depth>2.
Why is this happening?  Any ideas?
During these two phases the processor is 99% utilized, and the memory only 11%.

Re: CrawlDb and Generator time growing unnaturally

[Markus Jelsma <ma...@openindex.io>]

It's likely you're normalizing and filtering in both jobs. We don't do 
filtering or normalization at all for both jobs and rely on ParseOutputFormat 
instead.

> I run Nutch1.3 crawl with topN = 5000, and depth=20.
> For the first two crawl cycles the Generator and CrawlDb Update phases
> take ~1hour.  Around the 3rd cycle this increases to 3.5 hours, then
> around the 9th cycle these two phases take over 12 hours.  I have
> plotted out this time and it's not growing naturally as in linearly or
> exponsntially or anything like that.  There are distinct digital steps
> in the Generator and CrawlDb time.  I expect these phases to take
> longer as I have more links but not like this.
> After the crawling is complete I started crawling again and the
> Generator and CrawlDb time go back to taking ~1 hours.  It seems that
> I can keep these times at 1hour if I do not use a depth>2.
> Why is this happening?  Any ideas?
> During these two phases the processor is 99% utilized, and the memory only
> 11%.