You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@nutch.apache.org by cesar voulgaris <ce...@gmail.com> on 2007/01/17 05:57:21 UTC

DB_unfetched status

hi I.m using the 0.8.1 version. I have some problem with the
fetcher.threads.fetch property.
Setting the value of this property to 100, then crawling (depth 3)an a
readdb, I get:

CrawlDb statistics start: crawltest/crawldb/
Statistics for CrawlDb: crawltest/crawldb/
TOTAL urls:     82
retry 0:        82
min score:      0.0
avg score:      0.026024392
max score:      1.018
status 1 (DB_unfetched ):        68
status 2 (DB_fetched):  10
status 3 (DB_gone):     4
CrawlDb statistics: done

Lowering the value I get more TOTAL urls and a bigger
DB_fetched/DB_unfetched ratio (always exactly the same crawl )
Does this means that the crawler is missing urls?. What means anyway
DB_unfetched, if i try with fetcher.threads.fetch =1, I still
get lots DB_unfetched results

The hadoop.log doesn`t show any error or distintive message (in nutch
0.7.2I got https exceeded max delay messages with high values)
I preciate any comments, I do´t want to start a big crawl without knowing
the performance of the crawler...thanks