You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@nutch.apache.org by Lyndon Maydwell <ma...@gmail.com> on 2007/09/06 05:39:47 UTC

Slow search

Hello,

I'm using nutch through the opensearch interface, and am noticing very
slow search speeds, ie: 3-4 seconds.

I really need to find some way to speed the search up significantly.

During the search 'top' indicates that it is using close to 100% CPU
and around 40M of ram

line from top when not running:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
24458 ceims 22 0 218m 35m 10m S 0.0 3.5 0:15.20 java

My stats are :

CrawlDb statistics start: crawl/crawldb
Statistics for CrawlDb: crawl/crawldb
TOTAL urls:     27478
retry 0:        27367
retry 1:        87
retry 2:        21
retry 3:        3
min score:      0.0
avg score:      0.017
max score:      18.102
status 1 (db_unfetched):        13909
status 2 (db_fetched):  12738
status 3 (db_gone):     417
status 4 (db_redir_temp):       180
status 5 (db_redir_perm):       234
CrawlDb statistics: done

and I'm running JRE version 1.6.0_01-b06

and tomcat version Apache Tomcat/6.0.13