Re: Parse benchmark/performance

[reddibabu <re...@gmail.com>]

Hi Folks,

I am running nutch 1.7 on Linux box. I have given threads 100, depth 100 and
topN 100. My Nutch crawl command took 2 and half hours to crawl 9,100 urls
and index them into Solr. It means, for every url it takes almost 1 second. 

I have updated only one entry in "conf/nutch-default.xml", changed
"fetcher.server.delay" value from 5 sec to 0.5 sec. Here, fetcher is taking
little more time compared to parser while crawling.

Is there any more configurations required to crawl and index more url's per
second.

@Roland Von Herget: You mentioned that you achieved performance of
20urls/sec, please let me know what are all the configuration you have done
to achieve this.

@amuseme: You mentioned that you achieved performance of 1500pages/28 sec,
please let me know what are all the configurations have you did.

@ytthet: You mentioned that you achieved performance of 53 pages/sec, please
let me know what are all the configurations have you did.

Please provide me updates so that I can improve my applications performance.

Thanks,
ReddiBabu



--
View this message in context: http://lucene.472066.n3.nabble.com/Parse-benchmark-performance-tp4045827p4127085.html
Sent from the Nutch - User mailing list archive at Nabble.com.