You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@nutch.apache.org by Christophe Noel <ch...@cetic.be> on 2005/11/03 10:11:51 UTC

Crawling unpolite problem

Hello,

I'm fetching about 150 web servers in Belgium. My total bandwith used is 
around 2 Mbits. Today I had a big problem, a phone call from Belgian 
gouvernment saying i'm breaking down their web server. I'm crawling with 
unpolite parameters like (fetcher.server.delay = 0.5 and 
threads.per.host=15 and http.max.delay=1500).

To have a polite crawler, what are the best parameters with 
threads.per.host =1 ?

Thank you very much for your answer.

Christophe Noel