You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@nutch.apache.org by Howie Wang <ho...@hotmail.com> on 2006/03/12 00:02:15 UTC

crawling etiquette

I was wondering what others are setting the max number of fetches
per host to. I'm currently doing between 500-1000. Do you not set
this at all and just set a timeout between fetches to the same host?

Howie