You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by Ian Reardon <ir...@gmail.com> on 2005/05/13 21:16:37 UTC

Server Delay when crawling

What is a safe number to delay between page requests from the same
host?  I want to crawl as much information as possible in the shortest
amount of time, but I also don't want to hurt the server i'm
crawling....  What do you guys use?  I am using 5 seconds right now.