You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@nutch.apache.org by karthik085 <ka...@gmail.com> on 2007/04/04 23:14:03 UTC

crawl-delay and nutch

You can add a "Crawl-delay: xx" instruction, where "xx" is a delay value
between successive crawler accesses
[http://help.yahoo.com/help/us/ysearch/slurp/slurp-03.html]

If I do this for Nutch bot, will it work? Does it get Crawl-delay
instructions and update the interval? Any other similar ones for threads....
-- 
View this message in context: http://www.nabble.com/crawl-delay-and-nutch-tf3528305.html#a9845743
Sent from the Nutch - User mailing list archive at Nabble.com.