You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by zouzhile <zo...@126.com> on 2010/10/10 07:37:08 UTC
Crawl speed control and HTTP Post
Hi all,
I am new to Nutch, and have two questions that I couldn't find the answer via the web pages and configuration files. May I kindly ask you to give me some suggestions/hints on them please? many thanks in advance.
1) Can I control the actually crawl speed? E.g. If I know a web site would block me if I send one request per two seconds, can I control to make sure nutch wouldn't crawl faster than that? and How?
2) Can nutch send HTTP Post for each crawl request (Not for authentication purpose)? Some web sites require the requests send via http post instead of http get.
Re: Crawl speed control and HTTP Post
Posted by Markus Jelsma <ma...@openindex.io>.
Yes, pay attention to these settings in your configuration:
fetcher.server.delay
fetcher.threads.fetch
fetcher.threads.per.host
On Sunday, October 10, 2010 07:37:08 am zouzhile wrote:
> Hi all,
> I am new to Nutch, and have two questions that I couldn't find the answer
> via the web pages and configuration files. May I kindly ask you to give me
> some suggestions/hints on them please? many thanks in advance. 1) Can I
> control the actually crawl speed? E.g. If I know a web site would block me
> if I send one request per two seconds, can I control to make sure nutch
> wouldn't crawl faster than that? and How? 2) Can nutch send HTTP Post for
> each crawl request (Not for authentication purpose)? Some web sites
> require the requests send via http post instead of http get.
--
Markus Jelsma - CTO - Openindex
http://www.linkedin.com/in/markus17
050-8536600 / 06-50258350