Unlimited access to a web server for Nutch

[Ngoc Giang Nguyen <gi...@gmail.com>]

 Hi all,

I'm running Nutch to crawl some specific websites that I know the web admins
personally. So is there anyway to change the settings of the target web
servers such that they give my Nutch higher priority, let's say unlimited
access, assuming they are all Apache servers? Because usually I observed
that Nutch has a lot of "HTTP max delay" even when I set the timeout quite
large and the network connections are quite perfect (I also double check by
visiting those websites by browser, and they respond well).

All suggestions are welcomed. Thanks a lot.

Best regards,
Giang

Re: Unlimited access to a web server for Nutch

[Doug Cutting <cu...@nutch.org>]

Ngoc Giang Nguyen wrote:
> I'm running Nutch to crawl some specific websites that I know the web admins
> personally. So is there anyway to change the settings of the target web
> servers such that they give my Nutch higher priority, let's say unlimited
> access, assuming they are all Apache servers? Because usually I observed
> that Nutch has a lot of "HTTP max delay" even when I set the timeout quite
> large and the network connections are quite perfect (I also double check by
> visiting those websites by browser, and they respond well).

Try something like fetcher.server.delay=0 and fetcher.threads.per.host=10.

Doug