Protocol-http - problematic behaviour of the address blocking routine

[Andrzej Bialecki <ab...@getopt.org>]

Hi,

This is just an observation and a warning for those of you who are 
crawling single sites in depth, and encountered frequent "Exceeded 
http.max.delays" exception.

Assume the following scenario: a user runs the CrawlTool to crawl a 
single site. Fetchlists generated by the CrawlTool will contain only 
URLs from that site (which is to say, from the same IP). But now the 
logic in Http.blockAddr() badly interacts with Fetcher, and deadlocks 
FetcherThreads - by default the Fetcher starts 10 threads. Each of these 
threads tries to access the same IP address, but the default value of 
fetcher.threads.per.host is just 1. This means that only the first 
thread will be allowed to run, other 9 threads will be spinning, waiting 
for the first thread to finish. Eventually, some of these waiting 
threads will exceed the maximum wait time, thus throwing the above 
exception.

I don't see immediately what would be the best solution to this problem. 
One way to solve this would be for the CrawlTool to automatically adjust 
"fetcher.threads.per.host" value so that it follows the formula 
"fetcher.threads / number of urls in the urllist". I did this manually 
in the config file, and on the command line to the CrawlTool, so can you 
to avoid this problem for now.

Any comments?

-- 
Best regards,
Andrzej Bialecki
  ___. ___ ___ ___ _ _   __________________________________
[__ || __|__/|__||\/|  Information Retrieval, Semantic Web
___|||__||  \|  ||  |  Embedded Unix, System Integration
http://www.sigram.com  Contact: info at sigram dot com

Re: Protocol-http - problematic behaviour of the address blocking routine

[Doug Cutting <cu...@nutch.org>]

Andrzej Bialecki wrote:
> I don't see immediately what would be the best solution to this problem. 
> One way to solve this would be for the CrawlTool to automatically adjust 
> "fetcher.threads.per.host" value so that it follows the formula 
> "fetcher.threads / number of urls in the urllist". I did this manually 
> in the config file, and on the command line to the CrawlTool, so can you 
> to avoid this problem for now.
> 
> Any comments?

We should not set fetcher.threads.per.host above one by default, since 
that makes the fetcher impolite, ignoring fetcher.server.delay.

A better way to fix this might be to add per-host queues of urls to be 
fetched, waiting for a thread.  When a maximum number of urls are queued 
(globally? per-host?), then the fetcher can either: (a) pause until the 
queues diminish; or (b) discard some urls.  When the fetchlist is 
exhausted, the fetcher may decide to drop all queued urls, or patiently 
wait until all queues are exhausted.  The current implementation is 
effectively (b) where threads are used as crude queues.  The old 
RequestScheduler fetcher implementation used queues as described here, 
but it had bugs and would sometimes lock up.

It comes down to three modes of fetcher operation:
1. impolite but fast;
2. polite but slow, fetching everything;
3. polite but fast, dropping urls from sites that are slow and/or have 
many pages to fetch.

Currently, mode (1) is selected by setting fetcher.server.delay to 
something small, and fetcher.threads.per.host.  Mode (2) is selected by 
setting fetcher.max.delays to something very large.  Mode (3) is 
selected by setting fetcher.max.delays to something small.

The fetcher must always be polite by default.  Folks may configure it 
otherwise for intranet crawling of their own servers but should never 
crawl the open web with an impolite Nutch-based crawler.

Doug