Re: Fetcher threads & automation

[Nicolás Lichtmaier <ni...@reloco.com.ar>]

> The default is 1 also because of netiquette - it's generally frowned 
> upon by webmasters if someone uses more than 1 concurrent request, or 
> makes requests too often (without any delay) - and quite often all 
> requests from such sources get blocked at the firewall level - 
> sometimes, even whole IP classes get blocked.
>
> So, t(h)read carefully ...

Web browsers have traditionally used connections at the same time, so 
that shoul be a reasonable number to use and one that shouldn't get you 
blocked.

Re: Fetcher threads & automation

[Andrzej Bialecki <ab...@getopt.org>]

Nicolás Lichtmaier wrote:
>
>> The default is 1 also because of netiquette - it's generally frowned 
>> upon by webmasters if someone uses more than 1 concurrent request, or 
>> makes requests too often (without any delay) - and quite often all 
>> requests from such sources get blocked at the firewall level - 
>> sometimes, even whole IP classes get blocked.
>>
>> So, t(h)read carefully ...
>
> Web browsers have traditionally used connections at the same time, so 
> that shoul be a reasonable number to use and one that shouldn't get 
> you blocked.

But web browsers don't crawl a single site exhaustively and quickly, 
rather they make a bunch of requests for resources tied to a single 
page, then wait relatively long time, and then make another bunch of 
requests ... So, the request pattern is still more "fair" than in the 
case of a mad crawler.

-- 
Best regards,
Andrzej Bialecki     <><
 ___. ___ ___ ___ _ _   __________________________________
[__ || __|__/|__||\/|  Information Retrieval, Semantic Web
___|||__||  \|  ||  |  Embedded Unix, System Integration
http://www.sigram.com  Contact: info at sigram dot com