You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by "S.L" <si...@gmail.com> on 2014/06/26 14:47:49 UTC
Crawl-Delay in robots.txt and fetcher.threads.per.queue config property.
Hello All
If I set fetcher.threads.per.queue property to more than 1 , I believe the behavior would be to have those many number of threads per host from Nutch, in that case would Nutch still respect the Crawl-Delay directive in robots.txt and not crawl at a faster pace that what is specified in robots.txt.
In short what I am trying to ask is if setting fetcher.threads.per.queue to 1 is required for being as polite as Crawl-Delay in robots.txt expects?
Thx
Sent from my HTC
Re: Crawl-Delay in robots.txt and fetcher.threads.per.queue config property.
Posted by "Meraj A. Khan" <me...@gmail.com>.
Perfect, thank you Julien!
On Thu, Jun 26, 2014 at 10:21 AM, Julien Nioche <
lists.digitalpebble@gmail.com> wrote:
> >
> > If I set fetcher.threads.per.queue property to more than 1 , I believe
> the
> > behavior would be to have those many number of threads per host from
> Nutch,
> > in that case would Nutch still respect the Crawl-Delay directive in
> > robots.txt and not crawl at a faster pace that what is specified in
> > robots.txt.
> >
>
> > In short what I am trying to ask is if setting fetcher.threads.per.queue
> > to 1 is required for being as polite as Crawl-Delay in robots.txt
> expects?
> >
>
> Using more than 1 thread per queue will ignore any crawl-delay obtained
> from robots.txt (see
>
> https://github.com/apache/nutch/blob/trunk/src/java/org/apache/nutch/fetcher/Fetcher.java#L317
> )
> and use the fetcher.server.min.delay configuration which has a default
> value of 0. So yes, setting fetcher.threads.per.queue to 1 is required for
> being as polite as Crawl-Delay in robots.txt expects.
>
> HTH
>
> Julien
>
> --
>
> Open Source Solutions for Text Engineering
>
> http://digitalpebble.blogspot.com/
> http://www.digitalpebble.com
> http://twitter.com/digitalpebble
>
Re: Crawl-Delay in robots.txt and fetcher.threads.per.queue config property.
Posted by Julien Nioche <li...@gmail.com>.
>
> If I set fetcher.threads.per.queue property to more than 1 , I believe the
> behavior would be to have those many number of threads per host from Nutch,
> in that case would Nutch still respect the Crawl-Delay directive in
> robots.txt and not crawl at a faster pace that what is specified in
> robots.txt.
>
> In short what I am trying to ask is if setting fetcher.threads.per.queue
> to 1 is required for being as polite as Crawl-Delay in robots.txt expects?
>
Using more than 1 thread per queue will ignore any crawl-delay obtained
from robots.txt (see
https://github.com/apache/nutch/blob/trunk/src/java/org/apache/nutch/fetcher/Fetcher.java#L317)
and use the fetcher.server.min.delay configuration which has a default
value of 0. So yes, setting fetcher.threads.per.queue to 1 is required for
being as polite as Crawl-Delay in robots.txt expects.
HTH
Julien
--
Open Source Solutions for Text Engineering
http://digitalpebble.blogspot.com/
http://www.digitalpebble.com
http://twitter.com/digitalpebble