You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nutch.apache.org by "Stefan Groschupf (JIRA)" <ji...@apache.org> on 2006/06/01 19:24:29 UTC

[jira] Created: (NUTCH-293) support for Crawl-delay in Robots.txt

support for Crawl-delay in Robots.txt
-------------------------------------

         Key: NUTCH-293
         URL: http://issues.apache.org/jira/browse/NUTCH-293
     Project: Nutch
        Type: Improvement

  Components: fetcher  
    Versions: 0.8-dev    
    Reporter: Stefan Groschupf
    Priority: Critical


Nutch need support for Crawl-delay defined in robots.txt, it is not a standard but a de-facto standard.
See:
http://help.yahoo.com/help/us/ysearch/slurp/slurp-03.html
Webmasters start blocking nutch since we do not support it.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira


Re: [jira] Commented: (NUTCH-293) support for Crawl-delay in Robots.txt

Posted by Andrzej Bialecki <ab...@getopt.org>.
Sami Siren wrote:
> Andrzej Bialecki (JIRA) wrote:
>
>>    [ 
>> http://issues.apache.org/jira/browse/NUTCH-293?page=comments#action_12422244 
>> ]            Andrzej Bialecki  commented on NUTCH-293:
>> -----------------------------------------
>>
>> I'm working on this patch to commit it. Just a quick note to Sami: 
>> Math.max() is not optimal, because it always picks up the longest 
>> wait period. We are interested in getting a right period - it may be 
>> longer, but it may also be shorter than the serverDelay. If it's 
>> shorter then we win, because we are allowed to crawl this site faster.
>>
>>  
>>
> I quess it depends on the angle you look at it :)
> "don't be polite, just as polite as it's required"
>
> I'm ok with the original logic.

Hmm. Let me try another explanation.

When you crawl, you _are_ interested in getting all pages as quickly as 
possible, right? Then, you want to observe the minimum level of 
"politeness" per site, as specified by webmasters and netiquette, and 
not the maximum level of politeness.

If a site allows you to crawl it with 5 sec delay, then you won't be 
impolite if you do that, even though you apply 20 sec. delay for all 
other sites - and you will reach your goal much quicker.

-- 
Best regards,
Andrzej Bialecki     <><
 ___. ___ ___ ___ _ _   __________________________________
[__ || __|__/|__||\/|  Information Retrieval, Semantic Web
___|||__||  \|  ||  |  Embedded Unix, System Integration
http://www.sigram.com  Contact: info at sigram dot com



Re: [jira] Commented: (NUTCH-293) support for Crawl-delay in Robots.txt

Posted by Sami Siren <ss...@gmail.com>.
Andrzej Bialecki (JIRA) wrote:

>    [ http://issues.apache.org/jira/browse/NUTCH-293?page=comments#action_12422244 ] 
>            
>Andrzej Bialecki  commented on NUTCH-293:
>-----------------------------------------
>
>I'm working on this patch to commit it. Just a quick note to Sami: Math.max() is not optimal, because it always picks up the longest wait period. We are interested in getting a right period - it may be longer, but it may also be shorter than the serverDelay. If it's shorter then we win, because we are allowed to crawl this site faster.
>
>  
>
I quess it depends on the angle you look at it :)
"don't be polite, just as polite as it's required"

I'm ok with the original logic.

--
 Sami Siren



[jira] Commented: (NUTCH-293) support for Crawl-delay in Robots.txt

Posted by "Andrzej Bialecki (JIRA)" <ji...@apache.org>.
    [ http://issues.apache.org/jira/browse/NUTCH-293?page=comments#action_12422244 ] 
            
Andrzej Bialecki  commented on NUTCH-293:
-----------------------------------------

I'm working on this patch to commit it. Just a quick note to Sami: Math.max() is not optimal, because it always picks up the longest wait period. We are interested in getting a right period - it may be longer, but it may also be shorter than the serverDelay. If it's shorter then we win, because we are allowed to crawl this site faster.

> support for Crawl-delay in Robots.txt
> -------------------------------------
>
>                 Key: NUTCH-293
>                 URL: http://issues.apache.org/jira/browse/NUTCH-293
>             Project: Nutch
>          Issue Type: Improvement
>          Components: fetcher
>    Affects Versions: 0.8-dev
>            Reporter: Stefan Groschupf
>            Priority: Critical
>         Attachments: crawlDelayv1.patch
>
>
> Nutch need support for Crawl-delay defined in robots.txt, it is not a standard but a de-facto standard.
> See:
> http://help.yahoo.com/help/us/ysearch/slurp/slurp-03.html
> Webmasters start blocking nutch since we do not support it.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] Commented: (NUTCH-293) support for Crawl-delay in Robots.txt

Posted by "Stefan Groschupf (JIRA)" <ji...@apache.org>.
    [ http://issues.apache.org/jira/browse/NUTCH-293?page=comments#action_12415236 ] 

Stefan Groschupf commented on NUTCH-293:
----------------------------------------

Hi Andrzej, 
I agree but writing a queue based fetcher is a big step. I already have some basic code (nio based).
Also I don't think that a new fetcher will be as stable as that we can put it into a .8 release. Since we plan to have .8 release it think it is a good idea for now to add this functionality. Maybe we do it configurable and switch it off by default?

In any case I suggest that we solve NUTCH-289 first and than getting the  fetcher done.


> support for Crawl-delay in Robots.txt
> -------------------------------------
>
>          Key: NUTCH-293
>          URL: http://issues.apache.org/jira/browse/NUTCH-293
>      Project: Nutch
>         Type: Improvement

>   Components: fetcher
>     Versions: 0.8-dev
>     Reporter: Stefan Groschupf
>     Priority: Critical
>  Attachments: crawlDelayv1.patch
>
> Nutch need support for Crawl-delay defined in robots.txt, it is not a standard but a de-facto standard.
> See:
> http://help.yahoo.com/help/us/ysearch/slurp/slurp-03.html
> Webmasters start blocking nutch since we do not support it.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira


[jira] Commented: (NUTCH-293) support for Crawl-delay in Robots.txt

Posted by "Stefan Groschupf (JIRA)" <ji...@apache.org>.
    [ http://issues.apache.org/jira/browse/NUTCH-293?page=comments#action_12415171 ] 

Stefan Groschupf commented on NUTCH-293:
----------------------------------------

Any comments? There was already a posting in the nutch agent mailing list, where someone had banned nutch since nutch does not support crawl-delay.
Becasue nutch tries to be polite from my point of view this is a small but important change.
If there are no improvement suggestions can someone of the committers take care of that _please_? :-) 

> support for Crawl-delay in Robots.txt
> -------------------------------------
>
>          Key: NUTCH-293
>          URL: http://issues.apache.org/jira/browse/NUTCH-293
>      Project: Nutch
>         Type: Improvement

>   Components: fetcher
>     Versions: 0.8-dev
>     Reporter: Stefan Groschupf
>     Priority: Critical
>  Attachments: crawlDelayv1.patch
>
> Nutch need support for Crawl-delay defined in robots.txt, it is not a standard but a de-facto standard.
> See:
> http://help.yahoo.com/help/us/ysearch/slurp/slurp-03.html
> Webmasters start blocking nutch since we do not support it.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira


[jira] Commented: (NUTCH-293) support for Crawl-delay in Robots.txt

Posted by "Sami Siren (JIRA)" <ji...@apache.org>.
    [ http://issues.apache.org/jira/browse/NUTCH-293?page=comments#action_12421930 ] 
            
Sami Siren commented on NUTCH-293:
----------------------------------

perhaps instead of
delay = crawlDelay > 0 ? crawlDelay : serverDelay;

we could do
delay=Math.max(crawlDelay, serverDelay);

also the delay could be calculated only once and passed as a parameter 
to blockAddr, unblockAddr



> support for Crawl-delay in Robots.txt
> -------------------------------------
>
>                 Key: NUTCH-293
>                 URL: http://issues.apache.org/jira/browse/NUTCH-293
>             Project: Nutch
>          Issue Type: Improvement
>          Components: fetcher
>    Affects Versions: 0.8-dev
>            Reporter: Stefan Groschupf
>            Priority: Critical
>         Attachments: crawlDelayv1.patch
>
>
> Nutch need support for Crawl-delay defined in robots.txt, it is not a standard but a de-facto standard.
> See:
> http://help.yahoo.com/help/us/ysearch/slurp/slurp-03.html
> Webmasters start blocking nutch since we do not support it.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] Commented: (NUTCH-293) support for Crawl-delay in Robots.txt

Posted by "Andrzej Bialecki (JIRA)" <ji...@apache.org>.
    [ http://issues.apache.org/jira/browse/NUTCH-293?page=comments#action_12415202 ] 

Andrzej Bialecki  commented on NUTCH-293:
-----------------------------------------

Stefan, as you remember we had a discussion on modifying the fetcher, and specifically changing the thread spin-waiting mechanism into a worker-queue. As it is now this is a can of worms that I'd rather not touch - there are many subtle conditions here that would be further complicated by this patch. E.g. the number of spin-waiting threads vs. the number of free threads is normally affected only by five factors: total number of threads, non-uniqueness rate in the current fetchlist, sites' bandwidth, configured delay between requests, and allowed # of threads/host. This patch adds a sixth factor, variable per site .. which makes it much harder to predict how many threads you need to avoid dead-locking all of them.

I'm not strongly opposed to this change, quite contrary - this is a useful functionality. It's just that I'm concerned that it adds yet another functionality to a messy code that needs to be rewritten from scratch.

OTOH, it's a non-intrusive quick hack. If we have to have it now, it's definitely better than waiting for some distant future when we rewrite the fetcher ... ;)

> support for Crawl-delay in Robots.txt
> -------------------------------------
>
>          Key: NUTCH-293
>          URL: http://issues.apache.org/jira/browse/NUTCH-293
>      Project: Nutch
>         Type: Improvement

>   Components: fetcher
>     Versions: 0.8-dev
>     Reporter: Stefan Groschupf
>     Priority: Critical
>  Attachments: crawlDelayv1.patch
>
> Nutch need support for Crawl-delay defined in robots.txt, it is not a standard but a de-facto standard.
> See:
> http://help.yahoo.com/help/us/ysearch/slurp/slurp-03.html
> Webmasters start blocking nutch since we do not support it.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira


[jira] Closed: (NUTCH-293) support for Crawl-delay in Robots.txt

Posted by "Andrzej Bialecki (JIRA)" <ji...@apache.org>.
     [ http://issues.apache.org/jira/browse/NUTCH-293?page=all ]

Andrzej Bialecki  closed NUTCH-293.
-----------------------------------

    Fix Version/s: 0.8-dev
       Resolution: Fixed

Patch applied with minor changes. Thank you!

> support for Crawl-delay in Robots.txt
> -------------------------------------
>
>                 Key: NUTCH-293
>                 URL: http://issues.apache.org/jira/browse/NUTCH-293
>             Project: Nutch
>          Issue Type: Improvement
>          Components: fetcher
>    Affects Versions: 0.8-dev
>            Reporter: Stefan Groschupf
>            Priority: Critical
>             Fix For: 0.8-dev
>
>         Attachments: crawlDelayv1.patch
>
>
> Nutch need support for Crawl-delay defined in robots.txt, it is not a standard but a de-facto standard.
> See:
> http://help.yahoo.com/help/us/ysearch/slurp/slurp-03.html
> Webmasters start blocking nutch since we do not support it.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] Updated: (NUTCH-293) support for Crawl-delay in Robots.txt

Posted by "Stefan Groschupf (JIRA)" <ji...@apache.org>.
     [ http://issues.apache.org/jira/browse/NUTCH-293?page=all ]

Stefan Groschupf updated NUTCH-293:
-----------------------------------

    Attachment: crawlDelayv1.patch

A frist darft of a crawl delay support for nutch. The problem I see is that in case ip based delay is configured it can happen that we use the crawl delay of one host for a other host running on the same ip.
Feedback is welcome.

> support for Crawl-delay in Robots.txt
> -------------------------------------
>
>          Key: NUTCH-293
>          URL: http://issues.apache.org/jira/browse/NUTCH-293
>      Project: Nutch
>         Type: Improvement

>   Components: fetcher
>     Versions: 0.8-dev
>     Reporter: Stefan Groschupf
>     Priority: Critical
>  Attachments: crawlDelayv1.patch
>
> Nutch need support for Crawl-delay defined in robots.txt, it is not a standard but a de-facto standard.
> See:
> http://help.yahoo.com/help/us/ysearch/slurp/slurp-03.html
> Webmasters start blocking nutch since we do not support it.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira