[jira] Updated: (NUTCH-876) Remove remaining robots/IP blocking code in lib-http

["Andrzej Bialecki (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/NUTCH-876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Andrzej Bialecki  updated NUTCH-876:
------------------------------------

    Attachment: NUTCH-876.patch

Patch to fix the issue. If there are no objections I'll commit this shortly.

> Remove remaining robots/IP blocking code in lib-http
> ----------------------------------------------------
>
>                 Key: NUTCH-876
>                 URL: https://issues.apache.org/jira/browse/NUTCH-876
>             Project: Nutch
>          Issue Type: Bug
>          Components: fetcher
>    Affects Versions: 2.0
>            Reporter: Andrzej Bialecki 
>            Assignee: Andrzej Bialecki 
>         Attachments: NUTCH-876.patch
>
>
> There are remains of the (very old) blocking code in lib-http/.../HttpBase.java. This code was used with the OldFetcher to manage politeness limits. New trunk doesn't have OldFetcher anymore, so this code is useless. Furthermore, there is an actual bug here - FetcherJob forgets to set Protocol.CHECK_BLOCKING and Protocol.CHECK_ROBOTS to false, and the defaults in lib-http are set to true.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.