[jira] [Comment Edited] (NUTCH-1995) Add support for wildcard to http.robot.rules.whitelist

["Sebastian Nagel (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/NUTCH-1995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14560659#comment-14560659 ] 

Sebastian Nagel edited comment on NUTCH-1995 at 5/27/15 9:04 AM:
-----------------------------------------------------------------

The result of {{conf.getStrings("http.robot.rules.whitelist")}} needs to be checked for null which is returned if the property is not defined or is empty. That's the default and is used for the tests.
[~gostep], can you add this check? And don't forget to mention this issue in CHANGES.txt. Thanks!


was (Author: wastl-nagel):
The result of {{conf.getStrings("http.robot.rules.whitelist")}} needs to be checked for null which is returned if the property is not defined or is empty. That's the default and is used for the tests.

> Add support for wildcard to http.robot.rules.whitelist
> ------------------------------------------------------
>
>                 Key: NUTCH-1995
>                 URL: https://issues.apache.org/jira/browse/NUTCH-1995
>             Project: Nutch
>          Issue Type: Improvement
>          Components: robots
>    Affects Versions: 1.10
>            Reporter: Giuseppe Totaro
>            Assignee: Giuseppe Totaro
>              Labels: memex
>             Fix For: 1.11
>
>         Attachments: NUTCH-1995.MattmannNagelTotaro.05-26-2015.patch, NUTCH-1995.MattmannNagelTotaro.patch, NUTCH-1995.patch
>
>
> The {{http.robot.rules.whitelist}} ([NUTCH-1927|https://issues.apache.org/jira/browse/NUTCH-1927]) configuration parameter allows to specify a comma separated list of hostnames or IP addresses to ignore robot rules parsing for.
> Adding support for wildcard in {{http.robot.rules.whitelist}} could be very useful and simplify the configuration, for example, if we need to give many hostnames/addresses. Here is an example:
> {noformat}
> <name>http.robot.rules.whitelist</name>
>   <value>*.sample.com</value>
>   <description>Comma separated list of hostnames or IP addresses to ignore 
>   robot rules parsing for. Use with care and only if you are explicitly
>   allowed by the site owner to ignore the site's robots.txt!
>   </description>
> </property>
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)