[jira] [Resolved] (NUTCH-1716) RobotRulesParser adds extra '*' to the robots name

["Tejas Patil (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/NUTCH-1716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tejas Patil resolved NUTCH-1716.
--------------------------------

    Resolution: Duplicate

Accidentally duplicated NUTCH-1715

> RobotRulesParser adds extra '*' to the robots name
> --------------------------------------------------
>
>                 Key: NUTCH-1716
>                 URL: https://issues.apache.org/jira/browse/NUTCH-1716
>             Project: Nutch
>          Issue Type: Bug
>          Components: fetcher
>    Affects Versions: 1.7, 2.2.1
>            Reporter: Tejas Patil
>            Assignee: Tejas Patil
>             Fix For: 2.3, 1.8
>
>
> In RobotRulesParser, when Nutch creates a agent string from multiple agents, it combines agents from both 'http.agent.name' and 'http.robots.agents'. Along with that it appends a wildcard (ie. *) to it in the end. This is sent to crawler commons while parsing the rules. The wildcard gets matched first in robots file with (User-agent: *) if that comes before any other matching rule thus resulting in a allowed url being robots denied. 
> This bug was reported by @Markus Jelsma. The discussion over nutch-user can be found here:
> http://mail-archives.apache.org/mod_mbox/nutch-user/201401.mbox/%3CCAFKhtFzBRpVv4MULSxw8RDRR_wbivOt%3DnhFX-w621BR8q%2BxVDQ%40mail.gmail.com%3E 



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)