[jira] [Updated] (TIKA-3489) Robots.txt files frequently identified as message/rfc822

["Tim Allison (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/TIKA-3489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tim Allison updated TIKA-3489:
------------------------------
    Fix Version/s:     (was: 2.0.1)
                   2.1.0

> Robots.txt files frequently identified as message/rfc822
> --------------------------------------------------------
>
>                 Key: TIKA-3489
>                 URL: https://issues.apache.org/jira/browse/TIKA-3489
>             Project: Tika
>          Issue Type: Bug
>          Components: mime
>    Affects Versions: 2.0.0, 1.25, 1.26, 1.27
>            Reporter: Sebastian Nagel
>            Assignee: Tim Allison
>            Priority: Minor
>             Fix For: 1.27.1, 2.1.0
>
>         Attachments: robots.txt
>
>
> The Tika MIME detector recognizes a robots.txt file as "message/rfc822" if the file starts with a "User-Agent" rule and contains also a second rule not too far away from the beginning, e.g.:
> {noformat}
> User-Agent: goodbot
> Disallow:
> User-Agent: badbot
> Disallow: /
> {noformat}
> The change [7769a2b|https://github.com/apache/tika/commit/7769a2b4fba2b4af7127eba0c7694f663fd97a13] requires that two different clauses are matched. However, the two occurrences of "User-Agent:" (initial and after a new line) are treated as different instead of equivalent matches.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)