You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@nutch.apache.org by "Sebastian Nagel (JIRA)" <ji...@apache.org> on 2013/04/26 21:16:16 UTC

[jira] [Updated] (NUTCH-966) Behavior of NOINDEX,FOLLOW is not intuitive

     [ https://issues.apache.org/jira/browse/NUTCH-966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sebastian Nagel updated NUTCH-966:
----------------------------------

    Fix Version/s: 1.8
    
> Behavior of NOINDEX,FOLLOW is not intuitive
> -------------------------------------------
>
>                 Key: NUTCH-966
>                 URL: https://issues.apache.org/jira/browse/NUTCH-966
>             Project: Nutch
>          Issue Type: Improvement
>          Components: indexer, parser
>    Affects Versions: 1.2
>            Reporter: Josh Pavel
>            Priority: Minor
>             Fix For: 2.3, 1.8
>
>
> If a page has NOINDEX,FOLLOW for the ROBOTS metatag, Nutch will still create a document that can be found in the index via metatag or URL matching.  Instead, Nutch should rely on doc or parse metadata but nothing should be stored by the html parser. (thanks to Julien Nioche for helping me to understand the issue). 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira