[jira] [Commented] (NUTCH-1553) Property 'indexer.delete.robots.noindex' not working when using parser-html.

["Sebastian Nagel (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/NUTCH-1553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15358834#comment-15358834 ] 

Sebastian Nagel commented on NUTCH-1553:
----------------------------------------

The failed test (TestHtmlParser.testEncodingDetection()) extracts the keywords from a syntactically not correctly closed meta element:
{code:html}
<meta name="keywords" content="français, español, русский язык, čeština, ελληνικά</meta>
{code}
Although, fixing the HTML syntax would also fix the test, it feels not the correct way since the HTML parser is expected to be robust for syntactically incorrect mark-up. And it worked before with invalid input!

> Property 'indexer.delete.robots.noindex' not working when using parser-html.
> ----------------------------------------------------------------------------
>
>                 Key: NUTCH-1553
>                 URL: https://issues.apache.org/jira/browse/NUTCH-1553
>             Project: Nutch
>          Issue Type: Bug
>          Components: indexer, parser
>    Affects Versions: 1.6
>            Reporter: Alfonso Presa
>            Assignee: Sebastian Nagel
>            Priority: Minor
>             Fix For: 1.13
>
>         Attachments: NUTCH-1553-trunk-1.patch
>
>
> May be I'm doing something wrong, but it seems to me that +NUTCH-1434+ patch only works when using tika's parser. When using parser-html, "robots" metatag is only populated if parse-metatags plugin is enabled and it's done with the prefix "metatag.". So parseData.getMeta("robots") returns nothing if not using tika.
> I guess the simplest solution would be to provide a fallback in case parseData.getMeta("robots") is null and then get parseData.getMeta("metatag.robots") in that case.
> Also dependency of this property with parse-metadata plugin when using parse-html would be something interesting to document somewhere... (nutch-default.xml?)
> Thanks!



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)