[jira] [Commented] (TIKA-2671) HtmlEncodingDetector doesnt take provided metadata into account

["Tim Allison (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/TIKA-2671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16514112#comment-16514112 ] 

Tim Allison commented on TIKA-2671:
-----------------------------------

+1 Thank you for raising this.  We're currently using CONTENT_TYPE_HINT to record what was extracted from the meta-header.  I guess we should create a new property that the HTMLEncodingDetector should use.

To confirm the algorithm:
1) Trust the actual http header
2) If that doesn't exist, trust the html meta-header
3) If that doesn't exist, fall back on charset detection

The best solution would probably be to try each (when they differ) and create an out-of-vocabulary score based on tika-eval's word lists and pick the encoding with the lowest OOV%.

Short of that, though, how does the above look?  Any interest in putting together a PR?

> HtmlEncodingDetector doesnt take provided metadata into account
> ---------------------------------------------------------------
>
>                 Key: TIKA-2671
>                 URL: https://issues.apache.org/jira/browse/TIKA-2671
>             Project: Tika
>          Issue Type: Bug
>            Reporter: Gerard Bouchar
>            Priority: Major
>
> org.apache.tika.parser.html.HtmlEncodingDetector ignores the document's metadata. So when using it to detect the charset of an HTML document that came with a conflicting charset specified at the transport layer level, the encoding specified inside the file is used instead.
> This behavior does not conform to what is [specified by the W3C for determining the character encoding of HTML pages|https://html.spec.whatwg.org/multipage/parsing.html#determining-the-character-encoding]. This causes bugs similar to NUTCH-2599.
>  
> If HtmlEncodingDetector is not meant to take into account meta-information about the document, then maybe another detector should be provided, that would be a CompositeDetector including, in that order:
>  * a new, simple, MetadataEncodingDetector, that would simply return the encoding
>  * the existing HtmlEncodingDetector
>  * a generic detector, like UniversalEncodingDetector



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)