[jira] [Created] (TIKA-2671) HtmlEncodingDetector doesnt take provided metadata into account

["Gerard Bouchar (JIRA)" <ji...@apache.org>]

Gerard Bouchar created TIKA-2671:
------------------------------------

             Summary: HtmlEncodingDetector doesnt take provided metadata into account
                 Key: TIKA-2671
                 URL: https://issues.apache.org/jira/browse/TIKA-2671
             Project: Tika
          Issue Type: Bug
            Reporter: Gerard Bouchar


org.apache.tika.parser.html.HtmlEncodingDetector ignores the document's metadata. So when using it to detect the charset of an HTML document that came with a conflicting charset specified at the transport layer level, the encoding specified inside the file is used instead.

This behavior does not conform to what is [specified by the W3C for determining the character encoding of HTML pages|https://html.spec.whatwg.org/multipage/parsing.html#determining-the-character-encoding]. This causes bugs like NUTCH-2599.

 

If HtmlEncodingDetector is not meant to take into account meta-information about the document, then maybe another detector should be provided, that would be a CompositeDetector including, in that order:
 * a new, simple, MetadataEncodingDetector, that would simply return the encoding
 * the existing HtmlEncodingDetector
 * a generic detector, like UniversalEncodingDetector



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)