You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@tika.apache.org by "Andreas Meier (JIRA)" <ji...@apache.org> on 2018/03/02 10:57:00 UTC
[jira] [Comment Edited] (TIKA-2592) HTML with charset unicode
handled as utf-16 instead utf-8
[ https://issues.apache.org/jira/browse/TIKA-2592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16383350#comment-16383350 ]
Andreas Meier edited comment on TIKA-2592 at 3/2/18 10:56 AM:
--------------------------------------------------------------
{quote}Before making this kind of change (default "unicode" to UTF-8), wouldn't we want to determine how many HTML documents use "unicode" as the charset and are UTF-16 vs. UTF-8?
{quote}
Good point, my suggestion to set UTF-8 as default was a bit short-sighted.(That's why I like discussions)
{quote}I'm not sure we should make a rule "unicode->UTF-8", rather we should ignore "unicode" if it is in a metaheader and let the charsetdetector do its work? What do you think?
{quote}
This might probably be the best solution for the problem,
Ignoring meta charset="unicode" solves the problem at least for the testfile.
What do you think about Tims suggestion [~kkrugler]?
Do you have a testcorpus or are you crawling the web [~tallison@mitre.org]?
Btw: I have tested some strings with Javas Charset.forName() function and you often get some default value like "Charset.forName("unicode") = UTF_16" :
"arabic" for example returns "iso-8859-6"
So I created two testfiles with windows-1256 encoded text and got the following output:
Tika returns
charset=x-MacCyrillic for [^TestHTMLCharsetCP1256.html]
and
charset=ISO-8859-6 for [^TestHTMLCharsetArabicCP1256.html]
I've also tested the webpages with firefox and chromium:
Firefox fails on all files, while chromium detects the correct charset.
Do we have to handle things like that? Do we have to find out all strings that return a charset when using java.nio.charset.Charset.forName()?
was (Author: andreasmeier):
{quote}
Before making this kind of change (default "unicode" to UTF-8), wouldn't we want to determine how many HTML documents use "unicode" as the charset and are UTF-16 vs. UTF-8?
{quote}
Good point, my suggestion to set UTF-8 as default was a bit short-sighted.(That's why I like discussions)
{quote}
I'm not sure we should make a rule "unicode->UTF-8", rather we should ignore "unicode" if it is in a metaheader and let the charsetdetector do its work? What do you think?
{quote}
This might probably be the best solution for the problem,
Ignoring meta charset="unicode" solves the problem at least for the testfile.
What do you think about Tims suggestion [~kkrugler]?
Do you have a testcorpus or are you crawling the web [~tallison@mitre.org]?
> HTML with charset unicode handled as utf-16 instead utf-8
> ---------------------------------------------------------
>
> Key: TIKA-2592
> URL: https://issues.apache.org/jira/browse/TIKA-2592
> Project: Tika
> Issue Type: Bug
> Affects Versions: 2.0, 1.16, 1.17
> Reporter: Andreas Meier
> Priority: Major
> Attachments: TestCharsetUnicodeHTML.html, TestHTMLCharsetArabicCP1256.html, TestHTMLCharsetCP1256.html, fix-for-TIKA2592-contributed-by-Andreas-Meier.patch
>
>
> HTML files are detected as utf-16 when meta content is set to "unicode".
> {code:XML}
> <meta http-equiv="Content-Type" content="text/html; charset="unicode">
> {code}
>
> Shouldn't the default be utf-8?
> The attached sample file is shown correctly in:
> Chromium Version 55.0.2883.75
> Firefox 50.1.0
> IE 11
> I am aware that there is no charset "unicode" (available character encodings: [http://www.iana.org/assignments/character-sets/character-sets.xhtml|http://www.iana.org/assignments/character-sets/character-sets.xhtml])
> Unfortunately there are many wrong encodings used out there.
> All unknown encodings should be validated or at least be set to default utf-8.
> Regards
> Andreas
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)