You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@tika.apache.org by "Shabanali Faghani (JIRA)" <ji...@apache.org> on 2016/08/04 10:22:20 UTC
[jira] [Issue Comment Deleted] (TIKA-2038) A more accurate facility
for detecting Charset Encoding of HTML documents
[ https://issues.apache.org/jira/browse/TIKA-2038?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Shabanali Faghani updated TIKA-2038:
------------------------------------
Comment: was deleted
(was: As I've said above URLs are available in the [./test-data/language-wise/|https://github.com/shabanali-faghani/IUST-HTMLCharDet/tree/master/test-data/language-wise] relative path of my repo. Note that you should use the last 8 files, not directories. Results of my evaluation are available within [results|https://github.com/shabanali-faghani/IUST-HTMLCharDet/tree/master/test-data/language-wise/results] sub-directory.)
> A more accurate facility for detecting Charset Encoding of HTML documents
> -------------------------------------------------------------------------
>
> Key: TIKA-2038
> URL: https://issues.apache.org/jira/browse/TIKA-2038
> Project: Tika
> Issue Type: Improvement
> Components: core, detector
> Reporter: Shabanali Faghani
> Priority: Minor
> Attachments: comparisons_20160803b.xlsx, iust_encodings.zip, tika_1_14-SNAPSHOT_encoding_detector.zip
>
>
> Currently, Tika uses icu4j for detecting charset encoding of HTML documents as well as the other naturally text documents. But the accuracy of encoding detector tools, including icu4j, in dealing with the HTML documents is meaningfully less than from which the other text documents. Hence, in our project I developed a library that works pretty well for HTML documents, which is available here: https://github.com/shabanali-faghani/IUST-HTMLCharDet
> Since Tika is widely used with and within some of other Apache stuffs such as Nutch, Lucene, Solr, etc. and these projects are strongly in connection with the HTML documents, it seems that having such an facility in Tika also will help them to become more accurate.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)