You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@tika.apache.org by "Aleksandr Dubinsky (JIRA)" <ji...@apache.org> on 2014/08/31 14:22:20 UTC

[jira] [Created] (TIKA-1406) Problems in TXT encoding detection

Aleksandr Dubinsky created TIKA-1406:
----------------------------------------

             Summary: Problems in TXT encoding detection
                 Key: TIKA-1406
                 URL: https://issues.apache.org/jira/browse/TIKA-1406
             Project: Tika
          Issue Type: Bug
          Components: parser
    Affects Versions: 1.6, 1.7
            Reporter: Aleksandr Dubinsky


The detection of TXT file encoding often makes mistakes. Two things can improve the detection in many cases:

- Increase the lookahead from 16k bytes to a larger number, such as 128k or larger.
- Improve on the brain-dead heuristic that if a file doesn't have a \r then it must be ISO 8859-1(5) instead of Windows-1252. (For one, it mis-detects files that don't have any newlines.)

A remaining problem that doesn't have an immediate solution, is the frequent mis-detection of Windows-1252 as Shift-JIS. A flag to forbit Shift-JIS is desirable.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)