You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@tika.apache.org by "Nick Burch (JIRA)" <ji...@apache.org> on 2011/09/24 11:49:26 UTC
[jira] [Resolved] (TIKA-729) TIKA CharsetDetector not detecting
UTF-16BE/UTF-16LE encodings
[ https://issues.apache.org/jira/browse/TIKA-729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Nick Burch resolved TIKA-729.
-----------------------------
Resolution: Duplicate
> TIKA CharsetDetector not detecting UTF-16BE/UTF-16LE encodings
> --------------------------------------------------------------
>
> Key: TIKA-729
> URL: https://issues.apache.org/jira/browse/TIKA-729
> Project: Tika
> Issue Type: Bug
> Components: parser
> Affects Versions: 0.9
> Reporter: Abhishek Jain
>
> Came across this bug when trying to convert Unicode files to UTF-16. For files written in UTF-16BE or UTF-16LE, CharsetDetector detects it as "ISO-8859-1".
> {code}
> import java.io.File;
> import java.io.FileOutputStream;
> import java.io.IOException;
> import java.io.InputStream;
> import java.io.OutputStreamWriter;
> import java.io.Writer;
> import org.apache.tika.exception.TikaException;
> import org.apache.tika.io.TikaInputStream;
> import org.apache.tika.metadata.Metadata;
> import org.apache.tika.parser.txt.CharsetDetector;
> import org.apache.tika.parser.txt.CharsetMatch;
> import org.xml.sax.SAXException;
> public class TikaTextConverter {
> public static void main(String args[]) throws IOException, SAXException, TikaException {
> String inputPath = "/tmp/input.csv";
>
> Writer writer = new OutputStreamWriter(new FileOutputStream(inputPath), "UTF-16LE");
> writer.write("Line1, Some text, Some more text");
> writer.close();
>
> InputStream inputStream = TikaInputStream.get(new File(inputPath).toURI().toURL(), new Metadata());
>
> CharsetDetector detector = new CharsetDetector();
> detector.setText(inputStream);
>
> CharsetMatch[] matches = detector.detectAll();
> for (CharsetMatch match : matches) {
> System.out.println(match.getName());
> }
> }
> }
> {code}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira