You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@tika.apache.org by chraj007 <ch...@gmail.com> on 2012/08/29 17:55:59 UTC

AutoDetectParser is not parsing UTF-16 content types

Hello,
   Im trying to parse a file whose content type is UTF-16. Im unable to
parse the document using the following code. Please Help me.

       ContentHandler textHandler = new BodyContentHandler();
        TeeContentHandler teeHandler 		= 	 new
TeeContentHandler(textHandler);
        parser.parse(input, teeHandler, metadata, context);      
        String tt = textHandler.toString();
//to print the text

 byte[] converttoBytes = tt.getBytes("UTF-16");
        String string = new String(converttoBytes, "utf-8");
       System.out.println(string);

but its printing along with all html tags.

Thank You,
Rajesh Chejerla



--
View this message in context: http://lucene.472066.n3.nabble.com/AutoDetectParser-is-not-parsing-UTF-16-content-types-tp4004075.html
Sent from the Apache Tika - Development mailing list archive at Nabble.com.

Re: AutoDetectParser is not parsing UTF-16 content types

Posted by Ken Krugler <kk...@transpac.com>.
On Aug 29, 2012, at 9:24am, Jukka Zitting wrote:

> Hi,
> 
> On Wed, Aug 29, 2012 at 6:02 PM, chraj007 <ch...@gmail.com> wrote:
>> http://lucene.472066.n3.nabble.com/file/n4004078/test.html test.html
> 
> Looks like that file has an incorrect http-equiv declaration:
> 
>    <META http-equiv="Content-Type" content="text/html; charset=utf-16">
> 
> The encoding of the file is not UTF-16.
> 
> Can you file a TIKA issue about this? Tika should be able to
> automatically detect the correct encoding and use it if the declared
> one is obviously incorrect.

See https://issues.apache.org/jira/browse/TIKA-539 for an existing issue that discusses the challenges of what information to trust with charset detection.

At the time of that issue, i was in favor of a heuristic that used server response/meta tags as truth (if they agreed), otherwise fall back to statistical analysis.

But maybe statistical analysis is now fast/accurate enough, and we should only use the meta tag as a hint for ICU.

-- Ken

--------------------------
Ken Krugler
http://www.scaleunlimited.com
custom big data solutions & training
Hadoop, Cascading, Mahout & Solr

Re: AutoDetectParser is not parsing UTF-16 content types

Posted by Jukka Zitting <ju...@gmail.com>.
Hi,

On Wed, Aug 29, 2012 at 6:02 PM, chraj007 <ch...@gmail.com> wrote:
> http://lucene.472066.n3.nabble.com/file/n4004078/test.html test.html

Looks like that file has an incorrect http-equiv declaration:

    <META http-equiv="Content-Type" content="text/html; charset=utf-16">

The encoding of the file is not UTF-16.

Can you file a TIKA issue about this? Tika should be able to
automatically detect the correct encoding and use it if the declared
one is obviously incorrect.

BR,

Jukka Zitting

Re: AutoDetectParser is not parsing UTF-16 content types

Posted by chraj007 <ch...@gmail.com>.
http://lucene.472066.n3.nabble.com/file/n4004078/test.html test.html 


I Uploaded the parsing file also.



--
View this message in context: http://lucene.472066.n3.nabble.com/AutoDetectParser-is-not-parsing-UTF-16-content-types-tp4004075p4004078.html
Sent from the Apache Tika - Development mailing list archive at Nabble.com.

Re: AutoDetectParser is not parsing UTF-16 content types

Posted by Ken Krugler <kk...@transpac.com>.
On Aug 29, 2012, at 8:55am, chraj007 wrote:

> Hello,
>   Im trying to parse a file whose content type is UTF-16. Im unable to
> parse the document using the following code. Please Help me.
> 
>       ContentHandler textHandler = new BodyContentHandler();
>        TeeContentHandler teeHandler 		= 	 new
> TeeContentHandler(textHandler);
>        parser.parse(input, teeHandler, metadata, context);      

Note that you don't need to use a TeeContentHandler here.

>        String tt = textHandler.toString();
> //to print the text
> 
> byte[] converttoBytes = tt.getBytes("UTF-16");
>        String string = new String(converttoBytes, "utf-8");

The above code won't do what I think you're hoping it will do.

The call to getBytes("UTF-16") will return the tt string as character data encoded using UTF-16.

The second call says to generate a string from bytes that are character data encoding using UTF-8 (which obviously isn't true).

>       System.out.println(string);
> 
> but its printing along with all html tags.

I'm unclear on what you mean by this.

But as Jukka noted in his response, the issue is that you have a document which is encoded as UTF-8, but the HTML has <meta http-equiv="Content-Type" content="text/html; charset=UTF-16">

Currently Tika treats this meta tag charset as the truth. See https://issues.apache.org/jira/browse/TIKA-539 for a discussion on this issue.

Regards,

-- Ken

--------------------------
Ken Krugler
http://www.scaleunlimited.com
custom big data solutions & training
Hadoop, Cascading, Mahout & Solr