You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@tika.apache.org by Jan Høydahl / Cominvent <ja...@cominvent.com> on 2010/08/21 17:55:41 UTC
Fails to detect language for UTF-8 file, but it works for ISO-latin
Hi,
Playing around with the CLI tool. I tried to detect the language of the test files, e.g. the en.test located at tika/tika-core/src/test/resources/org/apache/tika/language/en.test
lap:tika janhoy$ java -jar tika-app/target/tika-app-0.8-SNAPSHOT.jar -m tika-core/src/test/resources/org/apache/tika/language/en.test
Content-Encoding: UTF-8
Content-Length: 22427
Content-Type: text/plain
resourceName: en.test
As you can see, no language was detected.
Now I make a copy and converts it from UTF-8 to ISO-8859-1 and try again:
lap:tika janhoy$ iconv -f UTF-8 -t ISO-8859-1 <tika-core/src/test/resources/org/apache/tika/language/en.test >en-iso.txt
lap:tika janhoy$ java -jar tika-app/target/tika-app-0.8-SNAPSHOT.jar -m en-iso.txt Content-Encoding: ISO-8859-1
Content-Language: en
Content-Length: 22417
Content-Type: text/plain
language: en
resourceName: en-iso.txt
Detected as english. The same is true for the other test language files. It does not detect language for UTF-8 encoded files.
Does anyone see what's wrong?
--
Jan Høydahl, search solution architect
Cominvent AS - www.cominvent.com
Training in Europe - www.solrtraining.com
Re: Fails to detect language for UTF-8 file, but it works for ISO-latin
Posted by Paul Jakubik <pa...@purediscovery.com>.
+1
Hopefully it will be easier to find an efficient algorithm for detecting
character encoding if language estimates are dropped.
On Tue, Aug 24, 2010 at 10:03 AM, Mattmann, Chris A (388J) <
chris.a.mattmann@jpl.nasa.gov> wrote:
> +1...
>
>
>
> On 8/24/10 8:00 AM, "Jukka Zitting" <ju...@gmail.com> wrote:
>
> Hi,
>
> On Sat, Aug 21, 2010 at 5:55 PM, Jan Høydahl / Cominvent
> <ja...@cominvent.com> wrote:
> > Detected as english. The same is true for the other test language files.
> > It does not detect language for UTF-8 encoded files.
>
> The tika-app jar doesn't do language detection by default. The
> language metadata you're seeing is a result of the encoding-based
> language estimate that we get from the ICU4J code we're using.
> Apparently that data set categorizes ISO-8859-1 as an English-specific
> character encoding.
>
> We already dropped encoding-based language estimates from the HTML
> parser, and I think we should do the same also for plain text
> documents.
>
> BR,
>
> Jukka Zitting
>
>
>
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Chris Mattmann, Ph.D.
> Senior Computer Scientist
> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> Office: 171-266B, Mailstop: 171-246
> Email: *Chris.Mattmann@jpl.nasa.gov
> *WWW: *http://sunset.usc.edu/~mattmann/
> *++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Adjunct Assistant Professor, Computer Science Department
> University of Southern California, Los Angeles, CA 90089 USA
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
>
Re: Fails to detect language for UTF-8 file, but it works for ISO-latin
Posted by Jan Høydahl / Cominvent <ja...@cominvent.com>.
TIKA-501
--
Jan Høydahl, search solution architect
Cominvent AS - www.cominvent.com
Training in Europe - www.solrtraining.com
On 26. aug. 2010, at 12.30, Ingo Renner wrote:
>
> Am 24.08.2010 um 17:00 schrieb Jukka Zitting:
>
>> We already dropped encoding-based language estimates from the HTML
>> parser, and I think we should do the same also for plain text
>> documents.
>
> +1
>
> Ingo
>
> --
> Ingo Renner
> TYPO3 Core Developer, Release Manager TYPO3 4.2, Admin Google Summer of Code
>
>
>
>
>
>
Re: Fails to detect language for UTF-8 file, but it works for ISO-latin
Posted by Ingo Renner <in...@typo3.org>.
Am 24.08.2010 um 17:00 schrieb Jukka Zitting:
> We already dropped encoding-based language estimates from the HTML
> parser, and I think we should do the same also for plain text
> documents.
+1
Ingo
--
Ingo Renner
TYPO3 Core Developer, Release Manager TYPO3 4.2, Admin Google Summer of Code
Re: Fails to detect language for UTF-8 file, but it works for
ISO-latin
Posted by "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov>.
+1...
On 8/24/10 8:00 AM, "Jukka Zitting" <ju...@gmail.com> wrote:
Hi,
On Sat, Aug 21, 2010 at 5:55 PM, Jan Høydahl / Cominvent
<ja...@cominvent.com> wrote:
> Detected as english. The same is true for the other test language files.
> It does not detect language for UTF-8 encoded files.
The tika-app jar doesn't do language detection by default. The
language metadata you're seeing is a result of the encoding-based
language estimate that we get from the ICU4J code we're using.
Apparently that data set categorizes ISO-8859-1 as an English-specific
character encoding.
We already dropped encoding-based language estimates from the HTML
parser, and I think we should do the same also for plain text
documents.
BR,
Jukka Zitting
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: Chris.Mattmann@jpl.nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Re: Fails to detect language for UTF-8 file, but it works for ISO-latin
Posted by Jukka Zitting <ju...@gmail.com>.
Hi,
On Sat, Aug 21, 2010 at 5:55 PM, Jan Høydahl / Cominvent
<ja...@cominvent.com> wrote:
> Detected as english. The same is true for the other test language files.
> It does not detect language for UTF-8 encoded files.
The tika-app jar doesn't do language detection by default. The
language metadata you're seeing is a result of the encoding-based
language estimate that we get from the ICU4J code we're using.
Apparently that data set categorizes ISO-8859-1 as an English-specific
character encoding.
We already dropped encoding-based language estimates from the HTML
parser, and I think we should do the same also for plain text
documents.
BR,
Jukka Zitting