You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@tika.apache.org by "Georger Araújo (JIRA)" <ji...@apache.org> on 2011/08/03 19:31:27 UTC
[jira] [Issue Comment Edited] (TIKA-369) Improve accuracy of
language detection
[ https://issues.apache.org/jira/browse/TIKA-369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13078588#comment-13078588 ]
Georger Araújo edited comment on TIKA-369 at 8/3/11 5:29 PM:
-------------------------------------------------------------
I've had great results with the language-detection library [1,2].
Pros: great accuracy, fast, Apache licensed.
Cons: unsurprisingly, has trouble with short text.
[1] http://code.google.com/p/language-detection/
[2] http://www.slideshare.net/shuyo/language-detection-library-for-java
was (Author: georger_br):
I've had great results with the language-detection library [1,2].
Pros: great precision, fast, Apache licensed.
Cons: unsurprisingly, has trouble with short text.
[1] http://code.google.com/p/language-detection/
[2] http://www.slideshare.net/shuyo/language-detection-library-for-java
> Improve accuracy of language detection
> --------------------------------------
>
> Key: TIKA-369
> URL: https://issues.apache.org/jira/browse/TIKA-369
> Project: Tika
> Issue Type: Improvement
> Components: languageidentifier
> Affects Versions: 0.6
> Reporter: Ken Krugler
> Assignee: Ken Krugler
> Attachments: Surprise and Coincidence.pdf, lingdet-mccs.pdf, textcat.pdf
>
>
> Currently the LanguageProfile code uses 3-grams to find the best language profile using Pearson's chi-square test. This has three issues:
> 1. The results aren't very good for short runs of text. Ted Dunning's paper (attached) indicates that a log-likelihood ratio (LLR) test works much better, which would then make language detection faster due to less text needing to be processed.
> 2. The current LanguageIdentifier.isReasonablyCertain() method uses an exact value as a threshold for certainty. This is very sensitive to the amount of text being processed, and thus gives false negative results for short runs of text.
> 3. Certainty should also be based on how much better the result is for language X, compared to the next best language. If two languages both had identical sum-of-squares values, and this value was below the threshold, then the result is still not very certain.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira