You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@tika.apache.org by "Tim Allison (JIRA)" <ji...@apache.org> on 2019/07/16 14:14:00 UTC

[jira] [Resolved] (TIKA-2790) Consider switching lang-detection in tika-eval to open-nlp

     [ https://issues.apache.org/jira/browse/TIKA-2790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tim Allison resolved TIKA-2790.
-------------------------------
       Resolution: Fixed
         Assignee: Tim Allison
    Fix Version/s: 1.22

More work remains for short and/or noisy text, but the work performed on this ticket:
* Increases the number of langs detected from 75 to 121
* Increases the common words lists from 21 languages to 121 languages
* Improves processing speed over Optimaize on un-noised data > 1000 characters

Evaluation code is available: https://github.com/tballison/tika-addons/tree/master/lang-id

> Consider switching lang-detection in tika-eval to open-nlp
> ----------------------------------------------------------
>
>                 Key: TIKA-2790
>                 URL: https://issues.apache.org/jira/browse/TIKA-2790
>             Project: Tika
>          Issue Type: Improvement
>            Reporter: Tim Allison
>            Assignee: Tim Allison
>            Priority: Major
>             Fix For: 1.22
>
>         Attachments: fra_mixed_100000_0.0_0.txt, hasEnough.png, langid_20190509.zip, langid_20190510.zip, langid_20190514.zip, langid_20190514_plus_minus_1.zip, rollups_20190716.zip, timeVsLength.png
>
>




--
This message was sent by Atlassian JIRA
(v7.6.14#76016)