[jira] [Comment Edited] (OPENNLP-1270) Add new languages to the language detector

["Tim Allison (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/OPENNLP-1270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17310946#comment-17310946 ] 

Tim Allison edited comment on OPENNLP-1270 at 3/29/21, 7:37 PM:
----------------------------------------------------------------

We recently got a request on TIKA-3340 to add detection of Burmese.  Leipzig has a small amount of this: ~2k sentences, but the script is distinct, and it didn't look like there was a bunch of English or other languages in the Leipzig data.

In the link above which no longer works, I had unpacked the following leipzig *-sentences.txt files not currently covered by OpenNLP: amh asm azj ban div hat mhr ori tuk uig xho yid.

For Tika, I rebuilt a lang detect model on the original opennlp leipzig data and the extra langs.

If I redo the work to unpack the above languages and Burmese, are there any objections if I commit them to: https://svn.apache.org/repos/bigdata/opennlp/trunk/leipzig/data ?

Thank you!


was (Author: tallison@mitre.org):
We recently got a request on TIKA-3340 to add detection of Burmese.  Leipzig has a small amount of this: ~2k sentences, but the script is distinct, and it didn't look like there was a bunch of English or other languages in the Leipzig data.

In the link above which no longer works, I had unpacked the following leipzig langs not currently covered by OpenNLP: amh asm azj ban div hat mhr ori tuk uig xho yid.

For Tika, I rebuilt a lang detect model on the original opennlp leipzig data and the extra langs.

If I redo the work to unpack the above languages and Burmese, are there any objections if I commit them to: https://svn.apache.org/repos/bigdata/opennlp/trunk/leipzig/data ?

Thank you!

> Add new languages to the language detector
> ------------------------------------------
>
>                 Key: OPENNLP-1270
>                 URL: https://issues.apache.org/jira/browse/OPENNLP-1270
>             Project: OpenNLP
>          Issue Type: Task
>            Reporter: Tim Allison
>            Assignee: Tim Allison
>            Priority: Major
>             Fix For: 1.9.4
>
>         Attachments: report.txt, report.txt
>
>
> Leipzig has several other languages that might be useful to add to the language detector.  I've selected some with > 10k sentences.  Once I build the model and evaluate performance, I'll share the reports, the model and a tgz of the *-sentences.txt files.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)