[jira] [Updated] (OPENNLP-1894) Add dictionary-based tokenization for Japanese, Korean, and Chinese

["Richard Zowalla (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/OPENNLP-1894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Richard Zowalla updated OPENNLP-1894:
-------------------------------------
    Fix Version/s: 3.0.0

> Add dictionary-based tokenization for Japanese, Korean, and Chinese
> -------------------------------------------------------------------
>
>                 Key: OPENNLP-1894
>                 URL: https://issues.apache.org/jira/browse/OPENNLP-1894
>             Project: OpenNLP
>          Issue Type: Improvement
>            Reporter: Kristian Rickert
>            Assignee: Kristian Rickert
>            Priority: Major
>             Fix For: 3.0.0
>
>          Time Spent: 10m
>  Remaining Estimate: 0h
>
> Adds Viterbi lattice segmentation over user-supplied MeCab-format dictionaries, covering Japanese (IPADIC) and Korean (mecab-ko-dic) dictionary layouts, plus a unigram segmenter for Chinese.
> Measured on the real IPADIC dictionary:
> * about 5 million characters per second segmentation throughput, single-threaded
> * 392k dictionary entries load in under one second
> Dictionaries are always user-supplied and never bundled, keeping the distribution free of third-party lexical data. The tokenizer manual gains a section whose example is asserted by a test.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)