You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@opennlp.apache.org by "Kristian Rickert (Jira)" <ji...@apache.org> on 2026/07/21 09:22:00 UTC

[jira] [Created] (OPENNLP-1894) Add dictionary-based tokenization for Japanese, Korean, and Chinese

Kristian Rickert created OPENNLP-1894:
-----------------------------------------

             Summary: Add dictionary-based tokenization for Japanese, Korean, and Chinese
                 Key: OPENNLP-1894
                 URL: https://issues.apache.org/jira/browse/OPENNLP-1894
             Project: OpenNLP
          Issue Type: Improvement
            Reporter: Kristian Rickert
            Assignee: Kristian Rickert


Adds Viterbi lattice segmentation over user-supplied MeCab-format dictionaries, covering Japanese (IPADIC) and Korean (mecab-ko-dic) dictionary layouts, plus a unigram segmenter for Chinese.

Measured on the real IPADIC dictionary:
* about 5 million characters per second segmentation throughput, single-threaded
* 392k dictionary entries load in under one second

Dictionaries are always user-supplied and never bundled, keeping the distribution free of third-party lexical data. The tokenizer manual gains a section whose example is asserted by a test.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)