You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@opennlp.apache.org by "Kristian Rickert (Jira)" <ji...@apache.org> on 2026/07/21 09:22:00 UTC
[jira] [Created] (OPENNLP-1894) Add dictionary-based tokenization for Japanese, Korean, and Chinese
Kristian Rickert created OPENNLP-1894:
-----------------------------------------
Summary: Add dictionary-based tokenization for Japanese, Korean, and Chinese
Key: OPENNLP-1894
URL: https://issues.apache.org/jira/browse/OPENNLP-1894
Project: OpenNLP
Issue Type: Improvement
Reporter: Kristian Rickert
Assignee: Kristian Rickert
Adds Viterbi lattice segmentation over user-supplied MeCab-format dictionaries, covering Japanese (IPADIC) and Korean (mecab-ko-dic) dictionary layouts, plus a unigram segmenter for Chinese.
Measured on the real IPADIC dictionary:
* about 5 million characters per second segmentation throughput, single-threaded
* 392k dictionary entries load in under one second
Dictionaries are always user-supplied and never bundled, keeping the distribution free of third-party lexical data. The tokenizer manual gains a section whose example is asserted by a test.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)