(opennlp) 11/12: OPENNLP-1894: Document lattice CJK tokenization with a mirror-tested example

[kr...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

krickert pushed a commit to branch OPENNLP-1894-lattice-cjk
in repository https://gitbox.apache.org/repos/asf/opennlp.git

commit 104cd0e2954c30a9bed1a157ba7ae38341fef572
Author: Kristian Rickert <kr...@gmail.com>
AuthorDate: Mon Jul 20 04:44:21 2026 -0400

    OPENNLP-1894: Document lattice CJK tokenization with a mirror-tested example
    
    Add a lattice tokenizer section to the manual citing LatticeUsageExampleTest.
---
 opennlp-docs/src/docbkx/tokenizer.xml | 25 +++++++++++++++++++++++++
 1 file changed, 25 insertions(+)

diff --git a/opennlp-docs/src/docbkx/tokenizer.xml b/opennlp-docs/src/docbkx/tokenizer.xml
index cd1d8a2dd..fbc74fbb2 100644
--- a/opennlp-docs/src/docbkx/tokenizer.xml
+++ b/opennlp-docs/src/docbkx/tokenizer.xml
@@ -538,5 +538,30 @@ wordTokenizer.tokenize("The quick brown fox.", (start, end, type) -> {
 				</programlisting>
 			</para>
 		</section>
+		<section xml:id="tools.tokenizer.lattice">
+			<title>Lattice tokenization for CJK</title>
+			<para>
+				Languages written without spaces need a dictionary-backed segmenter.
+				<code>LatticeTokenizer</code> scores paths over a mecab-format dictionary and
+				emits the cheapest segmentation with spans in original text coordinates.
+				<code>UnigramSegmenter</code> does the same from a plain frequency lexicon.
+				Install a dictionary archive with <code>MecabDictionaryInstaller</code>, load it
+				as a <code>MecabDictionary</code>, and tokenize.
+				<code>LatticeUsageExampleTest</code> asserts the install-load-tokenize and
+				lexicon flows shown here.
+				<programlisting language="java">
+<![CDATA[MecabDictionaryInstaller.install(archiveUri, dictionaryDirectory);
+LatticeTokenizer tokenizer =
+    new LatticeTokenizer(MecabDictionary.load(dictionaryDirectory));
+
+String[] tokens = tokenizer.tokenize(text);
+Span[] spans = tokenizer.tokenizePos(text);
+List<Morpheme> morphemes = tokenizer.analyze(text);
+
+UnigramSegmenter segmenter = UnigramSegmenter.load(Path.of("words.txt"));
+String[] words = segmenter.tokenize(text);]]>
+				</programlisting>
+			</para>
+		</section>
 	</section>
 </chapter>