You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@opennlp.apache.org by kr...@apache.org on 2026/08/06 12:31:00 UTC

(opennlp) branch OPENNLP-1894-lattice-cjk updated (263035581 -> b79cc3104)

This is an automated email from the ASF dual-hosted git repository.

krickert pushed a change to branch OPENNLP-1894-lattice-cjk
in repository https://gitbox.apache.org/repos/asf/opennlp.git


    from 263035581 OPENNLP-1894: Move lattice tokenizer types into opennlp-api
     add b79cc3104 OPENNLP-1894: Cap archive extract budgets and harden MeCab load edges

No new revisions were added by this update.

Summary of changes:
 .../tools/tokenize/lattice/MecabDictionary.java    | 403 ++++++++++++---------
 .../tools/tokenize/lattice/UnigramSegmenter.java   |  81 +++--
 .../tokenize/lattice/MecabDictionaryInstaller.java | 225 ++++++++++--
 .../tokenize/lattice/LatticeTokenizerTest.java     |  41 ++-
 .../lattice/MecabDictionaryInstallerTest.java      |  71 ++++
 .../tools/tokenize/lattice/TarGzArchives.java      | 122 ++++++-
 opennlp-docs/src/docbkx/tokenizer.xml              |   3 +
 7 files changed, 698 insertions(+), 248 deletions(-)