You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@opennlp.apache.org by kr...@apache.org on 2026/08/18 20:59:18 UTC
(opennlp) branch OPENNLP-1909-resource-installer updated (762435880 -> 44ac89eae)
This is an automated email from the ASF dual-hosted git repository.
krickert pushed a change to branch OPENNLP-1909-resource-installer
in repository https://gitbox.apache.org/repos/asf/opennlp.git
from 762435880 util: Address review: extract repeated test literals, tighten entry-ceiling prose
add 3dc0950e7 OPENNLP-1893: Hunspell-format affix engine over user-supplied dictionaries
add ed7999288 OPENNLP-1893: Twofold suffix analysis through Hunspell continuation classes
add d37af0d32 OPENNLP-1893: Usage, threading, and malformed-input tests for the Hunspell engine, precise javadoc
add bc718eed4 OPENNLP-1893: Document Hunspell dictionary acquisition with a license-preserving download helper
add 322fd1a5a OPENNLP-1893: Cut morphology like hunspell does, accept UTF-8 flags and strip-only rules, and read the parser through the whitespace seam
add d16f3284c OPENNLP-1893: Read flags as code points, tolerate trailing morphology, and stem nothing from nothing
add 98320f530 OPENNLP-1893: Resolve numeric dictionary flags through the AF alias table
add e80f4d193 OPENNLP-1893: Walk only the affix rules that can apply, bucketed by their boundary character
add 2f0eab31e OPENNLP-1893: Decompose unanalyzed words into two flagged compound parts
add 7b7d2c458 OPENNLP-1893: Honor the blocking flags, circumfixes, and compound positioning
add 8d5afa229 OPENNLP-1893: Add hunspell manual coverage with mirror-tested examples
add dd0d2d928 OPENNLP-1893: Apply the review-convention pass: factual license prose, split null contracts, thread-safety annotations
add e0b2b4d68 OPENNLP-1893: Add {@inheritDoc} to the stemmer overrides and trim empty javadoc lines
add d8cc10e29 OPENNLP-1893: Address review: fold the affix twins, extract tags, complete javadoc
add ba8ab36bb OPENNLP-1893: Fail loud on result-altering unsupported affix directives
add 8cede5e29 OPENNLP-1893: Bound stream size and match affix conditions by code point
add 0b27d5ec1 OPENNLP-1893: Verify dictionary downloads by SHA-512 and add an opt-in URL catalog
add 39afc7e59 OPENNLP-1893: Sync shared DownloadUtil with the startup-overridable download ceiling
add 722f3bdf4 OPENNLP-1893: Trigger CI for the DownloadUtil sync commit
add 43a782513 OPENNLP-1893: Address review: unbox the boundary lookups and publish the stream ceiling
add e6d39bd6f OPENNLP-1893: Use a numeric character reference for the no-break space in the manual
add 9f811674f OPENNLP-1893: Reconcile the shared download test files with the sibling PR
add fe53f4c99 OPENNLP-1893: Expose silent COMPOUNDRULE, IGNORE, KEEPCASE and ungated full-strip rules with failing tests
add 39b576e45 OPENNLP-1893: Fail loud on COMPOUNDRULE, IGNORE, KEEPCASE and gate full-strip rules behind FULLSTRIP
add 317404aaf Merge branch 'OPENNLP-1893-hunspell' into OPENNLP-1909-resource-installer
add f733fcffc OPENNLP-1894: Lattice segmentation over user-supplied mecab-format dictionaries
add 781c22214 OPENNLP-1894: Character trie for lattice prefix search
add f1fada5c7 OPENNLP-1894: Frequency-driven segmentation over user-supplied lexicons
add 05737234a OPENNLP-1894: Usage example and edge-case tests for the lattice and unigram segmenters, corrected javadoc
add c7fadc913 OPENNLP-1894: Keep lattice test sources ASCII-only via Unicode escapes, add an EUC-JP loading example
add a256b90a6 OPENNLP-1894: Document dictionary acquisition with a checksum-verifying download helper
add 559fdd072 OPENNLP-1894: Categorize by code point, keep unknown candidates inside their category run, and validate context ids at load
add 04d380ce7 OPENNLP-1894: Precompute category runs, unbox the trie, and reject inexpressible dictionary values
add 625452d92 OPENNLP-1894: Hold the lexicon in a double-array trie with frequency-recoded labels
add cac8a31ea OPENNLP-1894: Chain lattice nodes intrusively instead of allocating per-position lists
add f28c02fc6 OPENNLP-1894: Document lattice CJK tokenization with a mirror-tested example
add 701f1db57 OPENNLP-1894: Apply the review-convention pass and drop unreferenced lexicon accessors
add fdc9f93a4 OPENNLP-1894: Trim parsed lines as Unicode whitespace and document the tokenizer overrides
add 7a98076c8 OPENNLP-1894: Address review: complete javadoc, hoist constants, and fold fixture duplication
add 263035581 OPENNLP-1894: Move lattice tokenizer types into opennlp-api
add b79cc3104 OPENNLP-1894: Cap archive extract budgets and harden MeCab load edges
add b81f34840 OPENNLP-1894: Verify dictionary downloads by SHA-512 and add an opt-in URL catalog
add e2a437acd OPENNLP-1894: Make download and extraction byte budgets overridable at startup
add ec43621bb OPENNLP-1894: Trigger CI for the budget-override commit
add f171dcad1 OPENNLP-1894: Address review: cite named formats and align test conventions
add 163e3960d OPENNLP-1894: Reconcile the shared download test files with the sibling PR
add 8e14a692c OPENNLP-1894: Add a remote-gated end-to-end test over the catalog dictionaries
add f3e2d3bc0 OPENNLP-1894: Bound matrix cells separately so real distributions load
add 85cb63f50 OPENNLP-1894: Extract dictionary files from the archive root only
add 02d9f3c22 Merge branch 'OPENNLP-1894-lattice-cjk' into OPENNLP-1909-resource-installer
add 0eeefacd1 util: Pin converged catalog installs and mecab pax and reinstall behavior (failing tests)
add c2c6aa698 util: Converge dictionary downloads and unpacking on ResourceInstaller
add 44ac89eae model: Document the converged download path for hunspell and mecab
No new revisions were added by this update.
Summary of changes:
dev/README-hunspell-dictionaries.md | 77 ++
dev/README-mecab-dictionaries.md | 126 ++
.../tools/tokenize/lattice/LatticeTokenizer.java | 373 ++++++
.../tools/tokenize/lattice/MecabDictionary.java | 1015 ++++++++++++++
.../opennlp/tools/tokenize/lattice/Morpheme.java | 64 +
.../tools/tokenize/lattice/UnigramSegmenter.java | 334 +++++
.../java/opennlp/tools/util/ResourceLimits.java | 83 ++
.../tools/ml/model/AbstractModelReader.java | 24 +-
.../tools/stemmer/hunspell/AffixCondition.java | 165 +++
.../tools/stemmer/hunspell/HunspellDictionary.java | 1254 ++++++++++++++++++
.../hunspell/HunspellDictionaryDownload.java | 69 +
.../tools/stemmer/hunspell/HunspellStemmer.java | 596 +++++++++
.../stemmer/hunspell/HunspellStemmerFactory.java | 60 +
.../tokenize/lattice/MecabDictionaryInstaller.java | 238 ++++
.../java/opennlp/tools/util/DictionaryCatalog.java | 177 +++
.../tools/util/dictionary-catalog.properties | 52 +
.../hunspell/HunspellDictionaryDownloadTest.java | 71 +
.../hunspell/HunspellManualExampleTest.java | 73 +
.../hunspell/HunspellRealDictionaryTest.java | 126 ++
.../hunspell/HunspellStemmerFactoryTest.java | 181 +++
.../stemmer/hunspell/HunspellStemmerTest.java | 1400 ++++++++++++++++++++
.../tokenize/lattice/LatticeTokenizerTest.java | 826 ++++++++++++
.../tokenize/lattice/LatticeUsageExampleTest.java | 181 +++
.../tokenize/lattice/MecabCatalogEndToEndTest.java | 122 ++
.../lattice/MecabDictionaryInstallerTest.java | 254 ++++
.../tools/tokenize/lattice/TarGzArchives.java | 69 +
.../tokenize/lattice/UnigramSegmenterTest.java | 224 ++++
.../opennlp/tools/util/DictionaryCatalogTest.java | 135 ++
.../java/opennlp/tools/util/DigestTestUtil.java | 64 +
.../opennlp/tools/util/InstallerTestSupport.java | 29 +-
opennlp-docs/src/docbkx/stemmer.xml | 41 +
opennlp-docs/src/docbkx/tokenizer.xml | 44 +
32 files changed, 8504 insertions(+), 43 deletions(-)
create mode 100644 dev/README-hunspell-dictionaries.md
create mode 100644 dev/README-mecab-dictionaries.md
create mode 100644 opennlp-api/src/main/java/opennlp/tools/tokenize/lattice/LatticeTokenizer.java
create mode 100644 opennlp-api/src/main/java/opennlp/tools/tokenize/lattice/MecabDictionary.java
create mode 100644 opennlp-api/src/main/java/opennlp/tools/tokenize/lattice/Morpheme.java
create mode 100644 opennlp-api/src/main/java/opennlp/tools/tokenize/lattice/UnigramSegmenter.java
create mode 100644 opennlp-api/src/main/java/opennlp/tools/util/ResourceLimits.java
create mode 100644 opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/hunspell/AffixCondition.java
create mode 100644 opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/hunspell/HunspellDictionary.java
create mode 100644 opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/hunspell/HunspellDictionaryDownload.java
create mode 100644 opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/hunspell/HunspellStemmer.java
create mode 100644 opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/hunspell/HunspellStemmerFactory.java
create mode 100644 opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/tokenize/lattice/MecabDictionaryInstaller.java
create mode 100644 opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/DictionaryCatalog.java
create mode 100644 opennlp-core/opennlp-runtime/src/main/resources/opennlp/tools/util/dictionary-catalog.properties
create mode 100644 opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/stemmer/hunspell/HunspellDictionaryDownloadTest.java
create mode 100644 opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/stemmer/hunspell/HunspellManualExampleTest.java
create mode 100644 opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/stemmer/hunspell/HunspellRealDictionaryTest.java
create mode 100644 opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/stemmer/hunspell/HunspellStemmerFactoryTest.java
create mode 100644 opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/stemmer/hunspell/HunspellStemmerTest.java
create mode 100644 opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/tokenize/lattice/LatticeTokenizerTest.java
create mode 100644 opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/tokenize/lattice/LatticeUsageExampleTest.java
create mode 100644 opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/tokenize/lattice/MecabCatalogEndToEndTest.java
create mode 100644 opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/tokenize/lattice/MecabDictionaryInstallerTest.java
create mode 100644 opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/tokenize/lattice/TarGzArchives.java
create mode 100644 opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/tokenize/lattice/UnigramSegmenterTest.java
create mode 100644 opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/util/DictionaryCatalogTest.java
create mode 100644 opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/util/DigestTestUtil.java