You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@opennlp.apache.org by rz...@apache.org on 2026/09/04 11:59:51 UTC
(opennlp) branch main updated: OPENNLP-1893: Support Hunspell affix dictionaries for stemming (#1190)
This is an automated email from the ASF dual-hosted git repository.
rzo1 pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/opennlp.git
The following commit(s) were added to refs/heads/main by this push:
new 1f929fad5 OPENNLP-1893: Support Hunspell affix dictionaries for stemming (#1190)
1f929fad5 is described below
commit 1f929fad538e53c3ac7b5dbf90a2fc88102522c2
Author: Kristian Rickert <kr...@gmail.com>
AuthorDate: Fri Sep 4 07:59:46 2026 -0400
OPENNLP-1893: Support Hunspell affix dictionaries for stemming (#1190)
* OPENNLP-1893: Hunspell-format affix engine over user-supplied dictionaries
A clean-room reader for .dic and .aff files with PFX/SFX rules, strip
strings, character-class conditions matched by a single scan, cross
products, and char, long, and num flag modes. No dictionary data is
bundled: users point at their own files, so dictionary licenses never
attach to the jar. Unsupported affix features fail closed, missing
analyses rather than inventing them.
(cherry picked from commit 0ecc39c7febdeaaafe1cfbdf052c9dd06ffbe4d4)
* OPENNLP-1893: Twofold suffix analysis through Hunspell continuation classes
Suffix rules now carry the continuation flags declared on their affix
text, and analysis undoes a stacked pair when the inner rule's classes
allow the outer one, so derived-then-inflected forms reduce to their
dictionary word.
(cherry picked from commit b543dea24c5241d5f3431b34be987f6e83fe887e)
* OPENNLP-1893: Usage, threading, and malformed-input tests for the Hunspell engine, precise javadoc
* OPENNLP-1893: Document Hunspell dictionary acquisition with a license-preserving download helper
* OPENNLP-1893: Cut morphology like hunspell does, accept UTF-8 flags and strip-only rules, and read the parser through the whitespace seam
* OPENNLP-1893: Read flags as code points, tolerate trailing morphology, and stem nothing from nothing
Loading the Spanish dictionary of the LibreOffice collection, the same
collection this module's README recommends, exposed three gaps against real
data. Flags under FLAG UTF-8 are now one code point each instead of one UTF-16
unit, since that dictionary names prefix rules with supplementary characters
that would otherwise split into two flags and abort the load; a variation
selector after a flag character selects presentation, not identity, and is
dropped, which the same file also relies on. A numeric or long flag run ends at
the first space or tabulator, the separators the word-list format defines, so
trailing morphological text without a tag no longer aborts the load; the
morphology cut itself now splits on exactly those two separators, the set the
reference implementation's hashmgr.cxx uses, which the javadoc previously
claimed while scanning wider whitespace. Stemming the empty word answers the
empty word instead of letting a strip-only rule conjure a stem from nothing.
All four downloaded dictionaries of the collection, English, Spanish,
Hungarian, and German, now load and stem; new tests pin the escaped slash, the
multi-word entry with trailing tags, and each corrected behavior.
* OPENNLP-1893: Resolve numeric dictionary flags through the AF alias table
The published Hungarian dictionary flags all of its entries as numeric
references into an AF alias table, so without alias support every entry loaded
flagless and stemming answered the surface form unchanged. The affix parser now
reads the AF table, the first line as the declared count and every further line
as one flag run with trailing comments discarded, and a purely numeric flag
field in the word list resolves as a 1-based reference into it, failing loud
with the line and table size when the reference is out of range. Without an AF
table numeric fields keep their FLAG num meaning. The Hungarian dictionary of
the LibreOffice collection now stems inflected forms; remaining gaps there are
compound territory, which is tracked separately.
* OPENNLP-1893: Walk only the affix rules that can apply, bucketed by their boundary character
Undoing a suffix requires the word to end with the rule's affix material, so
only rules whose material ends in the word's last character can ever apply,
and likewise for prefixes and the first character. The dictionary now buckets
its rules by that boundary character at load, and every scan in the stem path,
including the twofold and cross-product inner scans, walks the one bucket plus
the strip-only rules instead of the whole inventory. Measured on the
LibreOffice dictionaries at 4,000 words each: English 553k to 1,024k words per
second, Spanish 9.6k to 28.9k, German 132k to 287k.
* OPENNLP-1893: Decompose unanalyzed words into two flagged compound parts
When the affix analysis finds nothing and the affix file declares compounding,
a word now splits into two listed parts that the COMPOUNDFLAG or the positional
COMPOUNDBEGIN and COMPOUNDEND flags allow in their positions, honoring
COMPOUNDMIN, with the parts reported left to right. Affix analyses keep
precedence, listed words never decompose, and unflagged parts block a split.
Against the published Hungarian dictionary the unlisted kutyahaz decomposes
into its two nouns while listed compounds and inflected forms keep their
regular analyses. Longer chains, syllable rules, and the compound-only flags
stay unimplemented and simply leave such words unanalyzed.
* OPENNLP-1893: Honor the blocking flags, circumfixes, and compound positioning
A NEEDAFFIX (or PSEUDOROOT) entry is a virtual stem that exists only to
be affixed, an ONLYINCOMPOUND entry appears only inside compounds, and a
FORBIDDENWORD entry is listed to be blocked; none of them is a standalone
analysis anymore, per homonym flag set, and an affix carrying NEEDAFFIX
among its continuation classes yields no single-removal analysis while
its twofold and cross-product removals stand, the other affix being
exactly the further one required. A cross-product now also requires both
removed affixes' flags in the same homonym's flag set, and CIRCUMFIX
binds marked prefix and suffix halves to one another, so neither half
analyzes alone and a marked half never combines with an unmarked affix.
Decomposition grows from two verbatim parts to the compound machinery
the published German dictionary actually uses: any number of parts under
the positional COMPOUNDBEGIN/COMPOUNDMIDDLE/COMPOUNDEND flags and
COMPOUNDWORDMAX, parts standing on an entry plus one affix with
COMPOUNDPERMITFLAG required at internal boundaries and COMPOUNDFORBIDFLAG
barring marked forms, zero and dash linking suffixes included, an
uppercased retry for capitalized entries spelled lowercase inside a
compound, and the CHECKCOMPOUNDDUP, CHECKCOMPOUNDCASE, and
CHECKCOMPOUNDTRIPLE junction guards, case judged against the original
surface. A listed forbidden word never decomposes, and a fixed
part-licensing budget keeps adversarial input bounded, missing analyses
rather than stalling. Abbildungsverzeichnis, Haustuer, and Kinderzimmer
now decompose against de_DE_frami at 137k words/s single-threaded.
An opt-in test class checks everyday morphology against downloaded
dictionaries under -Dopennlp.hunspell.dict.dir; nothing is bundled.
* OPENNLP-1893: Add hunspell manual coverage with mirror-tested examples
Extend docbkx/stemmer.xml with the hunspell affix-stemmer section, wire the
chapter into the manual, and add StemmerFactoryUsageExampleTest and
HunspellManualExampleTest asserting the load-and-stem values the chapter prints.
Point the dictionary README at the new manual example.
* OPENNLP-1893: Apply the review-convention pass: factual license prose, split null contracts, thread-safety annotations
* OPENNLP-1893: Add {@inheritDoc} to the stemmer overrides and trim empty javadoc lines
* OPENNLP-1893: Address review: fold the affix twins, extract tags, complete javadoc
- Fold the two per-kind bucketing loops in the HunspellDictionary constructor into a
single bucketByBoundary helper that takes the rule list, the kind, and the sink for
the rules with empty affix material.
- Fold collectSuffixedPartStem and collectPrefixedPartStem, which differed only in the
boundary they face, into one collectAffixedPartStem with a suffix marker and an
atEdge marker; document what atEdge means at each end.
- Extract a parseValue helper for the single-integer directives so COMPOUNDMIN and
COMPOUNDWORDMAX no longer share one case body that re-tests which directive it is.
- Extract PREFIX_TAG, SUFFIX_TAG, and NO_MATERIAL constants and use them at the affix
block header, the rule lines, and the strip and affix material checks.
- Give FORBIDDENWORD its own case in the flag directive switch instead of letting the
catch-all default assign it, and make that default throw for a directive listed on
the outer switch but not handled on the inner one.
- Add the missing javadoc on the AffixCondition and HunspellDictionary constructors,
the Affix record components, and the splitLines, splitOn, and split helpers.
- Convert the single-line accessor javadoc on the compounding and affix bucket getters
to the {@return ...} form, and replace the hand-written prose on
HunspellStemmerFactory.newStemmer with {@inheritDoc} plus the instancing note.
- Trim commentary that restates the code: the bucketing rationale duplicated in
HunspellStemmer, the LibreOffice Spanish anecdote on the code point flag reader, and
the sentence left dangling in testGermanCompoundsDecompose.
- Drop the defensive null and directory guards from the test helpers
writeAndLoadFixture and load, which no caller can trip, and document what the real
dictionary tests assert.
- Fold the repeated ByteArrayInputStream plumbing in HunspellStemmerTest into two load
overloads, one UTF-8 and one taking the charset the SET declaration test needs.
- Turn the four table-style stemming tests into parameterized tests over their word and
expected stem pairs, so a failing row names itself.
- Add testNullArgumentsAreRejected, pinning the exact IllegalArgumentException message
of every public entry point including the argument names the stream loader reports.
- Correct the stemmer manual: name the example files after the fixture the test loads
rather than en_US, and state that the printed stems are the fixture's, since which
stem a published dictionary yields is that dictionary's decision.
* OPENNLP-1893: Fail loud on result-altering unsupported affix directives
Reject ICONV, OCONV, and COMPLEXPREFIXES at load time; keep skipping
cosmetic tables such as REP. Copy lookup results defensively and document
the compound search budget on HunspellStemmer.
* OPENNLP-1893: Bound stream size and match affix conditions by code point
Reject affix and dictionary streams above MAX_STREAM_BYTES. Affix
conditions and boundary bucketing use Unicode code points so
supplementary characters agree with FLAG UTF-8. Document the ceiling
in the stemmer chapter and pin both behaviors in tests.
* OPENNLP-1893: Verify dictionary downloads by SHA-512 and add an opt-in URL catalog
* OPENNLP-1893: Sync shared DownloadUtil with the startup-overridable download ceiling
The 512 MiB download ceiling becomes a default that
opennlp.download.max.bytes can raise at JVM startup; absent or invalid
values fall back. Keeps the file identical to the copy in the MeCab PR.
* OPENNLP-1893: Trigger CI for the DownloadUtil sync commit
* OPENNLP-1893: Address review: unbox the boundary lookups and publish the stream ceiling
The affix boundary buckets move from a boxed Integer map to a sorted
int index answered by binary search, closing the per-call boxing note
from the review; the strip-only rule lists are frozen at load so no
internal mutable list is handed out. MAX_STREAM_BYTES becomes public,
matching its citation in the load javadoc, the manual chapter, and the
dev README. A parameterized test pins trimming of word-list lines edged
by the no-break space and the ideographic space on both edges, and the
download and catalog test classes get the javadoc coverage the hunspell
tests already carry.
* OPENNLP-1893: Use a numeric character reference for the no-break space in the manual
DocBook XML defines no nbsp entity, so the PDF build rejects it.
* OPENNLP-1893: Reconcile the shared download test files with the sibling PR
Both PRs carry byte-identical copies; this folds the sibling's parameterized
invalid-limit test into this side's documented fixtures so the copies match again.
* OPENNLP-1893: Expose silent COMPOUNDRULE, IGNORE, KEEPCASE and ungated full-strip rules with failing tests
COMPOUNDRULE, IGNORE, and KEEPCASE alter analyses when ignored, so the
fail-closed loader policy requires them to fail at load time like ICONV,
OCONV, and COMPLEXPREFIXES; the loader currently accepts them silently.
A suffix rule whose strip string is the whole stem is applied without the
FULLSTRIP declaration hunspell requires for it, inventing a stem for a
surface form the dictionary does not license.
* OPENNLP-1893: Fail loud on COMPOUNDRULE, IGNORE, KEEPCASE and gate full-strip rules behind FULLSTRIP
COMPOUNDRULE licenses pattern compounds, IGNORE drops characters before
matching, and KEEPCASE forbids the capitalized variants this stemmer
analyzes through lowercasing, so ignoring any of them would change stems
with no signal; they now join ICONV, OCONV, and COMPLEXPREFIXES in the
load-time rejection. An affix rule whose strip string consumes the whole
stem is now undone only when the affix file declares FULLSTRIP; without
the declaration the rule is skipped at match time, which is what hunspell
itself does rather than rejecting the file. The manual and the dictionary
README follow the loader.
* OPENNLP-1893: Address Hunspell review feedback
Add named format constants, lower the download default to 64 MiB, and require applications to supply URL catalogs.
Red: the focused test compile failed for missing suffix constants and the caller-supplied catalog overload before the implementation was added.
---
dev/README-hunspell-dictionaries.md | 89 ++
.../tools/stemmer/hunspell/AffixCondition.java | 165 +++
.../tools/stemmer/hunspell/HunspellDictionary.java | 1272 ++++++++++++++++++
.../hunspell/HunspellDictionaryDownload.java | 93 ++
.../tools/stemmer/hunspell/HunspellStemmer.java | 596 +++++++++
.../stemmer/hunspell/HunspellStemmerFactory.java | 60 +
.../java/opennlp/tools/util/DictionaryCatalog.java | 149 +++
.../main/java/opennlp/tools/util/DownloadUtil.java | 206 ++-
.../hunspell/HunspellDictionaryDownloadTest.java | 148 +++
.../hunspell/HunspellManualExampleTest.java | 73 +
.../hunspell/HunspellRealDictionaryTest.java | 126 ++
.../hunspell/HunspellStemmerFactoryTest.java | 183 +++
.../stemmer/hunspell/HunspellStemmerTest.java | 1400 ++++++++++++++++++++
.../opennlp/tools/util/DictionaryCatalogTest.java | 154 +++
.../java/opennlp/tools/util/DigestTestUtil.java | 45 +
.../opennlp/tools/util/DownloadUtilFileTest.java | 177 +++
.../tools/util/dictionary-catalog.properties | 57 +
opennlp-docs/src/docbkx/stemmer.xml | 38 +
18 files changed, 5030 insertions(+), 1 deletion(-)
diff --git a/dev/README-hunspell-dictionaries.md b/dev/README-hunspell-dictionaries.md
new file mode 100644
index 000000000..0adda2250
--- /dev/null
+++ b/dev/README-hunspell-dictionaries.md
@@ -0,0 +1,89 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+# Hunspell dictionaries for the affix stemmer
+
+The Hunspell stemmer (`opennlp.tools.stemmer.hunspell`) implements the documented Hunspell dictionary format: a `.dic` word list plus its `.aff` affix companion, both supplied by the user. Apache OpenNLP bundles no dictionary data; whichever dictionary you download, its license is stated in the readme shipped alongside it.
+
+## Where dictionaries come from
+
+The LibreOffice project maintains a large collection of Hunspell dictionaries, one directory per language, at `github.com/LibreOffice/dictionaries`. Licenses differ per dictionary, which is why nothing is bundled: for example, the `en_US` dictionary derives from SCOWL and states its terms in `README_en_US.txt` in the same directory. Many other sources work too; the engine only cares that the pair follows the Hunspell format.
+
+OpenNLP does not ship a URL catalog. Applications that manage downloads can keep a
+properties file with an entry id followed by `.url`, `.sha512`, and optionally
+`.filename` keys. Pin each URL to a stable release or commit.
+
+## Option A: application catalog
+
+Catalog downloads stay inactive until you set `-Dopennlp.download.remote=true`.
+
+```java
+import java.io.InputStream;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import opennlp.tools.stemmer.hunspell.HunspellDictionaryDownload;
+import opennlp.tools.util.DictionaryCatalog;
+
+// JVM flag: -Dopennlp.download.remote=true
+try (InputStream in = Files.newInputStream(Path.of("dictionary-catalog.properties"))) {
+ DictionaryCatalog catalog = DictionaryCatalog.load(in);
+ HunspellDictionaryDownload.downloadFromCatalog(
+ catalog, "en_US", Path.of("/tmp/hunspell-en_US"));
+}
+```
+
+For `en_US`, the catalog ids are `hunspell.en_US.aff`, `hunspell.en_US.dic`, and
+optionally `hunspell.en_US.readme`. A complete catalog example lives at
+`opennlp-core/opennlp-runtime/src/test/resources/opennlp/tools/util/dictionary-catalog.properties`.
+The download test uses local file URLs to exercise this flow without network access.
+
+## Option B: your own files
+
+Fetch `.aff` / `.dic` (and the license readme) with any tool, or with
+`DownloadUtil.download(uri, path, sha512)`, then load them:
+
+```java
+import java.nio.file.Path;
+import opennlp.tools.stemmer.Stemmer;
+import opennlp.tools.stemmer.hunspell.HunspellDictionary;
+import opennlp.tools.stemmer.hunspell.HunspellStemmerFactory;
+
+HunspellDictionary dictionary = HunspellDictionary.load(
+ Path.of("/tmp/hunspell-en_US/en_US.aff"),
+ Path.of("/tmp/hunspell-en_US/en_US.dic"));
+HunspellStemmerFactory factory = new HunspellStemmerFactory(dictionary);
+
+Stemmer stemmer = factory.newStemmer();
+CharSequence stem = stemmer.stem("workers");
+```
+
+What `stem` evaluates to is decided by the dictionary you loaded, and this project ships no dictionary data, so no result is claimed here for `en_US`. The same load-and-stem flow is pinned by `HunspellManualExampleTest` (miniature in-memory dictionary, asserted stems for `workers` and `worker`) and by `HunspellStemmerFactoryTest#testEndToEndUsageFromFiles` (the same pair written to disk). The developer manual chapter `stemmer.xml` cites `HunspellManualExampleTest`.
+
+The dictionary is immutable and safe to share between threads; the factory hands out a fresh stemmer per call, so each thread takes its own from `newStemmer()`. A dictionary that declares a non-UTF-8 encoding through the `SET` directive in its `.aff` file is decoded accordingly; nothing needs converting beforehand.
+
+## Testing against real dictionaries
+
+The in-tree tests run against project-authored fixtures only. An opt-in test class, `HunspellRealDictionaryTest`, additionally checks everyday morphology against published dictionaries when pointed at a directory of `<name>.aff`/`<name>.dic` pairs (each test skips when its pair is absent):
+
+```
+./mvnw test -pl opennlp-core/opennlp-runtime -Dtest=HunspellRealDictionaryTest \
+ -Dopennlp.hunspell.dict.dir=/tmp/hunspell-dicts
+```
+
+## What the engine supports
+
+Supported affix features: `PFX` and `SFX` rules with strip strings, character-class conditions, cross-product combination of one prefix with one suffix, twofold suffixes through continuation classes, `FLAG` modes `char`, `UTF-8`, `long`, and `num`, the `AF` flag alias table, the `SET` encoding declaration, compound decomposition under `COMPOUNDFLAG`, the positional `COMPOUNDBEGIN`/`COMPOUNDMIDDLE`/`COMPOUNDEND` flags, `COMPOUNDMIN`, `COMPOUNDWORDMAX`, `COMPOUNDPERMITFLAG`, `COMPOUNDFORBI [...]
diff --git a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/hunspell/AffixCondition.java b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/hunspell/AffixCondition.java
new file mode 100644
index 000000000..19ce657cc
--- /dev/null
+++ b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/hunspell/AffixCondition.java
@@ -0,0 +1,165 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package opennlp.tools.stemmer.hunspell;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+
+/**
+ * One parsed affix condition: a fixed-length sequence of literal code points and
+ * bracketed character classes, matched with a single scan and no regular expressions.
+ * A suffix condition anchors at the end of the candidate stem, a prefix condition at
+ * its start; the condition {@code .} matches everything. Positions are Unicode code
+ * points so supplementary characters agree with {@code FLAG UTF-8} flag reading.
+ */
+final class AffixCondition {
+
+ /** The shared instance for the condition {@code .}, which accepts every stem. */
+ private static final AffixCondition ANY = new AffixCondition(new int[0][], null, true);
+
+ /** Per position: the accepted code points, or {@code null} for any code point. */
+ private final int[][] accepted;
+ /** Per position with a class: whether the class is negated; {@code null} rows unused. */
+ private final boolean[] negated;
+ /** Whether the owning rule is a suffix rule, which anchors the condition at the end. */
+ private final boolean suffix;
+
+ /**
+ * Initializes the condition.
+ *
+ * @param accepted The accepted code points per position.
+ * @param negated The negation marker per position.
+ * @param suffix Whether the owning rule is a suffix rule.
+ */
+ private AffixCondition(int[][] accepted, boolean[] negated, boolean suffix) {
+ this.accepted = accepted;
+ this.negated = negated;
+ this.suffix = suffix;
+ }
+
+ /**
+ * Parses a condition field. Each pattern position is a literal code point, a
+ * {@code .} matching any code point, or a bracketed class such as {@code [sx]}; a
+ * class starting with {@code ^} is negated and matches any code point outside it.
+ *
+ * @param pattern The condition text from the affix rule.
+ * @param suffix Whether the owning rule is a suffix rule.
+ * @param lineNumber The affix file line, for error messages.
+ * @return The parsed condition. Never {@code null}.
+ * @throws IOException Thrown if a character class is unterminated.
+ */
+ static AffixCondition parse(String pattern, boolean suffix, int lineNumber)
+ throws IOException {
+ if (".".equals(pattern)) {
+ return ANY;
+ }
+ final List<int[]> positions = new ArrayList<>();
+ final List<Boolean> negations = new ArrayList<>();
+ int i = 0;
+ while (i < pattern.length()) {
+ final int codePoint = pattern.codePointAt(i);
+ if (codePoint == '[') {
+ final int end = pattern.indexOf(']', i + 1);
+ if (end < 0) {
+ throw new IOException("unterminated character class at line " + lineNumber);
+ }
+ String members = pattern.substring(i + 1, end);
+ boolean negate = false;
+ if (members.startsWith("^")) {
+ negate = true;
+ members = members.substring(1);
+ }
+ positions.add(toCodePoints(members));
+ negations.add(negate);
+ i = end + 1;
+ } else if (codePoint == '.') {
+ positions.add(null);
+ negations.add(false);
+ i++;
+ } else {
+ positions.add(new int[] {codePoint});
+ negations.add(false);
+ i += Character.charCount(codePoint);
+ }
+ }
+ final int[][] accepted = positions.toArray(new int[0][]);
+ final boolean[] negated = new boolean[accepted.length];
+ for (int p = 0; p < negated.length; p++) {
+ negated[p] = negations.get(p);
+ }
+ return new AffixCondition(accepted, negated, suffix);
+ }
+
+ /**
+ * Collects the code points of a character-class body.
+ *
+ * @param members The class body text.
+ * @return The code points in order. Never {@code null}.
+ */
+ private static int[] toCodePoints(String members) {
+ final int[] codePoints = new int[members.codePointCount(0, members.length())];
+ int i = 0;
+ int out = 0;
+ while (i < members.length()) {
+ final int codePoint = members.codePointAt(i);
+ codePoints[out++] = codePoint;
+ i += Character.charCount(codePoint);
+ }
+ return codePoints;
+ }
+
+ /**
+ * Tests a candidate stem against the condition at its anchored side: the last
+ * positions of the stem for a suffix condition, the first positions for a prefix
+ * condition. A stem shorter than the condition never matches. Length is in code
+ * points.
+ *
+ * @param stem The candidate stem after affix removal and strip restoration.
+ * @return {@code true} if the stem satisfies the condition.
+ */
+ boolean matches(String stem) {
+ if (accepted.length == 0) {
+ return true;
+ }
+ final int stemPoints = stem.codePointCount(0, stem.length());
+ if (stemPoints < accepted.length) {
+ return false;
+ }
+ int offset = suffix ? stem.offsetByCodePoints(0, stemPoints - accepted.length) : 0;
+ for (int p = 0; p < accepted.length; p++) {
+ final int[] members = accepted[p];
+ final int codePoint = stem.codePointAt(offset);
+ offset += Character.charCount(codePoint);
+ if (members == null) {
+ continue;
+ }
+ boolean member = false;
+ for (final int candidate : members) {
+ if (candidate == codePoint) {
+ member = true;
+ break;
+ }
+ }
+ if (member == negated[p]) {
+ return false;
+ }
+ }
+ return true;
+ }
+}
diff --git a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/hunspell/HunspellDictionary.java b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/hunspell/HunspellDictionary.java
new file mode 100644
index 000000000..ab24a6885
--- /dev/null
+++ b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/hunspell/HunspellDictionary.java
@@ -0,0 +1,1272 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package opennlp.tools.stemmer.hunspell;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.charset.Charset;
+import java.nio.charset.IllegalCharsetNameException;
+import java.nio.charset.StandardCharsets;
+import java.nio.charset.UnsupportedCharsetException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import opennlp.tools.commons.ThreadSafe;
+import opennlp.tools.util.StringUtil;
+
+/**
+ * An immutable, in-memory Hunspell-format dictionary: the word list of a {@code .dic}
+ * file and the prefix and suffix rules of its {@code .aff} companion, loaded from
+ * user-supplied files. The engine implements the documented format directly; no
+ * dictionary data is bundled, dictionaries are supplied by the user.
+ *
+ * <p>Supported affix features: {@code PFX} and {@code SFX} rules with strip strings,
+ * character-class conditions, and cross-product combination of one prefix with one
+ * suffix; twofold suffixes through the continuation classes on suffix rules;
+ * {@code FLAG} modes {@code char} (default), {@code UTF-8}, {@code long}, and
+ * {@code num}; the {@code AF} flag alias table; the {@code SET} encoding declaration;
+ * compound decomposition under {@code COMPOUNDFLAG}, the positional
+ * {@code COMPOUNDBEGIN}/{@code COMPOUNDMIDDLE}/{@code COMPOUNDEND} flags,
+ * {@code COMPOUNDMIN}, {@code COMPOUNDWORDMAX}, {@code COMPOUNDPERMITFLAG},
+ * {@code COMPOUNDFORBIDFLAG}, and the {@code CHECKCOMPOUNDDUP},
+ * {@code CHECKCOMPOUNDCASE}, and {@code CHECKCOMPOUNDTRIPLE} declarations, with
+ * compound parts standing on their entries alone or on an entry plus one affix; the
+ * blocking flags
+ * {@code NEEDAFFIX} (with its historical alias {@code PSEUDOROOT}),
+ * {@code ONLYINCOMPOUND}, and {@code FORBIDDENWORD}, which suppress analyses the
+ * dictionary marks as virtual stems, compound-only parts, or forbidden words; and
+ * {@code CIRCUMFIX}, which binds marked prefix and suffix halves to one another; and
+ * the {@code FULLSTRIP} declaration, without which a rule that strips a whole stem is
+ * not applied, matching hunspell.
+ * Directives that would change stems when ignored ({@code ICONV}, {@code OCONV},
+ * {@code COMPLEXPREFIXES}, {@code COMPOUNDRULE}, {@code IGNORE},
+ * {@code KEEPCASE}) are rejected at load time. Cosmetic tables such as
+ * {@code REP}, {@code MAP}, and {@code KEY} are skipped, so analyses that would need
+ * them are missed rather than invented.</p>
+ *
+ * <p>Instances are immutable and safe to share between threads.</p>
+ *
+ * @see HunspellStemmer
+ * @see HunspellStemmerFactory
+ * @since 3.0.0
+ */
+@ThreadSafe
+public final class HunspellDictionary {
+
+ /** The file suffix for a Hunspell affix file. */
+ public static final String AFFIX_FILE_SUFFIX = ".aff";
+
+ /** The file suffix for a Hunspell word-list file. */
+ public static final String DICTIONARY_FILE_SUFFIX = ".dic";
+
+ /**
+ * Inclusive upper bound on bytes buffered from one affix or dictionary stream
+ * during {@link #load(InputStream, InputStream)}. The limit is 64 MiB. Larger
+ * streams fail with {@link IOException}.
+ */
+ public static final int MAX_STREAM_BYTES = 64 * 1024 * 1024;
+
+ /**
+ * One parsed affix rule of a {@code PFX} or {@code SFX} block.
+ *
+ * @param flag The flag naming the rule's block, which an entry carries to accept it.
+ * @param crossProduct Whether the rule may combine with an affix of the opposite kind.
+ * @param strip The stem material the rule replaces, restored during analysis.
+ * @param affix The surface material the rule adds to the stem.
+ * @param condition The condition the stem must satisfy for the rule to apply.
+ * @param continuation The flags of the further affixes that may stack on this one.
+ */
+ record Affix(int flag, boolean crossProduct, String strip, String affix,
+ AffixCondition condition, int[] continuation) {
+
+ /**
+ * Checks whether a further affix may stack on this one.
+ *
+ * @param otherFlag The stacking affix's flag.
+ * @return {@code true} if this affix's continuation classes allow it.
+ */
+ boolean allowsContinuation(int otherFlag) {
+ for (final int candidate : continuation) {
+ if (candidate == otherFlag) {
+ return true;
+ }
+ }
+ return false;
+ }
+ }
+
+ /** The place a part takes in a compound, deciding which positional flag admits it. */
+ enum CompoundPosition {
+ /** The first part. */
+ BEGIN,
+ /** Any part between the first and the last. */
+ MIDDLE,
+ /** The last part. */
+ END
+ }
+
+ /** The shared empty bucket answered for characters no affix rule is keyed under. */
+ private static final List<Affix> NO_AFFIXES = List.of();
+
+ /** The line tag of a prefix block and of every rule line inside it. */
+ private static final String PREFIX_TAG = "PFX";
+
+ /** The line tag of a suffix block and of every rule line inside it. */
+ private static final String SUFFIX_TAG = "SFX";
+
+ /** Prefix used by comment lines. */
+ private static final String COMMENT_PREFIX = "#";
+
+ /** The affix-file directive that declares the character encoding. */
+ private static final String SET_TAG = "SET";
+
+ /** The {@code SET} directive followed by a space. */
+ private static final String SET_PREFIX = SET_TAG + " ";
+
+ /** The {@code SET} directive followed by a tab. */
+ private static final String SET_TAB_PREFIX = SET_TAG + "\t";
+
+ /** The affix format's marker for absent strip or affix material. */
+ private static final String NO_MATERIAL = "0";
+
+ private final Map<String, List<int[]>> entries;
+ private final BoundaryIndex suffixesByLast;
+ private final List<Affix> suffixesWithoutMaterial;
+ private final BoundaryIndex prefixesByFirst;
+ private final List<Affix> prefixesWithoutMaterial;
+ private final int compoundFlag;
+ private final int compoundBegin;
+ private final int compoundEnd;
+ private final int compoundMin;
+ private final int needAffix;
+ private final int onlyInCompound;
+ private final int forbiddenWord;
+ private final int circumfix;
+ private final int compoundMiddle;
+ private final int compoundPermit;
+ private final int compoundForbid;
+ private final int compoundWordMax;
+ private final boolean checkCompoundDup;
+ private final boolean checkCompoundCase;
+ private final boolean checkCompoundTriple;
+ private final boolean fullStrip;
+
+ /**
+ * Initializes the dictionary from the two parsed files.
+ *
+ * @param entries The words mapped to the flag sets of their entries.
+ * @param affix The parsed affix file.
+ */
+ private HunspellDictionary(Map<String, List<int[]>> entries, AffixFile affix) {
+ this.compoundFlag = affix.compoundFlag;
+ this.compoundBegin = affix.compoundBegin;
+ this.compoundEnd = affix.compoundEnd;
+ this.compoundMin = affix.compoundMin;
+ this.needAffix = affix.needAffix;
+ this.onlyInCompound = affix.onlyInCompound;
+ this.forbiddenWord = affix.forbiddenWord;
+ this.circumfix = affix.circumfix;
+ this.compoundMiddle = affix.compoundMiddle;
+ this.compoundPermit = affix.compoundPermit;
+ this.compoundForbid = affix.compoundForbid;
+ this.compoundWordMax = affix.compoundWordMax;
+ this.checkCompoundDup = affix.checkCompoundDup;
+ this.checkCompoundCase = affix.checkCompoundCase;
+ this.checkCompoundTriple = affix.checkCompoundTriple;
+ this.fullStrip = affix.fullStrip;
+ this.entries = entries;
+ // A material-bearing rule can only be undone from a word whose boundary
+ // character matches its affix material, so bucketing by that character
+ // narrows each scan to one bucket plus the strip-only rules.
+ final List<Affix> suffixesWithout = new ArrayList<>();
+ this.suffixesByLast = bucketByBoundary(affix.suffixes, true, suffixesWithout);
+ this.suffixesWithoutMaterial = List.copyOf(suffixesWithout);
+ final List<Affix> prefixesWithout = new ArrayList<>();
+ this.prefixesByFirst = bucketByBoundary(affix.prefixes, false, prefixesWithout);
+ this.prefixesWithoutMaterial = List.copyOf(prefixesWithout);
+ }
+
+ /**
+ * An immutable index of affix rules keyed by the boundary code point of their affix
+ * material, answering each lookup by binary search so the per-word scans in
+ * {@link HunspellStemmer} allocate nothing.
+ */
+ private static final class BoundaryIndex {
+
+ /** The boundary code points, sorted ascending. */
+ private final int[] boundaries;
+ /** The rule bucket for each boundary, aligned with {@link #boundaries}. */
+ private final List<List<Affix>> buckets;
+
+ /**
+ * Initializes the index from mutable buckets, freezing each one.
+ *
+ * @param byBoundary The rule buckets keyed by boundary code point.
+ */
+ private BoundaryIndex(Map<Integer, List<Affix>> byBoundary) {
+ this.boundaries = new int[byBoundary.size()];
+ int b = 0;
+ for (final Integer boundary : byBoundary.keySet()) {
+ boundaries[b++] = boundary;
+ }
+ Arrays.sort(boundaries);
+ this.buckets = new ArrayList<>(boundaries.length);
+ for (final int boundary : boundaries) {
+ buckets.add(List.copyOf(byBoundary.get(boundary)));
+ }
+ }
+
+ /**
+ * The rules bucketed under a boundary code point.
+ *
+ * @param codePoint The boundary code point to look up.
+ * @return The bucket, possibly empty. Never {@code null}.
+ */
+ List<Affix> bucket(int codePoint) {
+ final int index = Arrays.binarySearch(boundaries, codePoint);
+ return index >= 0 ? buckets.get(index) : NO_AFFIXES;
+ }
+ }
+
+ /**
+ * Buckets affix rules by the boundary code point of their affix material, the last
+ * code point for a suffix rule and the first for a prefix rule.
+ *
+ * @param rules The rules of one kind, in file order.
+ * @param suffix Whether the rules are suffix rules.
+ * @param withoutMaterial Collects the rules with empty affix material, which no
+ * boundary code point keys.
+ * @return The rules indexed by their boundary code point. Never {@code null}.
+ */
+ private static BoundaryIndex bucketByBoundary(List<Affix> rules,
+ boolean suffix, List<Affix> withoutMaterial) {
+ final Map<Integer, List<Affix>> byBoundary = new HashMap<>();
+ for (final Affix rule : rules) {
+ final String material = rule.affix();
+ if (material.isEmpty()) {
+ withoutMaterial.add(rule);
+ } else {
+ final int boundary = suffix
+ ? material.codePointBefore(material.length())
+ : material.codePointAt(0);
+ byBoundary.computeIfAbsent(boundary, key -> new ArrayList<>()).add(rule);
+ }
+ }
+ return new BoundaryIndex(byBoundary);
+ }
+
+ /**
+ * Loads a dictionary from its two files.
+ *
+ * @param affixFile The {@code .aff} affix file. Must not be {@code null}.
+ * @param dictionaryFile The {@code .dic} word list. Must not be {@code null}.
+ * @return The loaded dictionary. Never {@code null}.
+ * @throws IOException Thrown if reading fails or a file is malformed.
+ * @throws IllegalArgumentException Thrown if a parameter is {@code null}.
+ */
+ public static HunspellDictionary load(Path affixFile, Path dictionaryFile)
+ throws IOException {
+ if (affixFile == null) {
+ throw new IllegalArgumentException("affixFile must not be null");
+ }
+ if (dictionaryFile == null) {
+ throw new IllegalArgumentException("dictionaryFile must not be null");
+ }
+ try (InputStream affix = Files.newInputStream(affixFile);
+ InputStream dictionary = Files.newInputStream(dictionaryFile)) {
+ return load(affix, dictionary);
+ }
+ }
+
+ /**
+ * Loads a dictionary from its two streams. Each stream is buffered up to
+ * {@link #MAX_STREAM_BYTES} bytes; a larger stream fails with {@link IOException}.
+ *
+ * @param affixStream The {@code .aff} affix content. Must not be {@code null}. Not
+ * closed.
+ * @param dictionaryStream The {@code .dic} word list content. Must not be
+ * {@code null}. Not closed.
+ * @return The loaded dictionary. Never {@code null}.
+ * @throws IOException Thrown if reading fails, a stream exceeds
+ * {@link #MAX_STREAM_BYTES}, or the content is malformed.
+ * @throws IllegalArgumentException Thrown if a parameter is {@code null}.
+ */
+ public static HunspellDictionary load(InputStream affixStream,
+ InputStream dictionaryStream) throws IOException {
+ if (affixStream == null) {
+ throw new IllegalArgumentException("affixStream must not be null");
+ }
+ if (dictionaryStream == null) {
+ throw new IllegalArgumentException("dictionaryStream must not be null");
+ }
+ final byte[] affixBytes = readBounded(affixStream, MAX_STREAM_BYTES, "affix stream");
+ final Charset charset = declaredCharset(affixBytes);
+ final AffixFile affix = parseAffix(new String(affixBytes, charset));
+ final Map<String, List<int[]>> entries = parseWordList(
+ new String(readBounded(dictionaryStream, MAX_STREAM_BYTES, "dictionary stream"),
+ charset),
+ affix.flagMode, affix.flagAliases);
+ return new HunspellDictionary(entries, affix);
+ }
+
+ /**
+ * Reads an input stream into a byte array, failing when more than {@code maxBytes}
+ * arrive.
+ *
+ * @param in The stream to read. Not closed.
+ * @param maxBytes The inclusive upper bound on buffered bytes.
+ * @param label The stream name used in the error message.
+ * @return The buffered bytes. Never {@code null}.
+ * @throws IOException Thrown if reading fails or the stream exceeds {@code maxBytes}.
+ */
+ static byte[] readBounded(InputStream in, int maxBytes, String label)
+ throws IOException {
+ final byte[] chunk = new byte[8192];
+ byte[] buffer = new byte[Math.min(8192, maxBytes)];
+ int size = 0;
+ int n;
+ while ((n = in.read(chunk)) >= 0) {
+ if (size + n > maxBytes) {
+ throw new IOException(label + " size exceeds safe limit of " + maxBytes);
+ }
+ if (size + n > buffer.length) {
+ buffer = Arrays.copyOf(buffer, Math.min(maxBytes, Math.max(buffer.length * 2, size + n)));
+ }
+ System.arraycopy(chunk, 0, buffer, size, n);
+ size += n;
+ }
+ return size == buffer.length ? buffer : Arrays.copyOf(buffer, size);
+ }
+
+ /**
+ * Looks up a word's flag sets.
+ *
+ * @param word The word exactly as listed.
+ * @return The flag sets of all matching entries, or {@code null} when absent.
+ */
+ List<int[]> lookup(String word) {
+ final List<int[]> found = entries.get(word);
+ if (found == null) {
+ return null;
+ }
+ final List<int[]> copy = new ArrayList<>(found.size());
+ for (final int[] flags : found) {
+ copy.add(flags.clone());
+ }
+ return copy;
+ }
+
+ /**
+ * The suffix rules whose affix material ends in the given code point, which are the
+ * only material-bearing rules that can be undone from a word ending in it.
+ *
+ * @param last The word's last code point.
+ * @return The bucket, possibly empty. Never {@code null}.
+ */
+ List<Affix> suffixesEndingWith(int last) {
+ return suffixesByLast.bucket(last);
+ }
+
+ /** {@return the strip-only suffix rules, applicable to any word} Never {@code null}. */
+ List<Affix> suffixesWithoutMaterial() {
+ return suffixesWithoutMaterial;
+ }
+
+ /**
+ * The prefix rules whose affix material starts with the given code point, which are
+ * the only material-bearing rules that can be undone from a word starting with it.
+ *
+ * @param first The word's first code point.
+ * @return The bucket, possibly empty. Never {@code null}.
+ */
+ List<Affix> prefixesStartingWith(int first) {
+ return prefixesByFirst.bucket(first);
+ }
+
+ /** {@return the strip-only prefix rules, applicable to any word} Never {@code null}. */
+ List<Affix> prefixesWithoutMaterial() {
+ return prefixesWithoutMaterial;
+ }
+
+ /** {@return whether the affix file declares any compounding flag at all} */
+ boolean compoundsDeclared() {
+ return compoundFlag != 0 || compoundBegin != 0 || compoundEnd != 0
+ || compoundMiddle != 0;
+ }
+
+ /** {@return the smallest length a compound part may have} At least {@code 1}. */
+ int compoundMin() {
+ return compoundMin;
+ }
+
+ /** {@return the largest number of parts a compound may have} {@code 0} is unbounded. */
+ int compoundWordMax() {
+ return compoundWordMax;
+ }
+
+ /** {@return whether {@code CHECKCOMPOUNDDUP} forbids a part repeating its neighbor} */
+ boolean checkCompoundDup() {
+ return checkCompoundDup;
+ }
+
+ /** {@return whether {@code CHECKCOMPOUNDCASE} forbids uppercase at part boundaries} */
+ boolean checkCompoundCase() {
+ return checkCompoundCase;
+ }
+
+ /** {@return whether {@code CHECKCOMPOUNDTRIPLE} forbids triple letters at boundaries} */
+ boolean checkCompoundTriple() {
+ return checkCompoundTriple;
+ }
+
+ /** {@return whether {@code FULLSTRIP} allows an affix rule to strip a whole stem} */
+ boolean fullStrip() {
+ return fullStrip;
+ }
+
+ /**
+ * The flag admitting a part at a compound position, next to the general
+ * compounding flag.
+ *
+ * @param position The part's place in the compound.
+ * @return The dedicated positional flag, or {@code 0} when undeclared.
+ */
+ private int positionalFlag(CompoundPosition position) {
+ return switch (position) {
+ case BEGIN -> compoundBegin;
+ case MIDDLE -> compoundMiddle;
+ case END -> compoundEnd;
+ };
+ }
+
+ /**
+ * Checks whether a listed word may stand at a compound position: some homonym's
+ * flag set carries the general compounding flag or the position's dedicated flag
+ * and is not forbidden. A compound-only or virtual-stem homonym may take the
+ * position; that is what those flags permit.
+ *
+ * @param flagSets The word's flag sets from {@link #lookup(String)}.
+ * @param position The part's place in the compound.
+ * @return {@code true} if the word may stand at the position.
+ */
+ boolean mayStand(List<int[]> flagSets, CompoundPosition position) {
+ final int positional = positionalFlag(position);
+ for (final int[] flags : flagSets) {
+ if ((contains(flags, compoundFlag) || contains(flags, positional))
+ && !contains(flags, forbiddenWord) && !contains(flags, needAffix)) {
+ return true;
+ }
+ }
+ return false;
+ }
+
+ /**
+ * Checks whether some homonym supports an affixed compound part: its flag set
+ * carries the removed affix's flag, is not forbidden, and either the affix itself
+ * admits the position or the set carries the compounding or positional flag.
+ *
+ * @param flagSets The part stem's flag sets from {@link #lookup(String)}.
+ * @param affixFlag The removed affix's flag.
+ * @param position The part's place in the compound.
+ * @param affixAdmits Whether the affix's continuation classes admit the position,
+ * from {@link #affixAdmits(Affix, CompoundPosition)}.
+ * @return {@code true} if some homonym stands affixed at the position.
+ */
+ boolean supportsPart(List<int[]> flagSets, int affixFlag, CompoundPosition position,
+ boolean affixAdmits) {
+ final int positional = positionalFlag(position);
+ for (final int[] flags : flagSets) {
+ if (contains(flags, affixFlag) && !contains(flags, forbiddenWord)
+ && (affixAdmits || contains(flags, compoundFlag)
+ || contains(flags, positional))) {
+ return true;
+ }
+ }
+ return false;
+ }
+
+ /**
+ * Checks whether an affix admits its derived form at a compound position: its
+ * continuation classes carry the general compounding flag or the position's
+ * dedicated flag. Published dictionaries position their linking forms this way,
+ * through zero or dash suffixes whose continuation classes hold the positional
+ * flags.
+ *
+ * @param affix The affix rule applied to the part.
+ * @param position The part's place in the compound.
+ * @return {@code true} if the affixed form may stand at the position.
+ */
+ boolean affixAdmits(Affix affix, CompoundPosition position) {
+ return (compoundFlag != 0 && affix.allowsContinuation(compoundFlag))
+ || (positionalFlag(position) != 0
+ && affix.allowsContinuation(positionalFlag(position)));
+ }
+
+ /**
+ * Checks whether an affix may sit at a compound-internal boundary: it carries the
+ * {@code COMPOUNDPERMITFLAG} among its continuation classes. Without the flag a
+ * suffix fits only the last part and a prefix only the first.
+ *
+ * @param affix The affix rule applied to the part.
+ * @return {@code true} if the affix may face another part.
+ */
+ boolean permitsInside(Affix affix) {
+ return compoundPermit != 0 && affix.allowsContinuation(compoundPermit);
+ }
+
+ /**
+ * Checks whether an affix bars its derived form from compounds altogether: it
+ * carries the {@code COMPOUNDFORBIDFLAG} among its continuation classes.
+ *
+ * @param affix The affix rule applied to the part.
+ * @return {@code true} if the affixed form may not join a compound.
+ */
+ boolean forbidsInCompound(Affix affix) {
+ return compoundForbid != 0 && affix.allowsContinuation(compoundForbid);
+ }
+
+ /**
+ * Checks whether any of a word's flag sets is forbidden, which a dictionary uses
+ * to block one specific ill-formed compound while its parts stay productive.
+ *
+ * @param flagSets The word's flag sets from {@link #lookup(String)}.
+ * @return {@code true} if some homonym carries the forbidden-word flag.
+ */
+ boolean anyForbidden(List<int[]> flagSets) {
+ return hasFlag(flagSets, forbiddenWord);
+ }
+
+ /**
+ * Checks whether any of a word's flag sets carries a flag.
+ *
+ * @param flagSets The flag sets from {@link #lookup(String)}.
+ * @param flag The flag to look for.
+ * @return {@code true} if some flag set contains the flag.
+ */
+ static boolean hasFlag(List<int[]> flagSets, int flag) {
+ for (final int[] flags : flagSets) {
+ if (contains(flags, flag)) {
+ return true;
+ }
+ }
+ return false;
+ }
+
+ /**
+ * Checks one flag set for a flag. An undeclared flag, encoded as {@code 0}, is
+ * carried by no entry.
+ *
+ * @param flags One entry's flag set.
+ * @param flag The flag to look for.
+ * @return {@code true} if the set contains the flag.
+ */
+ private static boolean contains(int[] flags, int flag) {
+ if (flag == 0) {
+ return false;
+ }
+ for (final int candidate : flags) {
+ if (candidate == flag) {
+ return true;
+ }
+ }
+ return false;
+ }
+
+ /**
+ * Checks whether a listed word is valid on its own: some homonym's flag set carries
+ * none of the blocking flags. An entry whose every flag set is marked
+ * {@code NEEDAFFIX} is a virtual stem that exists only to be affixed, one marked
+ * {@code ONLYINCOMPOUND} appears only inside compounds, and one marked
+ * {@code FORBIDDENWORD} is listed to be blocked; none of them is a word by itself.
+ *
+ * @param flagSets The word's flag sets from {@link #lookup(String)}.
+ * @return {@code true} if some homonym stands on its own.
+ */
+ boolean validStandalone(List<int[]> flagSets) {
+ for (final int[] flags : flagSets) {
+ if (!contains(flags, needAffix) && !contains(flags, onlyInCompound)
+ && !contains(flags, forbiddenWord)) {
+ return true;
+ }
+ }
+ return false;
+ }
+
+ /**
+ * Checks whether some homonym supports an affix analysis: its flag set carries the
+ * affix's flag and is neither compound-only nor forbidden. A {@code NEEDAFFIX} set
+ * does support the analysis, because the removed affix is exactly what the virtual
+ * stem needs.
+ *
+ * @param flagSets The stem's flag sets from {@link #lookup(String)}.
+ * @param flag The removed affix's flag.
+ * @return {@code true} if some homonym carries the flag and may stand affixed.
+ */
+ boolean supports(List<int[]> flagSets, int flag) {
+ for (final int[] flags : flagSets) {
+ if (contains(flags, flag) && !contains(flags, onlyInCompound)
+ && !contains(flags, forbiddenWord)) {
+ return true;
+ }
+ }
+ return false;
+ }
+
+ /**
+ * Checks whether some homonym supports a cross-product analysis: one flag set
+ * carries both removed affixes' flags and is neither compound-only nor forbidden.
+ * The two flags must sit in the same set, because homonyms are separate words and
+ * each removal must be licensed by the same one.
+ *
+ * @param flagSets The stem's flag sets from {@link #lookup(String)}.
+ * @param prefixFlag The removed prefix's flag.
+ * @param suffixFlag The removed suffix's flag.
+ * @return {@code true} if some homonym carries both flags and may stand affixed.
+ */
+ boolean supports(List<int[]> flagSets, int prefixFlag, int suffixFlag) {
+ for (final int[] flags : flagSets) {
+ if (contains(flags, prefixFlag) && contains(flags, suffixFlag)
+ && !contains(flags, onlyInCompound) && !contains(flags, forbiddenWord)) {
+ return true;
+ }
+ }
+ return false;
+ }
+
+ /**
+ * Checks whether a form made with this affix alone is still a virtual stem: the
+ * affix carries the {@code NEEDAFFIX} flag among its continuation classes, so a
+ * further affix must join before the form is a word.
+ *
+ * @param affix The affix rule to inspect.
+ * @return {@code true} if the affix alone does not finish a word.
+ */
+ boolean needsFurtherAffix(Affix affix) {
+ return needAffix != 0 && affix.allowsContinuation(needAffix);
+ }
+
+ /**
+ * Checks whether an affix applies only inside compounds: it carries the
+ * {@code ONLYINCOMPOUND} flag among its continuation classes.
+ *
+ * @param affix The affix rule to inspect.
+ * @return {@code true} if the affix never applies to a standalone word.
+ */
+ boolean compoundOnly(Affix affix) {
+ return onlyInCompound != 0 && affix.allowsContinuation(onlyInCompound);
+ }
+
+ /**
+ * Checks whether an affix is one half of a circumfix: it carries the
+ * {@code CIRCUMFIX} flag among its continuation classes, so it is only valid on a
+ * word that also carries a circumfix-marked affix of the other kind, the German
+ * {@code ge...t} participle being the model.
+ *
+ * @param affix The affix rule to inspect.
+ * @return {@code true} if the affix never applies without its other half.
+ */
+ boolean circumfixOnly(Affix affix) {
+ return circumfix != 0 && affix.allowsContinuation(circumfix);
+ }
+
+ /**
+ * Finds the {@code SET} declaration by scanning the raw affix bytes as ASCII, which
+ * is safe because the declaration itself is ASCII in every supported encoding. Both
+ * files are then decoded with the declared charset.
+ *
+ * @param affixBytes The raw affix file content.
+ * @return The declared charset, or UTF-8 when no declaration is present.
+ * @throws IOException Thrown if the declared encoding name is not supported.
+ */
+ private static Charset declaredCharset(byte[] affixBytes) throws IOException {
+ final String ascii = new String(affixBytes, StandardCharsets.US_ASCII);
+ for (final String line : splitLines(ascii)) {
+ final String trimmed = trim(line);
+ if (trimmed.startsWith(SET_PREFIX) || trimmed.startsWith(SET_TAB_PREFIX)) {
+ final String name = trim(trimmed.substring(SET_PREFIX.length()));
+ try {
+ return Charset.forName(name);
+ } catch (IllegalCharsetNameException | UnsupportedCharsetException e) {
+ throw new IOException("unsupported SET encoding: " + name, e);
+ }
+ }
+ }
+ return StandardCharsets.UTF_8;
+ }
+
+ /** The flag encodings a dictionary may declare with the {@code FLAG} directive. */
+ private enum FlagMode {
+ /**
+ * The default: each single character is one flag. Also what {@code FLAG UTF-8}
+ * declares, which asks for single-character flags in a file the {@code SET}
+ * declaration already had decoded.
+ */
+ CHAR,
+ /** Declared as {@code FLAG long}: each pair of characters is one flag. */
+ LONG,
+ /** Declared as {@code FLAG num}: comma-separated decimal numbers are flags. */
+ NUM
+ }
+
+ /** The parsed affix file content. */
+ private static final class AffixFile {
+ private final List<Affix> prefixes = new ArrayList<>();
+ private final List<Affix> suffixes = new ArrayList<>();
+ private final List<int[]> flagAliases = new ArrayList<>();
+ private boolean aliasHeaderSeen;
+ private FlagMode flagMode = FlagMode.CHAR;
+ private int compoundFlag;
+ private int compoundBegin;
+ private int compoundEnd;
+ private int compoundMin = 3;
+ private int needAffix;
+ private int onlyInCompound;
+ private int forbiddenWord;
+ private int circumfix;
+ private int compoundMiddle;
+ private int compoundPermit;
+ private int compoundForbid;
+ private int compoundWordMax;
+ private boolean checkCompoundDup;
+ private boolean checkCompoundCase;
+ private boolean checkCompoundTriple;
+ private boolean fullStrip;
+ }
+
+ /**
+ * Parses the affix file: the {@code FLAG} declaration, the {@code AF} flag alias
+ * table, the compound and blocking flag declarations, and the {@code PFX} and
+ * {@code SFX} blocks. Result-altering unsupported directives fail loud;
+ * cosmetic ones are skipped.
+ *
+ * @param content The decoded affix file content.
+ * @return The parsed rules and flag mode. Never {@code null}.
+ * @throws IOException Thrown if a supported directive is malformed, or if
+ * {@code ICONV}, {@code OCONV}, {@code COMPLEXPREFIXES}, {@code COMPOUNDRULE},
+ * {@code IGNORE}, or {@code KEEPCASE} appears.
+ */
+ private static AffixFile parseAffix(String content) throws IOException {
+ final AffixFile result = new AffixFile();
+ final String[] lines = splitLines(content);
+ int i = 0;
+ while (i < lines.length) {
+ final String[] fields = split(lines[i]);
+ if (fields.length == 0 || fields[0].startsWith(COMMENT_PREFIX)) {
+ i++;
+ continue;
+ }
+ switch (fields[0]) {
+ case "FLAG":
+ if (fields.length < 2) {
+ throw new IOException("FLAG line without a mode at line " + (i + 1));
+ }
+ result.flagMode = switch (fields[1]) {
+ case "long" -> FlagMode.LONG;
+ case "num" -> FlagMode.NUM;
+ case "UTF-8" -> FlagMode.CHAR;
+ default -> throw new IOException(
+ "unsupported FLAG mode '" + fields[1] + "' at line " + (i + 1));
+ };
+ i++;
+ break;
+ case "COMPOUNDFLAG":
+ case "COMPOUNDBEGIN":
+ case "COMPOUNDMIDDLE":
+ case "COMPOUNDEND":
+ case "COMPOUNDPERMITFLAG":
+ case "COMPOUNDFORBIDFLAG":
+ case "NEEDAFFIX":
+ case "PSEUDOROOT":
+ case "ONLYINCOMPOUND":
+ case "FORBIDDENWORD":
+ case "CIRCUMFIX":
+ if (fields.length < 2) {
+ throw new IOException(fields[0] + " line without a flag at line " + (i + 1));
+ }
+ final int declared = parseFlag(fields[1], result.flagMode, i + 1);
+ switch (fields[0]) {
+ case "COMPOUNDFLAG" -> result.compoundFlag = declared;
+ case "COMPOUNDBEGIN" -> result.compoundBegin = declared;
+ case "COMPOUNDMIDDLE" -> result.compoundMiddle = declared;
+ case "COMPOUNDEND" -> result.compoundEnd = declared;
+ case "COMPOUNDPERMITFLAG" -> result.compoundPermit = declared;
+ case "COMPOUNDFORBIDFLAG" -> result.compoundForbid = declared;
+ // PSEUDOROOT is the directive's name before hunspell renamed it
+ case "NEEDAFFIX", "PSEUDOROOT" -> result.needAffix = declared;
+ case "ONLYINCOMPOUND" -> result.onlyInCompound = declared;
+ case "CIRCUMFIX" -> result.circumfix = declared;
+ case "FORBIDDENWORD" -> result.forbiddenWord = declared;
+ default -> throw new IOException(
+ "unhandled flag directive " + fields[0] + " at line " + (i + 1));
+ }
+ i++;
+ break;
+ case "COMPOUNDMIN":
+ result.compoundMin = Math.max(1, parseValue(fields, i + 1));
+ i++;
+ break;
+ case "COMPOUNDWORDMAX":
+ result.compoundWordMax = Math.max(0, parseValue(fields, i + 1));
+ i++;
+ break;
+ case "CHECKCOMPOUNDDUP":
+ result.checkCompoundDup = true;
+ i++;
+ break;
+ case "CHECKCOMPOUNDCASE":
+ result.checkCompoundCase = true;
+ i++;
+ break;
+ case "CHECKCOMPOUNDTRIPLE":
+ result.checkCompoundTriple = true;
+ i++;
+ break;
+ case "FULLSTRIP":
+ result.fullStrip = true;
+ i++;
+ break;
+ case "AF":
+ // the first AF line declares the alias count; every further AF line is one
+ // alias, a flag run whose 1-based position numeric dictionary flags refer to
+ if (fields.length >= 2) {
+ if (!result.aliasHeaderSeen) {
+ result.aliasHeaderSeen = true;
+ } else {
+ result.flagAliases.add(parseFlags(fields[1], result.flagMode, i + 1));
+ }
+ }
+ i++;
+ break;
+ case PREFIX_TAG:
+ case SUFFIX_TAG:
+ i = parseAffixBlock(lines, i, fields, result);
+ break;
+ case "ICONV":
+ case "OCONV":
+ case "COMPLEXPREFIXES":
+ // COMPOUNDRULE licenses pattern compounds, IGNORE drops characters before
+ // matching, and KEEPCASE forbids the case variants this stemmer analyzes;
+ // ignoring any of them would change stems with no signal
+ case "COMPOUNDRULE":
+ case "IGNORE":
+ case "KEEPCASE":
+ throw new IOException("unsupported affix directive '" + fields[0]
+ + "' at line " + (i + 1));
+ default:
+ i++;
+ break;
+ }
+ }
+ return result;
+ }
+
+ /**
+ * Parses the integer value of a directive that carries exactly one.
+ *
+ * @param fields The already-split directive line.
+ * @param lineNumber The source line, for error messages.
+ * @return The parsed value.
+ * @throws IOException Thrown if the value is missing or is not an integer.
+ */
+ private static int parseValue(String[] fields, int lineNumber) throws IOException {
+ if (fields.length < 2) {
+ throw new IOException(fields[0] + " line without a value at line " + lineNumber);
+ }
+ try {
+ return Integer.parseInt(fields[1]);
+ } catch (NumberFormatException e) {
+ throw new IOException("malformed " + fields[0] + " at line " + lineNumber, e);
+ }
+ }
+
+ /**
+ * Parses one {@code PFX} or {@code SFX} block: the header line naming the flag, the
+ * cross-product marker, and the rule count, followed by exactly that many rule
+ * lines.
+ *
+ * @param lines All lines of the affix file.
+ * @param index The line index of the block header.
+ * @param header The already-split header fields.
+ * @param result The parse target the rules are added to.
+ * @return The index of the first line after the block.
+ * @throws IOException Thrown if the header or a rule line is malformed.
+ */
+ private static int parseAffixBlock(String[] lines, int index, String[] header,
+ AffixFile result) throws IOException {
+ if (header.length < 4) {
+ throw new IOException("malformed affix header at line " + (index + 1));
+ }
+ final boolean suffix = SUFFIX_TAG.equals(header[0]);
+ final int flag = parseFlag(header[1], result.flagMode, index + 1);
+ final boolean crossProduct = "Y".equals(header[2]);
+ final int count;
+ try {
+ count = Integer.parseInt(header[3]);
+ } catch (NumberFormatException e) {
+ throw new IOException("malformed affix rule count at line " + (index + 1), e);
+ }
+ int line = index + 1;
+ for (int rule = 0; rule < count; rule++, line++) {
+ if (line >= lines.length) {
+ throw new IOException("affix block truncated at line " + (line + 1));
+ }
+ final String[] fields = split(lines[line]);
+ if (fields.length < 5 || !fields[0].equals(header[0])) {
+ throw new IOException("malformed affix rule at line " + (line + 1));
+ }
+ final String strip = NO_MATERIAL.equals(fields[2]) ? "" : fields[2];
+ String affixText = fields[3];
+ int[] continuation = new int[0];
+ final int slash = affixText.indexOf('/');
+ if (slash >= 0) {
+ continuation = parseFlags(affixText.substring(slash + 1), result.flagMode, line + 1);
+ affixText = affixText.substring(0, slash);
+ }
+ if (NO_MATERIAL.equals(affixText)) {
+ affixText = "";
+ }
+ final Affix affix = new Affix(flag, crossProduct, strip, affixText,
+ AffixCondition.parse(fields[4], suffix, line + 1), continuation);
+ if (suffix) {
+ result.suffixes.add(affix);
+ } else {
+ result.prefixes.add(affix);
+ }
+ }
+ return line;
+ }
+
+ /**
+ * Parses the word list: an optional leading entry count, then one entry per line
+ * consisting of the word, an optional {@code /flags} run, and optional trailing
+ * morphological fields, which are ignored. The morphological fields are cut off
+ * first, because the flag separator is only meaningful in what precedes them; a word
+ * may itself contain spaces. A slash escaped as {@code \/} belongs to the word itself
+ * and is unescaped in the stored key.
+ *
+ * @param content The decoded word-list content.
+ * @param flagMode The flag encoding declared by the affix file.
+ * @param flagAliases The affix file's {@code AF} alias table, possibly empty. When
+ * it is not empty, a purely numeric flag field is a 1-based
+ * reference into it rather than a flag run of its own.
+ * @return The words mapped to the flag sets of their entries. Never {@code null}.
+ * @throws IOException Thrown if a flag run is malformed or an alias reference is
+ * out of range.
+ */
+ private static Map<String, List<int[]>> parseWordList(String content,
+ FlagMode flagMode, List<int[]> flagAliases) throws IOException {
+ final String[] lines = splitLines(content);
+ final Map<String, List<int[]>> entries = new HashMap<>();
+ int start = 0;
+ if (lines.length > 0 && isCount(trim(lines[0]))) {
+ start = 1;
+ }
+ for (int i = start; i < lines.length; i++) {
+ final String line = trim(lines[i]);
+ if (line.isEmpty()) {
+ continue;
+ }
+ final int morphology = morphologyIndex(line);
+ final String entry = morphology < 0 ? line : trim(line.substring(0, morphology));
+ String word = entry;
+ int[] flags = new int[0];
+ final int slash = unescapedSlash(entry);
+ if (slash >= 0) {
+ word = entry.substring(0, slash);
+ String flagRun = entry.substring(slash + 1);
+ // The flag run ends at the first space or tabulator, the separators the
+ // word-list format defines; whatever follows is a morphological field even
+ // when it carries no two-letter tag, which hunspell tolerates and so do we.
+ for (int c = 0; c < flagRun.length(); c++) {
+ if (isFieldSeparator(flagRun.charAt(c))) {
+ flagRun = flagRun.substring(0, c);
+ break;
+ }
+ }
+ if (!flagAliases.isEmpty() && isCount(flagRun)) {
+ final int alias;
+ try {
+ alias = Integer.parseInt(flagRun);
+ } catch (NumberFormatException e) {
+ throw new IOException("malformed flag alias '" + flagRun + "' at line "
+ + (i + 1), e);
+ }
+ if (alias < 1 || alias > flagAliases.size()) {
+ throw new IOException("flag alias " + alias + " at line " + (i + 1)
+ + " is outside the AF table of " + flagAliases.size() + " aliases");
+ }
+ flags = flagAliases.get(alias - 1);
+ } else {
+ flags = parseFlags(flagRun, flagMode, i + 1);
+ }
+ }
+ entries.computeIfAbsent(word.replace("\\/", "/"), key -> new ArrayList<>(1))
+ .add(flags);
+ }
+ return entries;
+ }
+
+ /**
+ * Checks whether a line consists purely of decimal digits, which identifies the
+ * optional entry-count header of a word list.
+ *
+ * @param line The trimmed line to inspect.
+ * @return {@code true} if the line is a non-empty digit run.
+ */
+ private static boolean isCount(String line) {
+ if (line.isEmpty()) {
+ return false;
+ }
+ for (int i = 0; i < line.length(); i++) {
+ if (line.charAt(i) < '0' || line.charAt(i) > '9') {
+ return false;
+ }
+ }
+ return true;
+ }
+
+ /**
+ * Finds the first {@code /} that is not escaped as {@code \/}, which separates the
+ * word from its flag run in a word-list entry.
+ *
+ * @param line The word-list line to scan.
+ * @return The index of the separator, or {@code -1} when the entry has no flags.
+ */
+ private static int unescapedSlash(String line) {
+ for (int i = 0; i < line.length(); i++) {
+ if (line.charAt(i) == '/' && (i == 0 || line.charAt(i - 1) != '\\')) {
+ return i;
+ }
+ }
+ return -1;
+ }
+
+ /**
+ * Finds where the trailing morphological fields of a word-list entry begin, which
+ * terminates the word and its flag run. A morphological field is either introduced by
+ * a tabulator, the older separator, or written as a two-letter tag followed by
+ * {@code :} and preceded by a separator, such as {@code po:verb}. A separator that
+ * is not followed by such a tag belongs to the word, because a word-list entry may
+ * name several words. The separators are the space and the tabulator, exactly the
+ * two characters the reference implementation's {@code hashmgr.cxx} splits on; they
+ * are format delimiters of the word-list grammar, not a whitespace judgment, so
+ * wider whitespace such as a no-break space stays part of the word by design.
+ *
+ * @param line The trimmed word-list line to scan.
+ * @return The index at which the morphological fields begin, or {@code -1} if the
+ * entry carries none.
+ */
+ private static int morphologyIndex(String line) {
+ int cut = -1;
+ for (int i = 4; i < line.length(); i++) {
+ if (line.charAt(i) == ':' && isFieldSeparator(line.charAt(i - 3))) {
+ int fieldStart = i - 3;
+ while (fieldStart > 0 && isFieldSeparator(line.charAt(fieldStart - 1))) {
+ fieldStart--;
+ }
+ // a tag with no word in front of it is not a morphological field
+ cut = fieldStart == 0 ? -1 : fieldStart;
+ break;
+ }
+ }
+ final int tab = line.indexOf('\t');
+ if (tab >= 0 && (cut < 0 || tab < cut)) {
+ cut = tab;
+ }
+ return cut;
+ }
+
+ /**
+ * Checks one character against the word-list format's field separators, space and
+ * tabulator, the exact set the reference implementation splits morphological fields
+ * on.
+ *
+ * @param c The character to test.
+ * @return {@code true} if {@code c} separates fields in the word-list format.
+ */
+ private static boolean isFieldSeparator(char c) {
+ return c == ' ' || c == '\t';
+ }
+
+ /**
+ * Removes leading and trailing whitespace, using the whitespace definition the rest
+ * of the parser scans with.
+ *
+ * @param text The text to trim.
+ * @return The text without leading or trailing whitespace. Never {@code null}.
+ */
+ private static String trim(String text) {
+ int start = 0;
+ int end = text.length();
+ while (start < end && StringUtil.isWhitespace(text.charAt(start))) {
+ start++;
+ }
+ while (end > start && StringUtil.isWhitespace(text.charAt(end - 1))) {
+ end--;
+ }
+ return text.substring(start, end);
+ }
+
+ /**
+ * Parses a flag run according to the declared flag mode: single characters in
+ * {@code char} mode, character pairs packed into one {@code int} in {@code long}
+ * mode, and comma-separated decimal numbers in {@code num} mode.
+ *
+ * @param text The flag run without its leading {@code /}. An empty run carries no
+ * flags in every mode.
+ * @param mode The declared flag encoding.
+ * @param lineNumber The source line, for error messages.
+ * @return The parsed flags. Never {@code null}.
+ * @throws IOException Thrown if the run does not fit the declared encoding.
+ */
+ private static int[] parseFlags(String text, FlagMode mode, int lineNumber)
+ throws IOException {
+ if (text.isEmpty()) {
+ return new int[0];
+ }
+ switch (mode) {
+ case NUM: {
+ final String[] parts = splitOn(text, ',');
+ final int[] flags = new int[parts.length];
+ for (int i = 0; i < parts.length; i++) {
+ try {
+ flags[i] = Integer.parseInt(trim(parts[i]));
+ } catch (NumberFormatException e) {
+ throw new IOException("malformed numeric flag at line " + lineNumber, e);
+ }
+ }
+ return flags;
+ }
+ case LONG: {
+ if (text.length() % 2 != 0) {
+ throw new IOException("odd long-flag run at line " + lineNumber);
+ }
+ final int[] flags = new int[text.length() / 2];
+ for (int i = 0; i < flags.length; i++) {
+ flags[i] = (text.charAt(2 * i) << 16) | text.charAt(2 * i + 1);
+ }
+ return flags;
+ }
+ default: {
+ // One flag per code point: published dictionaries name affix rules with
+ // supplementary characters under FLAG UTF-8, and reading per UTF-16 unit
+ // would split such a flag into a surrogate pair. A variation selector
+ // (U+FE00..U+FE0F) only selects a flag character's presentation and is
+ // dropped from flag identity.
+ final int[] buffer = new int[text.codePointCount(0, text.length())];
+ int f = 0;
+ for (int i = 0; i < text.length(); ) {
+ final int codePoint = text.codePointAt(i);
+ i += Character.charCount(codePoint);
+ if (codePoint >= 0xFE00 && codePoint <= 0xFE0F) {
+ continue;
+ }
+ buffer[f++] = codePoint;
+ }
+ return f == buffer.length ? buffer : Arrays.copyOf(buffer, f);
+ }
+ }
+ }
+
+ /**
+ * Parses a field that must contain exactly one flag, such as the flag name in an
+ * affix block header.
+ *
+ * @param text The flag field.
+ * @param mode The declared flag encoding.
+ * @param lineNumber The source line, for error messages.
+ * @return The single parsed flag.
+ * @throws IOException Thrown if the field holds no flag or more than one.
+ */
+ private static int parseFlag(String text, FlagMode mode, int lineNumber)
+ throws IOException {
+ final int[] flags = parseFlags(text, mode, lineNumber);
+ if (flags.length != 1) {
+ throw new IOException("expected exactly one flag at line " + lineNumber);
+ }
+ return flags[0];
+ }
+
+ /**
+ * Splits text into lines with a single character scan, tolerating CRLF endings.
+ *
+ * @param content The text to split.
+ * @return The lines without their terminators. Never {@code null}.
+ */
+ private static String[] splitLines(String content) {
+ final List<String> lines = new ArrayList<>();
+ int start = 0;
+ for (int i = 0; i <= content.length(); i++) {
+ if (i == content.length() || content.charAt(i) == '\n') {
+ int end = i;
+ if (end > start && content.charAt(end - 1) == '\r') {
+ end--;
+ }
+ lines.add(content.substring(start, end));
+ start = i + 1;
+ }
+ }
+ return lines.toArray(new String[0]);
+ }
+
+ /**
+ * Splits text on a separator character with a single character scan.
+ *
+ * @param text The text to split.
+ * @param separator The separator character.
+ * @return The parts between the separators, empty ones included. Never {@code null}.
+ */
+ private static String[] splitOn(String text, char separator) {
+ final List<String> parts = new ArrayList<>();
+ int start = 0;
+ for (int i = 0; i <= text.length(); i++) {
+ if (i == text.length() || text.charAt(i) == separator) {
+ parts.add(text.substring(start, i));
+ start = i + 1;
+ }
+ }
+ return parts.toArray(new String[0]);
+ }
+
+ /**
+ * Splits a line on whitespace with a single character scan.
+ *
+ * @param line The line to split.
+ * @return The whitespace-separated fields, without empty ones. Never {@code null}.
+ */
+ private static String[] split(String line) {
+ final List<String> parts = new ArrayList<>();
+ int start = -1;
+ for (int i = 0; i <= line.length(); i++) {
+ if (i == line.length() || StringUtil.isWhitespace(line.charAt(i))) {
+ if (start >= 0) {
+ parts.add(line.substring(start, i));
+ start = -1;
+ }
+ } else if (start < 0) {
+ start = i;
+ }
+ }
+ return parts.toArray(new String[0]);
+ }
+}
diff --git a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/hunspell/HunspellDictionaryDownload.java b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/hunspell/HunspellDictionaryDownload.java
new file mode 100644
index 000000000..3775df4ac
--- /dev/null
+++ b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/hunspell/HunspellDictionaryDownload.java
@@ -0,0 +1,93 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package opennlp.tools.stemmer.hunspell;
+
+import java.io.IOException;
+import java.nio.file.Path;
+
+import opennlp.tools.util.DictionaryCatalog;
+
+/**
+ * Opt-in download of Hunspell {@code .aff}/{@code .dic} pairs and their license
+ * readme from an application-supplied {@link DictionaryCatalog}. Requires
+ * {@code -Dopennlp.download.remote=true}. OpenNLP bundles neither a catalog nor
+ * dictionary data.
+ *
+ * @since 3.0.0
+ */
+public final class HunspellDictionaryDownload {
+
+ private HunspellDictionaryDownload() {
+ }
+
+ /**
+ * Downloads the cataloged {@code .aff}, {@code .dic}, and readme files for
+ * {@code dictionaryId} into {@code targetDirectory}.
+ *
+ * @param catalog The application-supplied catalog. Must not be {@code null}.
+ * @param dictionaryId The catalog dictionary name, for example {@code en_US}.
+ * Must not be {@code null}.
+ * @param targetDirectory The directory to write into; created when absent. Must not
+ * be {@code null}.
+ * @throws IOException Thrown if remote downloads are disabled, a catalog entry is
+ * missing, or verification fails.
+ * @throws IllegalArgumentException Thrown if a parameter is {@code null}.
+ */
+ public static void downloadFromCatalog(DictionaryCatalog catalog, String dictionaryId,
+ Path targetDirectory) throws IOException {
+ if (catalog == null) {
+ throw new IllegalArgumentException("catalog must not be null");
+ }
+ if (dictionaryId == null) {
+ throw new IllegalArgumentException("dictionaryId must not be null");
+ }
+ if (targetDirectory == null) {
+ throw new IllegalArgumentException("targetDirectory must not be null");
+ }
+ final String prefix = "hunspell." + dictionaryId;
+ download(catalog, prefix + HunspellDictionary.AFFIX_FILE_SUFFIX, targetDirectory);
+ download(catalog, prefix + HunspellDictionary.DICTIONARY_FILE_SUFFIX, targetDirectory);
+ final String readmeId = prefix + ".readme";
+ if (catalog.ids().contains(readmeId)) {
+ download(catalog, readmeId, targetDirectory);
+ }
+ }
+
+ /**
+ * Downloads one catalog entry into {@code targetDirectory}, named by the entry's
+ * preferred file name or, when absent, by the last segment of its URI path.
+ *
+ * @param catalog The catalog holding {@code id}.
+ * @param id The catalog entry id.
+ * @param targetDirectory The directory to write into.
+ * @throws IOException Thrown if remote downloads are disabled, the entry is missing,
+ * or the download fails verification.
+ */
+ private static void download(DictionaryCatalog catalog, String id, Path targetDirectory)
+ throws IOException {
+ final DictionaryCatalog.Entry entry = catalog.get(id);
+ final String filename;
+ if (entry.filename() != null) {
+ filename = entry.filename();
+ } else {
+ final String path = entry.uri().getPath();
+ filename = path.substring(path.lastIndexOf('/') + 1);
+ }
+ catalog.download(id, targetDirectory.resolve(filename));
+ }
+}
diff --git a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/hunspell/HunspellStemmer.java b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/hunspell/HunspellStemmer.java
new file mode 100644
index 000000000..656064da3
--- /dev/null
+++ b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/hunspell/HunspellStemmer.java
@@ -0,0 +1,596 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package opennlp.tools.stemmer.hunspell;
+
+import java.util.ArrayList;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Set;
+
+import opennlp.tools.commons.ThreadSafe;
+import opennlp.tools.stemmer.Stemmer;
+import opennlp.tools.stemmer.hunspell.HunspellDictionary.Affix;
+import opennlp.tools.stemmer.hunspell.HunspellDictionary.CompoundPosition;
+import opennlp.tools.util.StringUtil;
+
+/**
+ * A dictionary-backed {@link Stemmer} over a {@link HunspellDictionary}: a surface form
+ * is reduced to the dictionary words it can be derived from by removing one suffix, one
+ * prefix, or a cross-product combination of both.
+ *
+ * <p>{@link #stem(CharSequence)} returns the first analysis, preferring the word's own
+ * dictionary entry; {@link #stemAll(CharSequence)} returns every distinct analysis. A
+ * word with no analysis is returned unchanged, so the stemmer degrades to identity on
+ * unknown vocabulary. A form containing uppercase characters is also analyzed in its
+ * lowercase variant, so sentence-initial capitalization does not hide an entry.
+ * Entries the dictionary marks as virtual stems ({@code NEEDAFFIX}), compound-only
+ * parts ({@code ONLYINCOMPOUND}), or forbidden words ({@code FORBIDDENWORD}) never
+ * count as standalone analyses, matching how hunspell reads those flags.</p>
+ *
+ * <p>Compound part search is capped at {@value #PART_CHECK_BUDGET} part-licensing
+ * attempts per input word; beyond that budget further compound analyses are skipped.
+ * The {@link Stemmer} interface leaves thread safety to the implementation. This
+ * implementation reads only the immutable dictionary state, so a single instance is
+ * safe to share between threads.</p>
+ *
+ * @since 3.0.0
+ */
+@ThreadSafe
+public final class HunspellStemmer implements Stemmer {
+
+ /**
+ * The most part-licensing attempts one decomposition search may spend. Compounding
+ * searches every split of every tail, which on adversarial input with a
+ * one-character minimum part length grows without useful bound; the budget stops
+ * the search there, missing analyses rather than stalling, in line with the
+ * engine's fail-closed posture.
+ */
+ private static final int PART_CHECK_BUDGET = 2048;
+
+ private final HunspellDictionary dictionary;
+
+ /**
+ * Initializes the stemmer.
+ *
+ * @param dictionary The dictionary to analyze against. Must not be {@code null}.
+ * @throws IllegalArgumentException Thrown if {@code dictionary} is {@code null}.
+ */
+ public HunspellStemmer(HunspellDictionary dictionary) {
+ if (dictionary == null) {
+ throw new IllegalArgumentException("dictionary must not be null");
+ }
+ this.dictionary = dictionary;
+ }
+
+ /**
+ * {@inheritDoc}
+ *
+ * <p>Returns the first analysis, which prefers the word's own dictionary entry.</p>
+ */
+ @Override
+ public CharSequence stem(CharSequence word) {
+ final List<CharSequence> analyses = stemAll(word);
+ return analyses.get(0);
+ }
+
+ /**
+ * {@inheritDoc}
+ *
+ * <p>Returns every distinct analysis, or a single-element list of the unchanged word
+ * when it has none.</p>
+ */
+ @Override
+ public List<CharSequence> stemAll(CharSequence word) {
+ if (word == null) {
+ throw new IllegalArgumentException("word must not be null");
+ }
+ final String surface = word.toString();
+ if (surface.isEmpty()) {
+ // a zero-length word has no morphology; without this guard a strip-only rule
+ // could restore its strip string onto nothing and answer a non-empty stem
+ return List.of(surface);
+ }
+ final Set<String> analyses = new LinkedHashSet<>();
+ for (final String variant : variants(surface)) {
+ analyze(variant, analyses);
+ }
+ if (analyses.isEmpty() && dictionary.compoundsDeclared()) {
+ for (final String variant : variants(surface)) {
+ decompose(variant, surface, analyses);
+ }
+ }
+ if (analyses.isEmpty()) {
+ return List.of(surface);
+ }
+ return List.copyOf(analyses);
+ }
+
+ /**
+ * Collects the case variants to analyze: the surface form first, then its lowercase
+ * form when the two differ. Ordering matters because the first analysis found wins
+ * in {@link #stem(CharSequence)}.
+ *
+ * @param surface The surface form.
+ * @return The variants in analysis order. Never {@code null} or empty.
+ */
+ private List<String> variants(String surface) {
+ final String lowered = StringUtil.toLowerCase(surface);
+ return lowered.equals(surface) ? List.of(surface) : List.of(surface, lowered);
+ }
+
+ /**
+ * Adds every analysis of one case variant to the result set: the word's own
+ * dictionary entry, single suffix removal, twofold suffix removal through
+ * continuation classes, single prefix removal, and cross-product removal of one
+ * prefix together with one suffix. Insertion order into the set fixes the
+ * preference order reported by {@link #stemAll(CharSequence)}.
+ *
+ * @param word The case variant to analyze.
+ * @param analyses The mutable, insertion-ordered set collecting the stems found.
+ */
+ private void analyze(String word, Set<String> analyses) {
+ final List<int[]> own = dictionary.lookup(word);
+ if (own != null && dictionary.validStandalone(own)) {
+ analyses.add(word);
+ }
+ for (final Affix suffix : dictionary.suffixesEndingWith(
+ word.codePointBefore(word.length()))) {
+ undoSuffix(word, suffix, analyses);
+ }
+ for (final Affix suffix : dictionary.suffixesWithoutMaterial()) {
+ undoSuffix(word, suffix, analyses);
+ }
+ for (final Affix prefix : dictionary.prefixesStartingWith(word.codePointAt(0))) {
+ undoPrefix(word, prefix, analyses);
+ }
+ for (final Affix prefix : dictionary.prefixesWithoutMaterial()) {
+ undoPrefix(word, prefix, analyses);
+ }
+ }
+
+ /**
+ * Decomposes a word into listed compound parts when the affix analysis found
+ * nothing: the first part must be admitted to open a compound, every further part
+ * to continue or close one, each at least the declared minimum length and counted
+ * against the declared maximum. A part stands on its own entry or on an entry plus
+ * one affix, the way published dictionaries position their linking forms through
+ * zero or dash suffixes. The stems of the parts of every successful splitting are
+ * reported left to right, so the head-most material comes last. A word the
+ * dictionary lists as forbidden never decomposes; that is how one specific
+ * ill-formed compound is blocked while its parts stay productive.
+ *
+ * @param word The case variant to decompose.
+ * @param surface The surface form the variant was derived from; character case at
+ * junctions is judged against it, so lowercasing a variant cannot
+ * sidestep a {@code CHECKCOMPOUNDCASE} declaration.
+ * @param analyses The mutable, insertion-ordered set collecting the part stems.
+ */
+ private void decompose(String word, String surface, Set<String> analyses) {
+ final List<int[]> own = dictionary.lookup(word);
+ if (own != null && dictionary.anyForbidden(own)) {
+ return;
+ }
+ if (word.length() < 2 * dictionary.compoundMin()) {
+ return;
+ }
+ // lowercasing may change the length in exceptional mappings, in which case the
+ // offsets no longer align and the variant itself is the only usable case source
+ final String caseSource = surface.length() == word.length() ? surface : word;
+ search(word, caseSource, 0, new ArrayList<>(), new ArrayList<>(), analyses,
+ new int[] {PART_CHECK_BUDGET});
+ }
+
+ /**
+ * Extends a partial decomposition with the part starting at {@code from}, trying
+ * every admissible length and recursing on the remainder. The boundary into this
+ * part honors the {@code CHECKCOMPOUNDCASE} and {@code CHECKCOMPOUNDTRIPLE}
+ * declarations, a part repeating its left neighbor honors
+ * {@code CHECKCOMPOUNDDUP}, and a completed decomposition flushes every part's
+ * stems into the analyses in part order.
+ *
+ * @param word The case variant under decomposition.
+ * @param caseSource The character-case source for junction checks, the surface
+ * form when its offsets align with the variant.
+ * @param from The index the next part starts at.
+ * @param surfaces The surface strings of the parts taken so far.
+ * @param stems The licensed stems of the parts taken so far, one list per part.
+ * @param analyses The mutable, insertion-ordered set collecting the part stems.
+ * @param budget The remaining part-licensing attempts, counted down in place.
+ */
+ private void search(String word, String caseSource, int from, List<String> surfaces,
+ List<List<String>> stems, Set<String> analyses, int[] budget) {
+ if (from > 0 && violatesBoundaryChecks(word, caseSource, from)) {
+ return;
+ }
+ final int min = dictionary.compoundMin();
+ final int max = dictionary.compoundWordMax();
+ final boolean first = from == 0;
+ // every split leaving room for a further part; a first-position part must also
+ // leave the closing part, so the whole word is never one part
+ if (max == 0 || surfaces.size() + 2 <= max) {
+ for (int end = from + min; end <= word.length() - min; end++) {
+ if (budget[0] <= 0) {
+ return;
+ }
+ budget[0]--;
+ final String part = word.substring(from, end);
+ if (duplicatesNeighbor(part, surfaces)) {
+ continue;
+ }
+ final List<String> partStems = partStems(part,
+ first ? CompoundPosition.BEGIN : CompoundPosition.MIDDLE, first, false);
+ if (partStems.isEmpty()) {
+ continue;
+ }
+ surfaces.add(part);
+ stems.add(partStems);
+ search(word, caseSource, end, surfaces, stems, analyses, budget);
+ surfaces.remove(surfaces.size() - 1);
+ stems.remove(stems.size() - 1);
+ }
+ }
+ // the closing part takes the whole remainder; a compound has at least two parts
+ if (first || word.length() - from < min
+ || (max > 0 && surfaces.size() + 1 > max) || budget[0] <= 0) {
+ return;
+ }
+ budget[0]--;
+ final String part = word.substring(from);
+ if (duplicatesNeighbor(part, surfaces)) {
+ return;
+ }
+ final List<String> partStems = partStems(part, CompoundPosition.END, false, true);
+ if (partStems.isEmpty()) {
+ return;
+ }
+ for (final List<String> earlier : stems) {
+ analyses.addAll(earlier);
+ }
+ analyses.addAll(partStems);
+ }
+
+ /**
+ * Applies the {@code CHECKCOMPOUNDDUP} declaration: a part must not repeat the
+ * part directly before it.
+ *
+ * @param part The candidate part.
+ * @param surfaces The surface strings of the parts taken so far.
+ * @return {@code true} if the declaration forbids this part here.
+ */
+ private boolean duplicatesNeighbor(String part, List<String> surfaces) {
+ return dictionary.checkCompoundDup() && !surfaces.isEmpty()
+ && part.equals(surfaces.get(surfaces.size() - 1));
+ }
+
+ /**
+ * Applies the character-level boundary declarations at the junction before
+ * {@code from}: {@code CHECKCOMPOUNDCASE} forbids an uppercase character on either
+ * side of the junction, and {@code CHECKCOMPOUNDTRIPLE} forbids the same character
+ * three times in a row across it.
+ *
+ * @param word The case variant under decomposition.
+ * @param caseSource The character-case source for the uppercase judgment.
+ * @param from The index the junction sits before; greater than zero.
+ * @return {@code true} if a declaration forbids this junction.
+ */
+ private boolean violatesBoundaryChecks(String word, String caseSource, int from) {
+ final char before = word.charAt(from - 1);
+ final char after = word.charAt(from);
+ if (dictionary.checkCompoundCase()
+ && (Character.isUpperCase(caseSource.charAt(from - 1))
+ || Character.isUpperCase(caseSource.charAt(from)))) {
+ return true;
+ }
+ if (dictionary.checkCompoundTriple() && before == after
+ && ((from >= 2 && word.charAt(from - 2) == after)
+ || (from + 1 < word.length() && word.charAt(from + 1) == after))) {
+ return true;
+ }
+ return false;
+ }
+
+ /**
+ * Collects the listed stems that admit one part at its compound position: the part
+ * as its own entry, or an entry plus one suffix or one prefix whose removal leaves
+ * a listed stem, zero-material rules included, because published dictionaries
+ * position their linking forms through zero and dash suffixes. An affix at a
+ * compound-internal boundary must carry the permit flag, a suffix facing the next
+ * part or a prefix facing the previous one. A part not found as written is also
+ * tried with its first letter uppercased, the way nouns listed capitalized appear
+ * lowercase inside a compound.
+ *
+ * @param part The part's surface text.
+ * @param position The part's place in the compound.
+ * @param first Whether the part opens the word.
+ * @param last Whether the part closes the word.
+ * @return The stems admitting the part, in discovery order. Never {@code null}.
+ */
+ private List<String> partStems(String part, CompoundPosition position,
+ boolean first, boolean last) {
+ final Set<String> stems = new LinkedHashSet<>();
+ collectPartStems(part, position, first, last, stems);
+ if (stems.isEmpty() && !part.isEmpty()) {
+ final int initial = part.codePointAt(0);
+ final int upper = Character.toUpperCase(initial);
+ if (upper != initial) {
+ collectPartStems(new StringBuilder().appendCodePoint(upper)
+ .append(part, Character.charCount(initial), part.length()).toString(),
+ position, first, last, stems);
+ }
+ }
+ return List.copyOf(stems);
+ }
+
+ /**
+ * Collects the stems admitting one spelling of a part, bare and through one affix.
+ *
+ * @param part The part spelling to look up.
+ * @param position The part's place in the compound.
+ * @param first Whether the part opens the word.
+ * @param last Whether the part closes the word.
+ * @param stems The mutable, insertion-ordered set collecting the stems.
+ */
+ private void collectPartStems(String part, CompoundPosition position,
+ boolean first, boolean last, Set<String> stems) {
+ final List<int[]> own = dictionary.lookup(part);
+ if (own != null && dictionary.mayStand(own, position)) {
+ stems.add(part);
+ }
+ for (final Affix suffix : dictionary.suffixesEndingWith(
+ part.codePointBefore(part.length()))) {
+ collectAffixedPartStem(part, suffix, true, position, last, stems);
+ }
+ for (final Affix suffix : dictionary.suffixesWithoutMaterial()) {
+ collectAffixedPartStem(part, suffix, true, position, last, stems);
+ }
+ for (final Affix prefix : dictionary.prefixesStartingWith(part.codePointAt(0))) {
+ collectAffixedPartStem(part, prefix, false, position, first, stems);
+ }
+ for (final Affix prefix : dictionary.prefixesWithoutMaterial()) {
+ collectAffixedPartStem(part, prefix, false, position, first, stems);
+ }
+ }
+
+ /**
+ * Adds the stem of one affixed part reading when the rule and the stem's entry admit
+ * it at the position.
+ *
+ * @param part The part spelling under analysis.
+ * @param affix The rule to undo.
+ * @param suffix Whether the rule is a suffix rule.
+ * @param position The part's place in the compound.
+ * @param atEdge Whether the part sits at the word end the rule faces, the closing part
+ * for a suffix rule and the opening part for a prefix rule; an affix
+ * facing another part instead needs the permit flag.
+ * @param stems The mutable, insertion-ordered set collecting the stems.
+ */
+ private void collectAffixedPartStem(String part, Affix affix, boolean suffix,
+ CompoundPosition position, boolean atEdge, Set<String> stems) {
+ if (dictionary.circumfixOnly(affix) || dictionary.forbidsInCompound(affix)
+ || (!atEdge && !dictionary.permitsInside(affix))) {
+ return;
+ }
+ final String stem = removeAffixInCompound(part, affix, suffix);
+ if (stem == null) {
+ return;
+ }
+ final List<int[]> flagSets = dictionary.lookup(stem);
+ if (flagSets != null && dictionary.supportsPart(flagSets, affix.flag(), position,
+ dictionary.affixAdmits(affix, position))) {
+ stems.add(stem);
+ }
+ }
+
+ /**
+ * Undoes one affix rule on a compound part. Unlike the standalone removals, a rule
+ * that neither adds nor removes material is undone here, to its own spelling with
+ * the condition checked, because dictionaries position compound parts through
+ * exactly such zero rules.
+ *
+ * @param part The part spelling under analysis.
+ * @param affix The rule to undo.
+ * @param suffix Whether the rule is a suffix rule.
+ * @return The candidate stem, or {@code null} when the rule does not apply.
+ */
+ private String removeAffixInCompound(String part, Affix affix, boolean suffix) {
+ if (affix.affix().isEmpty() && affix.strip().isEmpty()) {
+ return affix.condition().matches(part) ? part : null;
+ }
+ return suffix ? removeSuffix(part, affix) : removePrefix(part, affix);
+ }
+
+ /**
+ * Undoes one suffix rule and, through continuation classes, one further suffix on
+ * the intermediate stem, adding every dictionary-confirmed analysis. A rule that
+ * applies only inside compounds or only as half of a circumfix is not undone at all,
+ * the latter because no prefix accompanies it on this path; a rule marked as needing
+ * a further affix yields no single-removal analysis, because the surface form it
+ * makes alone is a virtual stem; its twofold analyses stand, the inner affix being
+ * exactly the further one required.
+ *
+ * @param word The case variant under analysis.
+ * @param suffix The suffix rule to undo.
+ * @param analyses The mutable, insertion-ordered set collecting the stems found.
+ */
+ private void undoSuffix(String word, Affix suffix, Set<String> analyses) {
+ if (dictionary.compoundOnly(suffix) || dictionary.circumfixOnly(suffix)) {
+ return;
+ }
+ final String stem = removeSuffix(word, suffix);
+ if (stem == null) {
+ return;
+ }
+ if (!dictionary.needsFurtherAffix(suffix)) {
+ final List<int[]> flagSets = dictionary.lookup(stem);
+ if (flagSets != null && dictionary.supports(flagSets, suffix.flag())) {
+ analyses.add(stem);
+ }
+ }
+ for (final Affix inner : dictionary.suffixesEndingWith(
+ stem.codePointBefore(stem.length()))) {
+ undoInnerSuffix(stem, suffix, inner, analyses);
+ }
+ for (final Affix inner : dictionary.suffixesWithoutMaterial()) {
+ undoInnerSuffix(stem, suffix, inner, analyses);
+ }
+ }
+
+ /**
+ * Undoes the second suffix of a twofold removal when the inner rule's continuation
+ * classes allow it after the outer one.
+ *
+ * @param stem The intermediate stem after the outer removal.
+ * @param outer The already-undone outer suffix rule.
+ * @param inner The candidate inner suffix rule.
+ * @param analyses The mutable, insertion-ordered set collecting the stems found.
+ */
+ private void undoInnerSuffix(String stem, Affix outer, Affix inner,
+ Set<String> analyses) {
+ if (!inner.allowsContinuation(outer.flag()) || dictionary.compoundOnly(inner)
+ || dictionary.circumfixOnly(inner)) {
+ return;
+ }
+ final String doubleStem = removeSuffix(stem, inner);
+ if (doubleStem == null) {
+ return;
+ }
+ final List<int[]> innerFlags = dictionary.lookup(doubleStem);
+ if (innerFlags != null && dictionary.supports(innerFlags, inner.flag())) {
+ analyses.add(doubleStem);
+ }
+ }
+
+ /**
+ * Undoes one prefix rule and, for cross-product rules, one further suffix on the
+ * intermediate stem, adding every dictionary-confirmed analysis. A rule that
+ * applies only inside compounds is not undone at all. A rule marked as needing a
+ * further affix or as half of a circumfix yields no single-removal analysis; its
+ * cross-product analyses stand, the suffix being exactly the further affix or the
+ * other circumfix half required.
+ *
+ * @param word The case variant under analysis.
+ * @param prefix The prefix rule to undo.
+ * @param analyses The mutable, insertion-ordered set collecting the stems found.
+ */
+ private void undoPrefix(String word, Affix prefix, Set<String> analyses) {
+ if (dictionary.compoundOnly(prefix)) {
+ return;
+ }
+ final String stem = removePrefix(word, prefix);
+ if (stem == null) {
+ return;
+ }
+ if (!dictionary.needsFurtherAffix(prefix) && !dictionary.circumfixOnly(prefix)) {
+ final List<int[]> flagSets = dictionary.lookup(stem);
+ if (flagSets != null && dictionary.supports(flagSets, prefix.flag())) {
+ analyses.add(stem);
+ }
+ }
+ if (!prefix.crossProduct()) {
+ return;
+ }
+ for (final Affix suffix : dictionary.suffixesEndingWith(
+ stem.codePointBefore(stem.length()))) {
+ undoCrossProductSuffix(stem, prefix, suffix, analyses);
+ }
+ for (final Affix suffix : dictionary.suffixesWithoutMaterial()) {
+ undoCrossProductSuffix(stem, prefix, suffix, analyses);
+ }
+ }
+
+ /**
+ * Undoes the suffix half of a cross-product removal when both rules opted in. The
+ * two rules must agree on circumfixing: a circumfix-marked affix is only valid with
+ * a marked affix of the other kind, so a pair of which exactly one is marked mixes
+ * an ordinary affix into a circumfix and is rejected.
+ *
+ * @param stem The intermediate stem after the prefix removal.
+ * @param prefix The already-undone prefix rule.
+ * @param suffix The candidate suffix rule.
+ * @param analyses The mutable, insertion-ordered set collecting the stems found.
+ */
+ private void undoCrossProductSuffix(String stem, Affix prefix, Affix suffix,
+ Set<String> analyses) {
+ if (!suffix.crossProduct() || dictionary.compoundOnly(suffix)
+ || dictionary.circumfixOnly(prefix) != dictionary.circumfixOnly(suffix)) {
+ return;
+ }
+ final String doubleStem = removeSuffix(stem, suffix);
+ if (doubleStem == null) {
+ return;
+ }
+ // a needs-further-affix marker on either rule is satisfied by the other rule,
+ // so no such check applies here; both flags must sit in one homonym's flag set
+ final List<int[]> both = dictionary.lookup(doubleStem);
+ if (both != null && dictionary.supports(both, prefix.flag(), suffix.flag())) {
+ analyses.add(doubleStem);
+ }
+ }
+
+ /**
+ * Undoes one suffix rule: cuts the affix material off the end of the word, restores
+ * the strip string the rule removed on application, and checks the rule's condition
+ * against the restored stem. A strip-only rule, whose affix material is empty, is
+ * undone by restoring its strip string alone. Rules that neither add nor remove
+ * material and candidates that would leave an empty stem are rejected. A word the
+ * affix material covers entirely reverses a full-strip application, which hunspell
+ * only performs when the affix file declares {@code FULLSTRIP}; without that
+ * declaration the rule does not apply.
+ *
+ * @param word The surface form.
+ * @param suffix The rule to undo.
+ * @return The candidate stem, or {@code null} when the rule does not apply.
+ */
+ private String removeSuffix(String word, Affix suffix) {
+ final String affix = suffix.affix();
+ final String strip = suffix.strip();
+ if (affix.isEmpty() && strip.isEmpty() || !word.endsWith(affix)
+ || word.length() - affix.length() + strip.length() == 0
+ || (word.length() == affix.length() && !dictionary.fullStrip())) {
+ return null;
+ }
+ final String stem = word.substring(0, word.length() - affix.length()) + strip;
+ return suffix.condition().matches(stem) ? stem : null;
+ }
+
+ /**
+ * Undoes one prefix rule: cuts the affix material off the start of the word,
+ * restores the strip string the rule removed on application, and checks the rule's
+ * condition against the restored stem. A strip-only rule, whose affix material is
+ * empty, is undone by restoring its strip string alone. Rules that neither add nor
+ * remove material and candidates that would leave an empty stem are rejected. A
+ * word the affix material covers entirely reverses a full-strip application, which
+ * hunspell only performs when the affix file declares {@code FULLSTRIP}; without
+ * that declaration the rule does not apply.
+ *
+ * @param word The surface form.
+ * @param prefix The rule to undo.
+ * @return The candidate stem, or {@code null} when the rule does not apply.
+ */
+ private String removePrefix(String word, Affix prefix) {
+ final String affix = prefix.affix();
+ final String strip = prefix.strip();
+ if (affix.isEmpty() && strip.isEmpty() || !word.startsWith(affix)
+ || word.length() - affix.length() + strip.length() == 0
+ || (word.length() == affix.length() && !dictionary.fullStrip())) {
+ return null;
+ }
+ final String stem = strip + word.substring(affix.length());
+ return prefix.condition().matches(stem) ? stem : null;
+ }
+}
diff --git a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/hunspell/HunspellStemmerFactory.java b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/hunspell/HunspellStemmerFactory.java
new file mode 100644
index 000000000..b8b7c4e62
--- /dev/null
+++ b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/hunspell/HunspellStemmerFactory.java
@@ -0,0 +1,60 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package opennlp.tools.stemmer.hunspell;
+
+import opennlp.tools.commons.ThreadSafe;
+import opennlp.tools.stemmer.Stemmer;
+import opennlp.tools.stemmer.StemmerFactory;
+
+/**
+ * The shareable handle for Hunspell stemming: holds one immutable
+ * {@link HunspellDictionary} and hands out {@link HunspellStemmer} instances over it.
+ *
+ * <p>The factory is immutable and safe to share across threads.</p>
+ *
+ * @since 3.0.0
+ */
+@ThreadSafe
+public class HunspellStemmerFactory implements StemmerFactory {
+
+ private final HunspellDictionary dictionary;
+
+ /**
+ * Initializes the factory.
+ *
+ * @param dictionary The dictionary to stem against. Must not be {@code null}.
+ * @throws IllegalArgumentException Thrown if {@code dictionary} is {@code null}.
+ */
+ public HunspellStemmerFactory(HunspellDictionary dictionary) {
+ if (dictionary == null) {
+ throw new IllegalArgumentException("dictionary must not be null");
+ }
+ this.dictionary = dictionary;
+ }
+
+ /**
+ * {@inheritDoc}
+ *
+ * <p>Every call creates a fresh {@link HunspellStemmer} over the same immutable
+ * dictionary.</p>
+ */
+ @Override
+ public Stemmer newStemmer() {
+ return new HunspellStemmer(dictionary);
+ }
+}
diff --git a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/DictionaryCatalog.java b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/DictionaryCatalog.java
new file mode 100644
index 000000000..d0704e8c6
--- /dev/null
+++ b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/DictionaryCatalog.java
@@ -0,0 +1,149 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package opennlp.tools.util;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.nio.file.Path;
+import java.util.Collections;
+import java.util.LinkedHashSet;
+import java.util.Properties;
+import java.util.Set;
+
+/**
+ * Opt-in catalog of remote dictionary archives and companion files, loaded from
+ * application-supplied properties containing URLs and SHA-512 digests. Fetching an
+ * entry requires {@link DownloadUtil#REMOTE_DOWNLOAD_PROPERTY} to be {@code true}.
+ *
+ * @since 3.0.0
+ */
+public final class DictionaryCatalog {
+
+ private final Properties properties;
+
+ private DictionaryCatalog(Properties properties) {
+ this.properties = properties;
+ }
+
+ /**
+ * Loads a catalog from a properties stream.
+ *
+ * @param in The properties content. Must not be {@code null}.
+ * @return The catalog. Never {@code null}.
+ * @throws IOException Thrown if reading fails.
+ * @throws IllegalArgumentException Thrown if {@code in} is {@code null}.
+ */
+ public static DictionaryCatalog load(InputStream in) throws IOException {
+ if (in == null) {
+ throw new IllegalArgumentException("in must not be null");
+ }
+ final Properties properties = new Properties();
+ properties.load(in);
+ return new DictionaryCatalog(properties);
+ }
+
+ /**
+ * {@return the catalog entry ids, in encounter order}
+ */
+ public Set<String> ids() {
+ final Set<String> ids = new LinkedHashSet<>();
+ for (final String key : properties.stringPropertyNames()) {
+ if (key.endsWith(".url")) {
+ ids.add(key.substring(0, key.length() - ".url".length()));
+ }
+ }
+ return Collections.unmodifiableSet(ids);
+ }
+
+ /**
+ * Looks up one catalog entry.
+ *
+ * @param id The entry id, for example {@code mecab.ipadic}.
+ * @return The entry. Never {@code null}.
+ * @throws IOException Thrown if the entry is incomplete or the URI is malformed.
+ * @throws IllegalArgumentException Thrown if {@code id} is {@code null}.
+ */
+ public Entry get(String id) throws IOException {
+ if (id == null) {
+ throw new IllegalArgumentException("id must not be null");
+ }
+ final String url = properties.getProperty(id + ".url");
+ final String sha512 = properties.getProperty(id + ".sha512");
+ if (url == null || sha512 == null) {
+ throw new IOException("unknown or incomplete dictionary catalog entry: " + id);
+ }
+ final String filename = properties.getProperty(id + ".filename");
+ try {
+ return new Entry(id, new URI(url), sha512.trim(), filename);
+ } catch (URISyntaxException e) {
+ throw new IOException("malformed catalog URI for " + id, e);
+ }
+ }
+
+ /**
+ * Downloads a catalog entry into {@code target} after checking that remote catalog
+ * downloads are enabled.
+ *
+ * @param id The entry id. Must not be {@code null}.
+ * @param target The local file to create. Must not be {@code null}.
+ * @throws IOException Thrown if the property is not enabled, the entry is missing,
+ * or the download fails verification.
+ * @throws IllegalArgumentException Thrown if a parameter is {@code null}.
+ */
+ public void download(String id, Path target) throws IOException {
+ if (target == null) {
+ throw new IllegalArgumentException("target must not be null");
+ }
+ if (!DownloadUtil.isRemoteDownloadEnabled()) {
+ throw new IOException("remote dictionary catalog downloads are disabled; set -D"
+ + DownloadUtil.REMOTE_DOWNLOAD_PROPERTY + "=true to enable");
+ }
+ final Entry entry = get(id);
+ DownloadUtil.download(entry.uri(), target, entry.sha512());
+ }
+
+ /**
+ * One pinned remote file: a stable URL and the SHA-512 of its bytes.
+ *
+ * @param id The catalog id.
+ * @param uri The absolute download URI.
+ * @param sha512 The expected SHA-512 hex digest.
+ * @param filename An optional preferred local file name; may be {@code null}.
+ */
+ public record Entry(String id, URI uri, String sha512, String filename) {
+ /**
+ * @param id The catalog id. Must not be {@code null}.
+ * @param uri The absolute download URI. Must not be {@code null}.
+ * @param sha512 The expected SHA-512 hex digest. Must not be {@code null}.
+ * @param filename An optional preferred local file name; may be {@code null}.
+ */
+ public Entry {
+ if (id == null) {
+ throw new IllegalArgumentException("id must not be null");
+ }
+ if (uri == null) {
+ throw new IllegalArgumentException("uri must not be null");
+ }
+ if (sha512 == null) {
+ throw new IllegalArgumentException("sha512 must not be null");
+ }
+ }
+ }
+}
diff --git a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/DownloadUtil.java b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/DownloadUtil.java
index 0f6f6800d..3e9c1c3bb 100644
--- a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/DownloadUtil.java
+++ b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/DownloadUtil.java
@@ -21,11 +21,15 @@ import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
+import java.io.OutputStream;
+import java.net.HttpURLConnection;
import java.net.MalformedURLException;
import java.net.URI;
import java.net.URISyntaxException;
import java.net.URL;
+import java.net.URLConnection;
import java.nio.charset.StandardCharsets;
+import java.nio.file.AtomicMoveNotSupportedException;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
@@ -38,6 +42,7 @@ import java.util.Collections;
import java.util.Formatter;
import java.util.HashMap;
import java.util.List;
+import java.util.Locale;
import java.util.Map;
import java.util.Objects;
import java.util.regex.Matcher;
@@ -51,7 +56,9 @@ import opennlp.tools.models.ModelType;
import opennlp.tools.util.model.BaseModel;
/**
- * This class facilitates the downloading of pretrained OpenNLP models.
+ * Downloads remote resources into a local path: pretrained OpenNLP models, and any
+ * other file fetched through {@link #download(URI, Path, String)} with an expected
+ * SHA-512 digest.
*/
public class DownloadUtil {
@@ -64,6 +71,34 @@ public class DownloadUtil {
private static final String OPENNLP_DOWNLOAD_HOME = "OPENNLP_DOWNLOAD_HOME";
private static final String CHECKSUM_EXTENSION = ".sha512";
+ /**
+ * System property that must be {@code true} before a
+ * {@link DictionaryCatalog} entry may be fetched. Explicit
+ * {@link #download(URI, Path, String)} calls do not require it: the caller already
+ * supplied the URI and digest.
+ */
+ public static final String REMOTE_DOWNLOAD_PROPERTY = "opennlp.download.remote";
+
+ /**
+ * System property for overriding {@link #MAX_DOWNLOAD_BYTES}. Set at JVM startup,
+ * e.g. {@code -Dopennlp.download.max.bytes=2147483648} for dictionaries larger than
+ * the default ceiling. Falls back to the default if absent, non-numeric, or not
+ * positive.
+ */
+ public static final String MAX_DOWNLOAD_BYTES_PROPERTY = "opennlp.download.max.bytes";
+
+ /**
+ * Inclusive ceiling on bytes buffered for one {@link #download(URI, Path, String)},
+ * 64 MiB unless overridden via {@link #MAX_DOWNLOAD_BYTES_PROPERTY}.
+ */
+ public static final long MAX_DOWNLOAD_BYTES =
+ configuredLimit(MAX_DOWNLOAD_BYTES_PROPERTY, 64L * 1024 * 1024);
+
+ private static final int CONNECT_TIMEOUT_MS = 30_000;
+ private static final int READ_TIMEOUT_MS = 300_000;
+ private static final int SHA512_HEX_LENGTH = 128;
+ private static final String DOWNLOAD_SUFFIX = ".download";
+
private static Map<String, Map<ModelType, URL>> availableModels;
/**
@@ -175,6 +210,175 @@ public class DownloadUtil {
}
}
+ /**
+ * Downloads {@code source} into {@code target} and requires the SHA-512 digest of the
+ * stored bytes to equal {@code expectedSha512}. The download is written to a sibling
+ * temporary file and moved into place only after the digest matches. The transfer is
+ * capped at {@link #MAX_DOWNLOAD_BYTES}; remote {@code http} and {@code https} URIs
+ * additionally use connect and read timeouts.
+ *
+ * @param source The absolute URI to fetch. Must not be {@code null}.
+ * @param target The local file to create or replace. Must not be {@code null}.
+ * @param expectedSha512 The expected SHA-512 digest as 128 lowercase or uppercase hex
+ * digits. Must not be {@code null}.
+ * @throws IOException Thrown if fetching fails, the size ceiling is exceeded, or the
+ * digest does not match.
+ * @throws IllegalArgumentException Thrown if a parameter is {@code null}, {@code source}
+ * is not absolute, or {@code expectedSha512} is not 128 hex digits.
+ */
+ public static void download(URI source, Path target, String expectedSha512)
+ throws IOException {
+ download(source, target, expectedSha512, MAX_DOWNLOAD_BYTES);
+ }
+
+ /**
+ * Downloads {@code source} into {@code target} under a caller-supplied byte ceiling.
+ *
+ * @param source The absolute URI to fetch. Must not be {@code null}.
+ * @param target The local file to create or replace. Must not be {@code null}.
+ * @param expectedSha512 The expected SHA-512 digest as 128 hex digits. Must not be
+ * {@code null}.
+ * @param maxBytes The inclusive ceiling on bytes read from {@code source}.
+ * @throws IOException Thrown if fetching fails, {@code maxBytes} is exceeded, or the
+ * digest does not match.
+ * @throws IllegalArgumentException Thrown if a parameter is invalid, see
+ * {@link #download(URI, Path, String)}.
+ */
+ static void download(URI source, Path target, String expectedSha512, long maxBytes)
+ throws IOException {
+ if (source == null) {
+ throw new IllegalArgumentException("source must not be null");
+ }
+ if (target == null) {
+ throw new IllegalArgumentException("target must not be null");
+ }
+ if (expectedSha512 == null) {
+ throw new IllegalArgumentException("expectedSha512 must not be null");
+ }
+ if (!source.isAbsolute()) {
+ throw new IllegalArgumentException("source must be an absolute URI");
+ }
+ final String normalized = normalizeSha512(expectedSha512);
+ final Path parent = target.getParent();
+ if (parent != null) {
+ Files.createDirectories(parent);
+ }
+ final Path partial = target.resolveSibling(target.getFileName() + DOWNLOAD_SUFFIX);
+ Files.deleteIfExists(partial);
+ try {
+ long size = 0L;
+ final MessageDigest digest = sha512Digest();
+ final URLConnection connection = open(source);
+ try (InputStream in = connection.getInputStream();
+ DigestInputStream digester = new DigestInputStream(in, digest);
+ OutputStream out = Files.newOutputStream(partial)) {
+ final byte[] buffer = new byte[8192];
+ int n;
+ while ((n = digester.read(buffer)) >= 0) {
+ size += n;
+ if (size > maxBytes) {
+ throw new IOException("download size exceeds safe limit of " + maxBytes);
+ }
+ out.write(buffer, 0, n);
+ }
+ } finally {
+ if (connection instanceof HttpURLConnection http) {
+ http.disconnect();
+ }
+ }
+ final String actual = byteArrayToHexString(digest.digest());
+ if (!actual.equals(normalized)) {
+ throw new IOException("SHA512 checksum validation failed for " + target.getFileName()
+ + ". Expected: " + normalized + ", but got: " + actual);
+ }
+ try {
+ Files.move(partial, target, StandardCopyOption.REPLACE_EXISTING,
+ StandardCopyOption.ATOMIC_MOVE);
+ } catch (AtomicMoveNotSupportedException e) {
+ Files.move(partial, target, StandardCopyOption.REPLACE_EXISTING);
+ }
+ } catch (IOException e) {
+ Files.deleteIfExists(partial);
+ throw e;
+ }
+ }
+
+ /**
+ * {@return {@code true} when {@link #REMOTE_DOWNLOAD_PROPERTY} is the string
+ * {@code true}, ignoring case}
+ */
+ public static boolean isRemoteDownloadEnabled() {
+ return Boolean.parseBoolean(System.getProperty(REMOTE_DOWNLOAD_PROPERTY));
+ }
+
+ /**
+ * Reads a byte-budget override from a system property. Budget constants are
+ * initialized from it once at class load, so overrides must be set at JVM startup.
+ *
+ * @param property The system property name to read.
+ * @param fallback The value to use when the property is absent or invalid.
+ * @return The property's value when it parses as a positive {@code long}, otherwise
+ * {@code fallback}.
+ */
+ public static long configuredLimit(String property, long fallback) {
+ final String value = System.getProperty(property, "").trim();
+ if (!value.isEmpty()) {
+ try {
+ final long parsed = Long.parseLong(value);
+ if (parsed > 0) {
+ return parsed;
+ }
+ } catch (NumberFormatException ignore) {
+ // Fall through to the default.
+ }
+ }
+ return fallback;
+ }
+
+ /**
+ * Opens a connection to {@code source} with connect and read timeouts applied.
+ *
+ * @param source The absolute URI to connect to.
+ * @return The configured, not yet connected, connection.
+ * @throws IOException Thrown if no connection can be created for {@code source}.
+ */
+ private static URLConnection open(URI source) throws IOException {
+ final URLConnection connection = source.toURL().openConnection();
+ connection.setConnectTimeout(CONNECT_TIMEOUT_MS);
+ connection.setReadTimeout(READ_TIMEOUT_MS);
+ return connection;
+ }
+
+ /**
+ * Trims and lowercases a SHA-512 hex digest.
+ *
+ * @param expectedSha512 The digest to normalize.
+ * @return The digest as 128 lowercase hex digits.
+ * @throws IllegalArgumentException Thrown if the digest is not 128 hex digits.
+ */
+ private static String normalizeSha512(String expectedSha512) {
+ final String hex = expectedSha512.trim().toLowerCase(Locale.ROOT);
+ if (hex.length() != SHA512_HEX_LENGTH || !hex.chars().allMatch(
+ c -> c >= '0' && c <= '9' || c >= 'a' && c <= 'f')) {
+ throw new IllegalArgumentException(
+ "expectedSha512 must be 128 hexadecimal digits");
+ }
+ return hex;
+ }
+
+ /**
+ * {@return a fresh SHA-512 {@link MessageDigest}}
+ *
+ * @throws IOException Thrown if the JVM does not provide the algorithm.
+ */
+ private static MessageDigest sha512Digest() throws IOException {
+ try {
+ return MessageDigest.getInstance("SHA-512");
+ } catch (NoSuchAlgorithmException e) {
+ throw new IOException("SHA-512 algorithm not found", e);
+ }
+ }
+
public static Map<String, Map<ModelType, URL>> getAvailableModels() {
if (availableModels == null) {
try {
diff --git a/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/stemmer/hunspell/HunspellDictionaryDownloadTest.java b/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/stemmer/hunspell/HunspellDictionaryDownloadTest.java
new file mode 100644
index 000000000..67d85af69
--- /dev/null
+++ b/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/stemmer/hunspell/HunspellDictionaryDownloadTest.java
@@ -0,0 +1,148 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package opennlp.tools.stemmer.hunspell;
+
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.Path;
+
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+
+import opennlp.tools.util.DictionaryCatalog;
+import opennlp.tools.util.DigestTestUtil;
+import opennlp.tools.util.DownloadUtil;
+
+/**
+ * Pins the Hunspell catalog download gate; network fetches are not exercised here.
+ */
+public class HunspellDictionaryDownloadTest {
+
+ /**
+ * Verifies that a catalog download without the remote-download property fails with
+ * the property name in the message, leaving the previous property value restored.
+ *
+ * @param target A scratch directory managed by the test framework.
+ * @throws IOException Thrown if the local catalog cannot be prepared.
+ */
+ @Test
+ void testDownloadRequiresRemoteProperty(@TempDir Path target) throws IOException {
+ final DictionaryCatalog catalog = localCatalog(target);
+ final String previous = System.getProperty(DownloadUtil.REMOTE_DOWNLOAD_PROPERTY);
+ System.clearProperty(DownloadUtil.REMOTE_DOWNLOAD_PROPERTY);
+ try {
+ final IOException e = Assertions.assertThrows(IOException.class,
+ () -> HunspellDictionaryDownload.downloadFromCatalog(catalog, "demo", target));
+ Assertions.assertTrue(e.getMessage().contains(DownloadUtil.REMOTE_DOWNLOAD_PROPERTY));
+ } finally {
+ restore(previous);
+ }
+ }
+
+ /**
+ * Verifies that an application-supplied catalog downloads a Hunspell pair and its
+ * license readme under the names declared by that catalog.
+ *
+ * @param target A scratch directory managed by the test framework.
+ * @throws IOException Thrown if the local catalog cannot be prepared or downloaded.
+ */
+ @Test
+ void testDownloadsFromApplicationCatalog(@TempDir Path target) throws IOException {
+ final DictionaryCatalog catalog = localCatalog(target);
+ final Path output = target.resolve("output");
+ final String previous = System.getProperty(DownloadUtil.REMOTE_DOWNLOAD_PROPERTY);
+ System.setProperty(DownloadUtil.REMOTE_DOWNLOAD_PROPERTY, "true");
+ try {
+ HunspellDictionaryDownload.downloadFromCatalog(catalog, "demo", output);
+ Assertions.assertEquals("SET UTF-8\n",
+ Files.readString(output.resolve("demo" + HunspellDictionary.AFFIX_FILE_SUFFIX)));
+ Assertions.assertEquals("1\nword\n",
+ Files.readString(output.resolve("demo" + HunspellDictionary.DICTIONARY_FILE_SUFFIX)));
+ Assertions.assertEquals("license\n", Files.readString(output.resolve("README.txt")));
+ } finally {
+ restore(previous);
+ }
+ }
+
+ /**
+ * Verifies that each required parameter is checked before a download begins.
+ *
+ * @param target A scratch directory managed by the test framework.
+ * @throws IOException Thrown if the local catalog cannot be prepared.
+ */
+ @Test
+ void testRejectsNullParameters(@TempDir Path target) throws IOException {
+ final DictionaryCatalog catalog = localCatalog(target);
+ Assertions.assertThrows(IllegalArgumentException.class,
+ () -> HunspellDictionaryDownload.downloadFromCatalog(null, "demo", target));
+ Assertions.assertThrows(IllegalArgumentException.class,
+ () -> HunspellDictionaryDownload.downloadFromCatalog(catalog, null, target));
+ Assertions.assertThrows(IllegalArgumentException.class,
+ () -> HunspellDictionaryDownload.downloadFromCatalog(catalog, "demo", null));
+ }
+
+ /**
+ * Creates an application-supplied catalog backed by local files.
+ *
+ * @param directory The directory to hold the source files.
+ * @return The loaded catalog. Never {@code null}.
+ * @throws IOException Thrown if the source files cannot be written.
+ */
+ private static DictionaryCatalog localCatalog(Path directory) throws IOException {
+ final byte[] affix = "SET UTF-8\n".getBytes(StandardCharsets.UTF_8);
+ final byte[] dictionary = "1\nword\n".getBytes(StandardCharsets.UTF_8);
+ final byte[] readme = "license\n".getBytes(StandardCharsets.UTF_8);
+ final Path affixSource = directory.resolve(
+ "source" + HunspellDictionary.AFFIX_FILE_SUFFIX);
+ final Path dictionarySource = directory.resolve(
+ "source" + HunspellDictionary.DICTIONARY_FILE_SUFFIX);
+ final Path readmeSource = directory.resolve("source-readme.txt");
+ Files.write(affixSource, affix);
+ Files.write(dictionarySource, dictionary);
+ Files.write(readmeSource, readme);
+
+ final String prefix = "hunspell.demo";
+ final String catalog = entry(prefix + HunspellDictionary.AFFIX_FILE_SUFFIX,
+ affixSource, affix, "demo" + HunspellDictionary.AFFIX_FILE_SUFFIX)
+ + entry(prefix + HunspellDictionary.DICTIONARY_FILE_SUFFIX,
+ dictionarySource, dictionary,
+ "demo" + HunspellDictionary.DICTIONARY_FILE_SUFFIX)
+ + entry(prefix + ".readme", readmeSource, readme, "README.txt");
+ return DictionaryCatalog.load(
+ new ByteArrayInputStream(catalog.getBytes(StandardCharsets.UTF_8)));
+ }
+
+ /** Builds one properties entry for a local source file. */
+ private static String entry(String id, Path source, byte[] content, String filename) {
+ return id + ".url=" + source.toUri() + "\n"
+ + id + ".sha512=" + DigestTestUtil.sha512(content) + "\n"
+ + id + ".filename=" + filename + "\n";
+ }
+
+ /** Restores the remote-download property to its previous value. */
+ private static void restore(String previous) {
+ if (previous == null) {
+ System.clearProperty(DownloadUtil.REMOTE_DOWNLOAD_PROPERTY);
+ } else {
+ System.setProperty(DownloadUtil.REMOTE_DOWNLOAD_PROPERTY, previous);
+ }
+ }
+}
diff --git a/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/stemmer/hunspell/HunspellManualExampleTest.java b/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/stemmer/hunspell/HunspellManualExampleTest.java
new file mode 100644
index 000000000..21daec7e0
--- /dev/null
+++ b/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/stemmer/hunspell/HunspellManualExampleTest.java
@@ -0,0 +1,73 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package opennlp.tools.stemmer.hunspell;
+
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.List;
+
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Test;
+
+import opennlp.tools.stemmer.Stemmer;
+
+/**
+ * Runs the manual's Hunspell examples (docbkx {@code stemmer.xml}) verbatim: every
+ * value the chapter states is asserted here, so a change breaking this test breaks the
+ * manual. The fixture dictionary is authored inside this class; no external dictionary
+ * data is involved.
+ */
+public class HunspellManualExampleTest {
+
+ /**
+ * Affix fixture matching the chapter: agentive {@code -er} with continuation class
+ * {@code S}, and the plural {@code -s}.
+ */
+ private static final String AFFIX = String.join("\n",
+ "SET UTF-8",
+ "SFX E Y 1",
+ "SFX E 0 er/S .",
+ "SFX S Y 1",
+ "SFX S 0 s [^sxy]",
+ "");
+
+ /** Word-list fixture: {@code work} accepts both suffixes. */
+ private static final String WORDS = "1\nwork/ES\n";
+
+ /**
+ * Loads the chapter's miniature dictionary, stems through a factory-minted stemmer,
+ * and asserts the exact stems the manual prints.
+ *
+ * @throws IOException Thrown if the in-memory fixture fails to load.
+ */
+ @Test
+ void testLoadAndStemWorkers() throws IOException {
+ final HunspellDictionary dictionary = HunspellDictionary.load(
+ new ByteArrayInputStream(AFFIX.getBytes(StandardCharsets.UTF_8)),
+ new ByteArrayInputStream(WORDS.getBytes(StandardCharsets.UTF_8)));
+ final Stemmer stemmer = new HunspellStemmerFactory(dictionary).newStemmer();
+
+ Assertions.assertEquals("work", stemmer.stem("workers").toString());
+ Assertions.assertEquals("work", stemmer.stem("worker").toString());
+ Assertions.assertEquals(List.of("work"),
+ stemmer.stemAll("workers").stream().map(CharSequence::toString).toList());
+ // unknown vocabulary passes through unchanged
+ Assertions.assertEquals("table", stemmer.stem("table").toString());
+ }
+}
diff --git a/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/stemmer/hunspell/HunspellRealDictionaryTest.java b/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/stemmer/hunspell/HunspellRealDictionaryTest.java
new file mode 100644
index 000000000..9221658dd
--- /dev/null
+++ b/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/stemmer/hunspell/HunspellRealDictionaryTest.java
@@ -0,0 +1,126 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package opennlp.tools.stemmer.hunspell;
+
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Assumptions;
+import org.junit.jupiter.api.Test;
+
+/**
+ * Gated checks against published dictionaries, which are never bundled: the tests run
+ * only when {@code -Dopennlp.hunspell.dict.dir} names a directory holding
+ * {@code <name>.aff}/{@code <name>.dic} pairs, and each test additionally skips when
+ * its dictionary pair is absent. The download helper in {@code dev/} fetches the pairs
+ * together with their license files; see {@code dev/README-hunspell-dictionaries.md}.
+ *
+ * <p>The assertions are limited to morphology stable across dictionary revisions:
+ * everyday inflections, and for German the decomposability of ordinary compounds.</p>
+ */
+public class HunspellRealDictionaryTest {
+
+ private static final String DICT_DIR_PROPERTY = "opennlp.hunspell.dict.dir";
+
+ /**
+ * Loads one dictionary pair from the gated directory, skipping the test when the
+ * gate or the pair is absent.
+ *
+ * @param name The dictionary base name, such as {@code en_US}.
+ * @return A stemmer over the loaded pair. Never {@code null}.
+ * @throws IOException Thrown if a present pair fails to load, which is a failure,
+ * not a skip.
+ */
+ private static HunspellStemmer loadOrSkip(String name) throws IOException {
+ final String dir = System.getProperty(DICT_DIR_PROPERTY);
+ Assumptions.assumeTrue(dir != null && !dir.isBlank(),
+ "no " + DICT_DIR_PROPERTY + " given");
+ final Path affix = Path.of(dir, name + HunspellDictionary.AFFIX_FILE_SUFFIX);
+ final Path words = Path.of(dir, name + HunspellDictionary.DICTIONARY_FILE_SUFFIX);
+ Assumptions.assumeTrue(Files.isReadable(affix) && Files.isReadable(words),
+ name + " pair not present under " + dir);
+ return new HunspellStemmer(HunspellDictionary.load(affix, words));
+ }
+
+ /**
+ * Checks everyday English inflections against {@code en_US}, plus the identity
+ * fallback on vocabulary no dictionary lists.
+ *
+ * @throws IOException Thrown if a present dictionary pair fails to load.
+ */
+ @Test
+ void testEnglishInflections() throws IOException {
+ final HunspellStemmer stemmer = loadOrSkip("en_US");
+ Assertions.assertEquals("worker", stemmer.stem("workers").toString());
+ Assertions.assertEquals("cat", stemmer.stem("cats").toString());
+ Assertions.assertEquals("unhappy", stemmer.stem("unhappiest").toString());
+ Assertions.assertEquals("quick", stemmer.stem("quickly").toString());
+ Assertions.assertEquals("look", stemmer.stem("looked").toString());
+ // unknown vocabulary degrades to identity
+ Assertions.assertEquals("zyzzyvax", stemmer.stem("zyzzyvax").toString());
+ }
+
+ /**
+ * Checks everyday German inflections against {@code de_DE_frami}: a plural, an
+ * umlauted plural, and a superlative.
+ *
+ * @throws IOException Thrown if a present dictionary pair fails to load.
+ */
+ @Test
+ void testGermanInflections() throws IOException {
+ final HunspellStemmer stemmer = loadOrSkip("de_DE_frami");
+ Assertions.assertEquals("Kind", stemmer.stem("Kinder").toString());
+ // Haeuser, written with a-umlaut, stems to Haus
+ Assertions.assertEquals("Haus", stemmer.stem("H\u00E4user").toString());
+ Assertions.assertEquals("schnell", stemmer.stem("schnellsten").toString());
+ }
+
+ /**
+ * Checks that ordinary German compounds decompose against {@code de_DE_frami}. Only
+ * the part count is asserted: the exact part spellings follow the dictionary's own
+ * entries and may shift between its revisions.
+ *
+ * @throws IOException Thrown if a present dictionary pair fails to load.
+ */
+ @Test
+ void testGermanCompoundsDecompose() throws IOException {
+ final HunspellStemmer stemmer = loadOrSkip("de_DE_frami");
+ // Haustuer, written with u-umlaut, is Haus + Tuer
+ Assertions.assertTrue(stemmer.stemAll("Haust\u00FCr").size() >= 2);
+ Assertions.assertTrue(stemmer.stemAll("Kinderzimmer").size() >= 2);
+ Assertions.assertTrue(stemmer.stemAll("Abbildungsverzeichnis").size() >= 2);
+ }
+
+ /**
+ * Checks everyday Hungarian inflections against {@code hu_HU}: a plural and two
+ * case-suffixed forms.
+ *
+ * @throws IOException Thrown if a present dictionary pair fails to load.
+ */
+ @Test
+ void testHungarianInflections() throws IOException {
+ final HunspellStemmer stemmer = loadOrSkip("hu_HU");
+ // kutyak, written with a-acute, is the plural of kutya
+ Assertions.assertEquals("kutya", stemmer.stem("kuty\u00E1k").toString());
+ Assertions.assertEquals("asztal", stemmer.stem("asztalon").toString());
+ // konyveket, written with o-umlaut, is an inflected form of konyv
+ Assertions.assertEquals("k\u00F6nyv", stemmer.stem("k\u00F6nyveket").toString());
+ }
+}
diff --git a/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/stemmer/hunspell/HunspellStemmerFactoryTest.java b/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/stemmer/hunspell/HunspellStemmerFactoryTest.java
new file mode 100644
index 000000000..6907fd143
--- /dev/null
+++ b/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/stemmer/hunspell/HunspellStemmerFactoryTest.java
@@ -0,0 +1,183 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package opennlp.tools.stemmer.hunspell;
+
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+
+import opennlp.tools.stemmer.Stemmer;
+
+/**
+ * Demonstrates the intended end-to-end usage of the Hunspell stemming classes: a user
+ * writes (or ships) a {@code .aff}/{@code .dic} file pair, loads it once into a
+ * {@link HunspellDictionary}, wraps the dictionary in a {@link HunspellStemmerFactory},
+ * and obtains {@link Stemmer} instances from the factory wherever stemming is needed.
+ * The fixture dictionary is authored inside this test class, so no external dictionary
+ * data is involved.
+ */
+public class HunspellStemmerFactoryTest {
+
+ /**
+ * The affix fixture: the prefix {@code re-}, the suffix {@code -er} whose continuation
+ * class {@code S} lets the plural {@code -s} stack on top of it, and the plural
+ * {@code -s} itself, restricted to stems not ending in {@code s}, {@code x}, or
+ * {@code y}. All three rules opt into cross-product combination.
+ */
+ private static final String AFFIX = String.join("\n",
+ "# project-authored test fixture",
+ "SET UTF-8",
+ "",
+ "PFX R Y 1",
+ "PFX R 0 re .",
+ "",
+ "SFX E Y 1",
+ "SFX E 0 er/S .",
+ "",
+ "SFX S Y 1",
+ "SFX S 0 s [^sxy]",
+ "");
+
+ /**
+ * The word-list fixture: {@code work} accepts the prefix and both suffixes,
+ * {@code paint} accepts only the agentive {@code -er}.
+ */
+ private static final String WORDS = String.join("\n",
+ "2",
+ "work/RES",
+ "paint/E",
+ "");
+
+ /**
+ * Writes the fixture dictionary pair into a directory and loads it through the
+ * file-based {@link HunspellDictionary#load(Path, Path)} entry point.
+ *
+ * @param directory The directory to write into.
+ * @return The loaded dictionary. Never {@code null}.
+ * @throws IOException Thrown if writing or loading fails.
+ */
+ private static HunspellDictionary writeAndLoadFixture(Path directory) throws IOException {
+ final Path affixFile = directory.resolve(
+ "fixture" + HunspellDictionary.AFFIX_FILE_SUFFIX);
+ final Path dictionaryFile = directory.resolve(
+ "fixture" + HunspellDictionary.DICTIONARY_FILE_SUFFIX);
+ Files.write(affixFile, AFFIX.getBytes(StandardCharsets.UTF_8));
+ Files.write(dictionaryFile, WORDS.getBytes(StandardCharsets.UTF_8));
+ return HunspellDictionary.load(affixFile, dictionaryFile);
+ }
+
+ /**
+ * Walks the whole intended flow on a single thread: files on disk, one dictionary,
+ * one factory, one stemmer, and exact stems for a prefixed form, a suffixed form, a
+ * twofold suffix chain, a cross-product form, an in-dictionary word, and an unknown
+ * word.
+ *
+ * @param tempDir A scratch directory managed by the test framework.
+ * @throws IOException Thrown if the fixture cannot be written or loaded.
+ */
+ @Test
+ void testEndToEndUsageFromFiles(@TempDir Path tempDir) throws IOException {
+ final HunspellDictionary dictionary = writeAndLoadFixture(tempDir);
+ final HunspellStemmerFactory factory = new HunspellStemmerFactory(dictionary);
+ final Stemmer stemmer = factory.newStemmer();
+
+ // one suffix removed
+ Assertions.assertEquals("work", stemmer.stem("worker").toString());
+ Assertions.assertEquals("paint", stemmer.stem("painter").toString());
+ // twofold suffixes: -s stacks on -er through the continuation class S
+ Assertions.assertEquals("work", stemmer.stem("workers").toString());
+ // one prefix removed
+ Assertions.assertEquals("work", stemmer.stem("rework").toString());
+ // cross product: the prefix re- and the suffix -s on the same stem
+ Assertions.assertEquals("work", stemmer.stem("reworks").toString());
+ // a word that is itself listed stems to itself
+ Assertions.assertEquals("work", stemmer.stem("work").toString());
+ // unknown vocabulary passes through unchanged
+ Assertions.assertEquals("table", stemmer.stem("table").toString());
+ }
+
+ /**
+ * Shares one factory between two threads: each thread obtains its own stemmer
+ * instance from the factory and stems the same inputs. The test asserts that the two
+ * instances are distinct objects and that their results are identical to each other
+ * and to the expected stems.
+ *
+ * @param tempDir A scratch directory managed by the test framework.
+ * @throws Exception Thrown if the fixture cannot be loaded or a worker fails.
+ */
+ @Test
+ void testFactorySharedAcrossThreads(@TempDir Path tempDir) throws Exception {
+ final HunspellStemmerFactory factory =
+ new HunspellStemmerFactory(writeAndLoadFixture(tempDir));
+ final List<String> inputs = List.of("workers", "reworks", "painter", "table");
+ final List<String> expected = List.of("work", "work", "paint", "table");
+
+ final Stemmer[] created = new Stemmer[2];
+ final ExecutorService pool = Executors.newFixedThreadPool(2);
+ try {
+ final List<Future<List<String>>> futures = new ArrayList<>(2);
+ for (int worker = 0; worker < 2; worker++) {
+ final int slot = worker;
+ futures.add(pool.submit(() -> {
+ final Stemmer stemmer = factory.newStemmer();
+ created[slot] = stemmer;
+ final List<String> stems = new ArrayList<>(inputs.size());
+ for (final String input : inputs) {
+ stems.add(stemmer.stem(input).toString());
+ }
+ return stems;
+ }));
+ }
+ final List<String> first = futures.get(0).get();
+ final List<String> second = futures.get(1).get();
+ Assertions.assertEquals(expected, first);
+ Assertions.assertEquals(expected, second);
+ } finally {
+ pool.shutdownNow();
+ }
+ Assertions.assertNotSame(created[0], created[1]);
+ }
+
+ /**
+ * Verifies that the file-based entry point rejects each {@code null} path with the
+ * documented exception naming the offending argument.
+ *
+ * @param tempDir A scratch directory managed by the test framework.
+ */
+ @Test
+ void testNullPathsAreRejected(@TempDir Path tempDir) {
+ final Path present = tempDir.resolve("present" + HunspellDictionary.AFFIX_FILE_SUFFIX);
+ IllegalArgumentException e = Assertions.assertThrows(IllegalArgumentException.class,
+ () -> HunspellDictionary.load(null, present));
+ Assertions.assertEquals("affixFile must not be null", e.getMessage());
+
+ e = Assertions.assertThrows(IllegalArgumentException.class,
+ () -> HunspellDictionary.load(present, null));
+ Assertions.assertEquals("dictionaryFile must not be null", e.getMessage());
+ }
+}
diff --git a/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/stemmer/hunspell/HunspellStemmerTest.java b/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/stemmer/hunspell/HunspellStemmerTest.java
new file mode 100644
index 000000000..8657b7e8c
--- /dev/null
+++ b/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/stemmer/hunspell/HunspellStemmerTest.java
@@ -0,0 +1,1400 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package opennlp.tools.stemmer.hunspell;
+
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.charset.Charset;
+import java.nio.charset.StandardCharsets;
+import java.util.Arrays;
+import java.util.List;
+
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.CsvSource;
+import org.junit.jupiter.params.provider.ValueSource;
+
+import opennlp.tools.stemmer.Stemmer;
+
+/**
+ * Tests the affix engine against a project-authored miniature dictionary; no external
+ * dictionary data is involved.
+ */
+public class HunspellStemmerTest {
+
+ private static final String AFFIX = String.join("\n",
+ "# project-authored test fixture",
+ "SET UTF-8",
+ "",
+ "PFX U Y 1",
+ "PFX U 0 un .",
+ "",
+ "SFX S Y 3",
+ "SFX S 0 s [^sxy]",
+ "SFX S y ies y",
+ "SFX S 0 es [sx]",
+ "",
+ "SFX G Y 2",
+ "SFX G 0 ing [^e]",
+ "SFX G e ing e",
+ "");
+
+ private static final String WORDS = String.join("\n",
+ "6",
+ "lock/USG",
+ "pony/S",
+ "make/G",
+ "cat/S",
+ "box/S",
+ "fish",
+ "");
+
+ private static HunspellStemmer stemmer;
+
+ /**
+ * Loads the shared fixture dictionary once for the tests that stem against it.
+ *
+ * @throws IOException Thrown if the fixture fails to load.
+ */
+ @BeforeAll
+ static void loadDictionary() throws IOException {
+ stemmer = new HunspellStemmer(load(AFFIX, WORDS));
+ }
+
+ /**
+ * Loads a dictionary from in-memory affix and word-list content, both encoded as
+ * UTF-8, through the stream-based entry point.
+ *
+ * @param affix The {@code .aff} content.
+ * @param words The {@code .dic} content.
+ * @return The loaded dictionary. Never {@code null}.
+ * @throws IOException Thrown if the content is malformed.
+ */
+ private static HunspellDictionary load(String affix, String words) throws IOException {
+ return load(affix, words, StandardCharsets.UTF_8);
+ }
+
+ /**
+ * Loads a dictionary from in-memory affix and word-list content encoded in the given
+ * charset, through the stream-based entry point.
+ *
+ * @param affix The {@code .aff} content.
+ * @param words The {@code .dic} content.
+ * @param charset The charset both contents are encoded with.
+ * @return The loaded dictionary. Never {@code null}.
+ * @throws IOException Thrown if the content is malformed.
+ */
+ private static HunspellDictionary load(String affix, String words, Charset charset)
+ throws IOException {
+ return HunspellDictionary.load(new ByteArrayInputStream(affix.getBytes(charset)),
+ new ByteArrayInputStream(words.getBytes(charset)));
+ }
+
+ /**
+ * Verifies the fixture's suffix rules: the plural {@code -s}, the {@code y} to
+ * {@code ies} replacement, the {@code -es} plural after a sibilant, and the
+ * progressive {@code -ing} with and without the silent {@code e}.
+ *
+ * @param word The surface form to stem.
+ * @param expected The stem the fixture licenses.
+ */
+ @ParameterizedTest
+ @CsvSource({"cats,cat", "ponies,pony", "boxes,box", "making,make", "locking,lock"})
+ void testSuffixRules(String word, String expected) {
+ Assertions.assertEquals(expected, stemmer.stem(word).toString());
+ }
+
+ /**
+ * Verifies prefix removal alone and combined with a suffix through the cross-product
+ * marker both rules declare.
+ *
+ * @param word The surface form to stem.
+ * @param expected The stem the fixture licenses.
+ */
+ @ParameterizedTest
+ @CsvSource({"unlock,lock", "unlocks,lock", "unlocking,lock"})
+ void testPrefixAndCrossProduct(String word, String expected) {
+ Assertions.assertEquals(expected, stemmer.stem(word).toString());
+ }
+
+ /**
+ * Verifies that an analysis a rule condition or a missing flag rejects is not
+ * reported: {@code boxs} fails the {@code [^sxy]} condition of the {@code -s} rule,
+ * {@code cat} carries no {@code G} flag, and {@code fish} carries no flag at all, so
+ * each surface form falls through unchanged.
+ *
+ * @param word The surface form to stem.
+ */
+ @ParameterizedTest
+ @CsvSource({"boxs", "cating", "fishs"})
+ void testConditionsBlockWrongAnalyses(String word) {
+ Assertions.assertEquals(word, stemmer.stem(word).toString());
+ }
+
+ /**
+ * Verifies that a listed word stems to itself and that a capitalized surface form is
+ * analyzed through its lowercase variant.
+ *
+ * @param word The surface form to stem.
+ * @param expected The stem the fixture licenses.
+ */
+ @ParameterizedTest
+ @CsvSource({"fish,fish", "Cats,cat", "Unlocks,lock"})
+ void testDirectLookupAndCase(String word, String expected) {
+ Assertions.assertEquals(expected, stemmer.stem(word).toString());
+ }
+
+ /** Verifies that a word with no analysis is returned unchanged as its only analysis. */
+ @Test
+ void testUnknownWordsPassThroughUnchanged() {
+ Assertions.assertEquals("zebras", stemmer.stem("zebras").toString());
+ Assertions.assertEquals(1, stemmer.stemAll("zebras").size());
+ }
+
+ /**
+ * Verifies that {@link HunspellStemmer#stemAll(CharSequence)} reports the analyses
+ * and that {@link HunspellStemmer#stem(CharSequence)} answers the first of them.
+ */
+ @Test
+ void testStemAllReportsEveryAnalysis() {
+ Assertions.assertEquals(1, stemmer.stemAll("unlocks").size());
+ Assertions.assertEquals("lock", stemmer.stemAll("unlocks").get(0).toString());
+ // the surface form itself is an entry AND an analysis target
+ Assertions.assertEquals("lock", stemmer.stemAll("lock").get(0).toString());
+ }
+
+ /**
+ * Verifies twofold suffix removal: the plural {@code -s} stacks on the comparative
+ * {@code -er} through the continuation class the outer rule declares, while the inner
+ * flag alone licenses nothing because no entry carries it.
+ *
+ * @throws IOException Thrown if the fixture fails to load.
+ */
+ @Test
+ void testTwofoldSuffixesThroughContinuationClasses() throws IOException {
+ final HunspellStemmer twofold = new HunspellStemmer(load(String.join("\n",
+ "SET UTF-8",
+ "SFX A Y 1",
+ "SFX A 0 er/B .",
+ "SFX B Y 1",
+ "SFX B 0 s .",
+ ""), String.join("\n", "1", "kind/A", "")));
+
+ Assertions.assertEquals("kind", twofold.stem("kinder").toString());
+ Assertions.assertEquals("kind", twofold.stem("kinders").toString());
+ // B alone never applies: no entry carries it directly
+ Assertions.assertEquals("kinds", twofold.stem("kinds").toString());
+ }
+
+ /**
+ * Verifies {@code FLAG num} mode: a comma-separated run of decimal numbers is the
+ * entry's flag set, and an affix block named by one of them applies.
+ *
+ * @throws IOException Thrown if the fixture fails to load.
+ */
+ @Test
+ void testNumericFlagMode() throws IOException {
+ final HunspellDictionary dictionary = load(String.join("\n",
+ "SET UTF-8",
+ "FLAG num",
+ "SFX 100 Y 1",
+ "SFX 100 0 s .",
+ ""), String.join("\n", "1", "walk/100,7", ""));
+ Assertions.assertEquals("walk",
+ new HunspellStemmer(dictionary).stem("walks").toString());
+ }
+
+ /**
+ * Verifies {@code FLAG long} mode: each pair of characters in the run is one flag,
+ * and an affix block named by such a pair applies.
+ *
+ * @throws IOException Thrown if the fixture fails to load.
+ */
+ @Test
+ void testLongFlagMode() throws IOException {
+ final HunspellDictionary dictionary = load(String.join("\n",
+ "SET UTF-8",
+ "FLAG long",
+ "SFX Aa Y 1",
+ "SFX Aa 0 s .",
+ ""), String.join("\n", "1", "walk/AaBb", ""));
+ Assertions.assertEquals("walk",
+ new HunspellStemmer(dictionary).stem("walks").toString());
+ }
+
+ /**
+ * Verifies that a stemmer minted by the factory analyzes against the same dictionary.
+ *
+ * @throws IOException Thrown if the fixture fails to load.
+ */
+ @Test
+ void testFactoryHandsOutWorkingStemmers() throws IOException {
+ final Stemmer fresh = new HunspellStemmerFactory(load(AFFIX, WORDS)).newStemmer();
+ Assertions.assertEquals("pony", fresh.stem("ponies").toString());
+ }
+
+ /**
+ * Verifies that cross-product combination of a prefix with a suffix only happens
+ * when both rules declare the cross-product marker {@code Y}. Removing just the one
+ * affix whose rule exists keeps working; the combined form must not be analyzed.
+ *
+ * @throws IOException Thrown if a fixture fails to load.
+ */
+ @Test
+ void testCrossProductRequiresBothRulesOptIn() throws IOException {
+ // the prefix rule declares N, so it never combines with the suffix
+ final HunspellStemmer prefixOptedOut = new HunspellStemmer(load(String.join("\n",
+ "PFX U N 1",
+ "PFX U 0 un .",
+ "SFX S Y 1",
+ "SFX S 0 s .",
+ ""), "1\nlock/US\n"));
+ Assertions.assertEquals("lock", prefixOptedOut.stem("unlock").toString());
+ Assertions.assertEquals("lock", prefixOptedOut.stem("locks").toString());
+ Assertions.assertEquals("unlocks", prefixOptedOut.stem("unlocks").toString());
+
+ // the suffix rule declares N, so the combined form is likewise not analyzed
+ final HunspellStemmer suffixOptedOut = new HunspellStemmer(load(String.join("\n",
+ "PFX U Y 1",
+ "PFX U 0 un .",
+ "SFX S N 1",
+ "SFX S 0 s .",
+ ""), "1\nlock/US\n"));
+ Assertions.assertEquals("lock", suffixOptedOut.stem("unlock").toString());
+ Assertions.assertEquals("lock", suffixOptedOut.stem("locks").toString());
+ Assertions.assertEquals("unlocks", suffixOptedOut.stem("unlocks").toString());
+ }
+
+ /**
+ * Verifies that a non-negated character class rejects a candidate stem: the
+ * {@code es} rule requires a stem ending in {@code s} or {@code x}, so removing
+ * {@code es} from {@code cates} produces {@code cat}, which the class rejects, and
+ * the surface form falls through unchanged.
+ */
+ @Test
+ void testPositiveCharacterClassRejectsCandidate() {
+ Assertions.assertEquals("cates", stemmer.stem("cates").toString());
+ Assertions.assertEquals(1, stemmer.stemAll("cates").size());
+ }
+
+ /**
+ * Verifies that the {@code SET} declaration selects the charset both files are
+ * decoded with: a word list holding the byte {@code 0xE9} only maps to the word
+ * caf\u00E9 (e with acute accent) when decoded as ISO-8859-1, as the affix file declares.
+ *
+ * @throws IOException Thrown if the fixture fails to load.
+ */
+ @Test
+ void testSetDeclarationSelectsEncoding() throws IOException {
+ final Charset latin1 = StandardCharsets.ISO_8859_1;
+ final HunspellDictionary dictionary = load(String.join("\n",
+ "SET ISO8859-1",
+ "SFX S Y 1",
+ "SFX S 0 s .",
+ ""), "1\ncaf\u00E9/S\n", latin1);
+ final HunspellStemmer latin1Stemmer = new HunspellStemmer(dictionary);
+ Assertions.assertEquals("caf\u00E9", latin1Stemmer.stem("caf\u00E9s").toString());
+ Assertions.assertEquals("caf\u00E9", latin1Stemmer.stem("caf\u00E9").toString());
+ }
+
+ /**
+ * Verifies that continuation classes also work in {@code FLAG long} mode, where a
+ * flag is a two-character run: the plural {@code Bb} stacks on the agentive
+ * {@code Aa} to analyze a twofold suffix chain.
+ *
+ * @throws IOException Thrown if the fixture fails to load.
+ */
+ @Test
+ void testLongFlagContinuation() throws IOException {
+ final HunspellStemmer longFlags = new HunspellStemmer(load(String.join("\n",
+ "FLAG long",
+ "SFX Aa Y 1",
+ "SFX Aa 0 er/Bb .",
+ "SFX Bb Y 1",
+ "SFX Bb 0 s .",
+ ""), "1\nkind/Aa\n"));
+ Assertions.assertEquals("kind", longFlags.stem("kinder").toString());
+ Assertions.assertEquals("kind", longFlags.stem("kinders").toString());
+ }
+
+ /**
+ * Verifies that cross-product prefix and suffix combination also works in
+ * {@code FLAG num} mode, where flags are comma-separated decimal numbers.
+ *
+ * @throws IOException Thrown if the fixture fails to load.
+ */
+ @Test
+ void testNumericFlagCrossProduct() throws IOException {
+ final HunspellStemmer numericFlags = new HunspellStemmer(load(String.join("\n",
+ "FLAG num",
+ "PFX 1 Y 1",
+ "PFX 1 0 un .",
+ "SFX 2 Y 1",
+ "SFX 2 0 s .",
+ ""), "1\nlock/1,2\n"));
+ Assertions.assertEquals("lock", numericFlags.stem("unlock").toString());
+ Assertions.assertEquals("lock", numericFlags.stem("locks").toString());
+ Assertions.assertEquals("lock", numericFlags.stem("unlocks").toString());
+ }
+
+ /**
+ * Verifies the exact exception and message for each malformed {@code FLAG}
+ * declaration the parser detects: a missing mode and an unrecognized mode name.
+ */
+ @Test
+ void testMalformedFlagDeclarationMessages() {
+ IOException e = Assertions.assertThrows(IOException.class,
+ () -> load("FLAG\n", "0\n"));
+ Assertions.assertEquals("FLAG line without a mode at line 1", e.getMessage());
+
+ e = Assertions.assertThrows(IOException.class, () -> load("FLAG short\n", "0\n"));
+ Assertions.assertEquals("unsupported FLAG mode 'short' at line 1", e.getMessage());
+ }
+
+ /**
+ * Verifies the exact exception and message for each malformed affix block the
+ * parser detects: a header with too few fields, a non-numeric rule count, a block
+ * with fewer rule lines than its count announces, a rule line whose type tag does
+ * not match its header, and an unterminated character class in a condition.
+ */
+ @Test
+ void testMalformedAffixBlockMessages() {
+ IOException e = Assertions.assertThrows(IOException.class,
+ () -> load("PFX U Y\n", "0\n"));
+ Assertions.assertEquals("malformed affix header at line 1", e.getMessage());
+
+ e = Assertions.assertThrows(IOException.class,
+ () -> load("SFX S Y many\nSFX S 0 s .\n", "0\n"));
+ Assertions.assertEquals("malformed affix rule count at line 1", e.getMessage());
+
+ e = Assertions.assertThrows(IOException.class,
+ () -> load("SFX S Y 2\nSFX S 0 s .", "0\n"));
+ Assertions.assertEquals("affix block truncated at line 3", e.getMessage());
+
+ e = Assertions.assertThrows(IOException.class,
+ () -> load("SFX S Y 1\nPFX S 0 s .\n", "0\n"));
+ Assertions.assertEquals("malformed affix rule at line 2", e.getMessage());
+
+ e = Assertions.assertThrows(IOException.class,
+ () -> load("SFX S Y 1\nSFX S 0 s [ab\n", "0\n"));
+ Assertions.assertEquals("unterminated character class at line 2", e.getMessage());
+ }
+
+ /**
+ * Verifies the exact exception and message for each malformed flag value the parser
+ * detects: an odd-length flag run in {@code FLAG long} mode, a non-numeric flag in
+ * {@code FLAG num} mode, an affix header naming more than one flag, and a
+ * {@code SET} declaration naming an unknown encoding.
+ */
+ @Test
+ void testMalformedFlagValueMessages() {
+ IOException e = Assertions.assertThrows(IOException.class,
+ () -> load("FLAG long\n", "1\nwalk/AaB\n"));
+ Assertions.assertEquals("odd long-flag run at line 2", e.getMessage());
+
+ e = Assertions.assertThrows(IOException.class,
+ () -> load("FLAG num\n", "1\nwalk/12,x\n"));
+ Assertions.assertEquals("malformed numeric flag at line 2", e.getMessage());
+
+ e = Assertions.assertThrows(IOException.class,
+ () -> load("FLAG long\nSFX AaBb Y 1\nSFX AaBb 0 s .\n", "0\n"));
+ Assertions.assertEquals("expected exactly one flag at line 2", e.getMessage());
+
+ e = Assertions.assertThrows(IOException.class,
+ () -> load("SET NO-SUCH-ENCODING\n", "0\n"));
+ Assertions.assertEquals("unsupported SET encoding: NO-SUCH-ENCODING", e.getMessage());
+ }
+
+ /**
+ * Verifies that a morphological field is cut off the entry before the flag separator
+ * is looked for, so a slash inside a morphological field is not mistaken for the
+ * separator: the entry {@code walk po:verb/noun} registers the word {@code walk}
+ * with no flags in every flag mode, and its morphology is ignored.
+ *
+ * @throws IOException Thrown if a fixture fails to load.
+ */
+ @Test
+ void testMorphologicalFieldsAreCutBeforeTheFlagSeparator() throws IOException {
+ final HunspellDictionary chars = load("SFX G Y 1\nSFX G 0 ing .\n",
+ "1\nwalk po:verb/noun\n");
+ Assertions.assertNotNull(chars.lookup("walk"));
+ Assertions.assertEquals(0, chars.lookup("walk").get(0).length);
+ Assertions.assertNull(chars.lookup("walk po:verb"));
+
+ final HunspellDictionary numbers = load("FLAG num\nSFX 1 Y 1\nSFX 1 0 ing .\n",
+ "1\nwalk po:verb/noun\n");
+ Assertions.assertNotNull(numbers.lookup("walk"));
+ Assertions.assertEquals(0, numbers.lookup("walk").get(0).length);
+
+ // the tabulator is the older morphological field separator
+ final HunspellDictionary tabbed = load("SFX G Y 1\nSFX G 0 ing .\n",
+ "1\nwalk\tpo:verb/noun\n");
+ Assertions.assertNotNull(tabbed.lookup("walk"));
+ Assertions.assertEquals(0, tabbed.lookup("walk").get(0).length);
+ }
+
+ /**
+ * Verifies that an entry keeps its flags when it carries both a flag run and a
+ * morphological field holding a slash, in every flag mode.
+ *
+ * @throws IOException Thrown if a fixture fails to load.
+ */
+ @Test
+ void testFlaggedEntriesKeepTheirFlagsBesideMorphology() throws IOException {
+ final HunspellDictionary chars = load("SFX A Y 1\nSFX A 0 ing .\n",
+ "1\nwalk/AB po:verb/noun\n");
+ Assertions.assertArrayEquals(new int[] {'A', 'B'}, chars.lookup("walk").get(0));
+ Assertions.assertEquals("walk",
+ new HunspellStemmer(chars).stem("walking").toString());
+
+ final HunspellDictionary numbers = load("FLAG num\nSFX 1 Y 1\nSFX 1 0 ing .\n",
+ "1\nwalk/1,2 po:verb/noun\n");
+ Assertions.assertArrayEquals(new int[] {1, 2}, numbers.lookup("walk").get(0));
+ Assertions.assertEquals("walk",
+ new HunspellStemmer(numbers).stem("walking").toString());
+ }
+
+ /**
+ * Verifies that a multi-word entry keeps both its spaces and its flags: the word of
+ * a word-list entry runs up to its morphological fields, not up to its first space.
+ *
+ * @throws IOException Thrown if the fixture fails to load.
+ */
+ @Test
+ void testMultiWordEntriesKeepTheirSpacesAndFlags() throws IOException {
+ final HunspellDictionary dictionary = load("FLAG num\nSFX 39 Y 1\nSFX 39 0 s .\n",
+ "1\nall right/39\n");
+ Assertions.assertArrayEquals(new int[] {39}, dictionary.lookup("all right").get(0));
+ Assertions.assertNull(dictionary.lookup("all"));
+ }
+
+ /**
+ * Verifies that the parser trims word-list entries with the same whitespace
+ * definition it uses to find their fields: an entry edged by Unicode whitespace,
+ * leading or trailing, is registered under its real word, both with and without a
+ * flag run.
+ *
+ * @param space The whitespace character at the line edges: the no-break space
+ * U+00A0 and the ideographic space U+3000, both whitespace to
+ * {@code StringUtil.isWhitespace} but not to {@code String.trim()}.
+ * @throws IOException Thrown if a fixture fails to load.
+ */
+ @ParameterizedTest
+ @ValueSource(strings = {"\u00A0", "\u3000"})
+ void testEntriesEdgedByUnicodeWhitespaceAreTrimmed(String space) throws IOException {
+ final HunspellDictionary dictionary = load("SFX S Y 1\nSFX S 0 s .\n",
+ "2\n" + space + "fish" + space + "\n" + space + "cat/S" + space + "\n");
+ Assertions.assertNotNull(dictionary.lookup("fish"));
+ Assertions.assertNotNull(dictionary.lookup("cat"));
+ Assertions.assertNull(dictionary.lookup(""));
+ Assertions.assertEquals("cat", new HunspellStemmer(dictionary).stem("cats").toString());
+ }
+
+ /**
+ * Verifies that {@code FLAG UTF-8}, which declares single-character flags, is
+ * accepted and read exactly like the default single-character mode, including a flag
+ * outside ASCII.
+ *
+ * @throws IOException Thrown if a fixture fails to load.
+ */
+ @Test
+ void testUtf8FlagModeDeclaresSingleCharacterFlags() throws IOException {
+ final HunspellStemmer plain = new HunspellStemmer(load(String.join("\n",
+ "FLAG UTF-8",
+ "SFX S Y 1",
+ "SFX S 0 s .",
+ ""), "1\nwalk/S\n"));
+ Assertions.assertEquals("walk", plain.stem("walks").toString());
+
+ // \u00E9 is e with an acute accent, a single-character flag outside ASCII
+ final HunspellStemmer accented = new HunspellStemmer(load(String.join("\n",
+ "FLAG UTF-8",
+ "SFX \u00E9 Y 1",
+ "SFX \u00E9 0 s .",
+ ""), "1\nwalk/\u00E9\n"));
+ Assertions.assertEquals("walk", accented.stem("walks").toString());
+ }
+
+ /**
+ * Verifies that a strip-only rule, whose affix material is empty and which therefore
+ * only removes stem material, is undone: the suffix rule turns the entry
+ * {@code bake} into the surface form {@code bak}, and the prefix rule turns
+ * {@code apple} into {@code pple}.
+ *
+ * @throws IOException Thrown if a fixture fails to load.
+ */
+ @Test
+ void testStripOnlyAffixRulesAreUndone() throws IOException {
+ final HunspellStemmer suffixStripping = new HunspellStemmer(load(String.join("\n",
+ "SFX A Y 1",
+ "SFX A e 0 e",
+ ""), "1\nbake/A\n"));
+ Assertions.assertEquals("bake", suffixStripping.stem("bak").toString());
+
+ final HunspellStemmer prefixStripping = new HunspellStemmer(load(String.join("\n",
+ "PFX B Y 1",
+ "PFX B a 0 a",
+ ""), "1\napple/B\n"));
+ Assertions.assertEquals("apple", prefixStripping.stem("pple").toString());
+ }
+
+ /**
+ * Verifies that an entry written with an empty flag run loads and carries no flags in
+ * every flag mode, rather than failing the load in {@code FLAG num} mode alone.
+ *
+ * @throws IOException Thrown if a fixture fails to load.
+ */
+ @Test
+ void testEmptyFlagRunYieldsNoFlagsInEveryMode() throws IOException {
+ Assertions.assertEquals(0, load("", "1\nword/\n").lookup("word").get(0).length);
+ Assertions.assertEquals(0,
+ load("FLAG long\n", "1\nword/\n").lookup("word").get(0).length);
+ Assertions.assertEquals(0,
+ load("FLAG num\n", "1\nword/\n").lookup("word").get(0).length);
+ }
+
+ /** Verifies that a malformed affix file aborts the load instead of loading partially. */
+ @Test
+ void testMalformedInputFailsLoud() {
+ Assertions.assertThrows(IOException.class,
+ () -> load("SFX S Y 2\nSFX S 0 s .\n", "1\ncat/S\n"));
+ Assertions.assertThrows(IOException.class,
+ () -> load("SET NO-SUCH-ENCODING\n", "0\n"));
+ Assertions.assertThrows(IOException.class, () -> load("SFX S 0 s [a\n", "0\n"));
+ }
+
+ /**
+ * Verifies that every entry point rejects a {@code null} argument with the documented
+ * exception, and that the stream-based loader names the offending argument the way
+ * its file-based sibling does.
+ */
+ @Test
+ void testNullArgumentsAreRejected() {
+ final InputStream present = new ByteArrayInputStream(new byte[0]);
+ IllegalArgumentException e = Assertions.assertThrows(IllegalArgumentException.class,
+ () -> HunspellDictionary.load(null, present));
+ Assertions.assertEquals("affixStream must not be null", e.getMessage());
+
+ e = Assertions.assertThrows(IllegalArgumentException.class,
+ () -> HunspellDictionary.load(present, null));
+ Assertions.assertEquals("dictionaryStream must not be null", e.getMessage());
+
+ e = Assertions.assertThrows(IllegalArgumentException.class,
+ () -> new HunspellStemmer(null));
+ Assertions.assertEquals("dictionary must not be null", e.getMessage());
+
+ e = Assertions.assertThrows(IllegalArgumentException.class,
+ () -> new HunspellStemmerFactory(null));
+ Assertions.assertEquals("dictionary must not be null", e.getMessage());
+
+ e = Assertions.assertThrows(IllegalArgumentException.class,
+ () -> stemmer.stemAll(null));
+ Assertions.assertEquals("word must not be null", e.getMessage());
+ }
+
+ /**
+ * Verifies hunspell's tolerance for trailing text after a numeric or long flag run:
+ * the flag run ends at the first space, and whatever follows is a morphological
+ * field even without a two-letter tag, so such an entry loads instead of aborting
+ * the whole dictionary.
+ *
+ * @throws IOException Thrown if a fixture fails to load.
+ */
+ @Test
+ void testTrailingTextAfterNumericFlagRunIsMorphologyNotAnError() throws IOException {
+ final HunspellDictionary numbers = load("FLAG num\n",
+ "2\nwalk/39 blah\nrun/7,9 xyz abc\n");
+ Assertions.assertNotNull(numbers.lookup("walk"));
+ Assertions.assertTrue(HunspellDictionary.hasFlag(numbers.lookup("walk"), 39));
+ Assertions.assertTrue(HunspellDictionary.hasFlag(numbers.lookup("run"), 7));
+ Assertions.assertTrue(HunspellDictionary.hasFlag(numbers.lookup("run"), 9));
+
+ final HunspellDictionary longs = load("FLAG long\n", "1\nwalk/AB cd\n");
+ Assertions.assertTrue(HunspellDictionary.hasFlag(longs.lookup("walk"),
+ ('A' << 16) | 'B'));
+ }
+
+ /**
+ * Verifies that stemming the empty word answers the empty word: a zero-length
+ * surface has no morphology, and a strip-only rule must not restore its strip
+ * string onto nothing and answer a non-empty stem.
+ *
+ * @throws IOException Thrown if a fixture fails to load.
+ */
+ @Test
+ void testEmptyWordStemsToItself() throws IOException {
+ final HunspellStemmer stripOnly = new HunspellStemmer(load(
+ "PFX P Y 1\nPFX P xy 0 .\n",
+ "1\nxy/P\n"));
+ Assertions.assertEquals("", stripOnly.stem("").toString());
+ Assertions.assertEquals(List.of(""), stripOnly.stemAll(""));
+ }
+
+ /**
+ * Verifies the escaped-slash feature: {@code \/} belongs to the word, so an entry
+ * naming a slashed term keeps its slash while the first unescaped slash still
+ * separates the flag run.
+ *
+ * @throws IOException Thrown if a fixture fails to load.
+ */
+ @Test
+ void testEscapedSlashBelongsToTheWord() throws IOException {
+ final HunspellDictionary slashed = load("FLAG num\n",
+ "2\nTCP\\/IP/39\nAC\\/DC\n");
+ Assertions.assertNotNull(slashed.lookup("TCP/IP"));
+ Assertions.assertTrue(HunspellDictionary.hasFlag(slashed.lookup("TCP/IP"), 39));
+ Assertions.assertNotNull(slashed.lookup("AC/DC"));
+ Assertions.assertNull(slashed.lookup("TCP"));
+ }
+
+ /**
+ * Verifies the sharpest combination of the morphology cut: an entry that is both a
+ * multi-word term and carries trailing tag morphology keeps the whole multi-word
+ * surface and its flags, and the tags stay out of the word.
+ *
+ * @throws IOException Thrown if a fixture fails to load.
+ */
+ @Test
+ void testMultiWordEntryWithTrailingTagMorphology() throws IOException {
+ final HunspellDictionary phrases = load("FLAG num\n",
+ "1\nall right/39 po:phrase st:allright\n");
+ Assertions.assertNotNull(phrases.lookup("all right"));
+ Assertions.assertTrue(HunspellDictionary.hasFlag(phrases.lookup("all right"), 39));
+ Assertions.assertNull(phrases.lookup("all right po:phrase st:allright"));
+ }
+
+ /**
+ * Pins FLAG UTF-8 for a supplementary flag character: a flag is one code point, so
+ * a character above U+FFFF is one flag carrying its code point value, never two
+ * surrogate-unit flags. The Spanish dictionary of the LibreOffice collection names
+ * affix rules with such characters, so an affix keyed by a supplementary flag must
+ * connect to the entries that carry it.
+ *
+ * @throws IOException Thrown if a fixture fails to load.
+ */
+ @Test
+ void testSupplementaryFlagCharacterIsOneCodePointFlag() throws IOException {
+ // U+1F600 as a flag, written as its surrogate pair
+ final HunspellDictionary emoji = load("FLAG UTF-8\n",
+ "1\nwalk/\uD83D\uDE00\n");
+ Assertions.assertTrue(HunspellDictionary.hasFlag(emoji.lookup("walk"), 0x1F600));
+ Assertions.assertFalse(HunspellDictionary.hasFlag(emoji.lookup("walk"), 0xD83D));
+
+ final HunspellStemmer supplementaryFlags = new HunspellStemmer(load(
+ "FLAG UTF-8\nSFX \uD83D\uDE00 Y 1\nSFX \uD83D\uDE00 0 s .\n",
+ "1\nwalk/\uD83D\uDE00\n"));
+ Assertions.assertEquals("walk", supplementaryFlags.stem("walks").toString());
+ }
+
+ /**
+ * Pins the variation-selector rule the Spanish dictionary of the LibreOffice
+ * collection relies on: a variation selector after a flag character selects its
+ * presentation and is no flag of its own, so an affix rule named with the emoji
+ * form of a character connects to entries flagged with either spelling.
+ *
+ * @throws IOException Thrown if a fixture fails to load.
+ */
+ @Test
+ void testVariationSelectorIsDroppedFromFlagIdentity() throws IOException {
+ // U+260E BLACK TELEPHONE followed by U+FE0F VARIATION SELECTOR-16, the exact
+ // shape of a prefix flag in the published es_ES affix file
+ final HunspellStemmer stemmer = new HunspellStemmer(load(
+ "FLAG UTF-8\nPFX \u260E\uFE0F Y 1\nPFX \u260E\uFE0F 0 tele .\n",
+ "1\nfono/\u260E\n"));
+ Assertions.assertEquals("fono", stemmer.stem("telefono").toString());
+ }
+
+ /**
+ * Pins the documented rejection of rules that neither add nor remove material: a
+ * suffix rule with strip {@code 0} and affix {@code 0} loads without error and
+ * never fires, so stemming a flagged dictionary word answers that word exactly
+ * once.
+ *
+ * @throws IOException Thrown if a fixture fails to load.
+ */
+ @Test
+ void testRuleThatNeitherAddsNorRemovesLoadsAndNeverFires() throws IOException {
+ final HunspellStemmer identity = new HunspellStemmer(load(
+ "SFX X Y 1\nSFX X 0 0 .\n",
+ "1\nwalk/X\n"));
+ Assertions.assertEquals(List.of("walk"), identity.stemAll("walk"));
+ }
+
+ /**
+ * Verifies the AF flag alias table: the first AF line declares the count, every
+ * further AF line is one flag run, and a purely numeric flag field in the word
+ * list is a 1-based reference into that table, the layout the published Hungarian
+ * dictionary uses for all of its ninety-seven thousand entries. Alias lines may
+ * carry trailing comments, which the field split already discards.
+ *
+ * @throws IOException Thrown if a fixture fails to load.
+ */
+ @Test
+ void testNumericDictionaryFlagsResolveThroughTheAliasTable() throws IOException {
+ final HunspellStemmer stemmer = new HunspellStemmer(load(
+ String.join("\n",
+ "AF 2",
+ "AF S # 1",
+ "AF SP # 2",
+ "SFX S Y 1",
+ "SFX S 0 s .",
+ "PFX P Y 1",
+ "PFX P 0 re .",
+ ""),
+ "2\nwalk/1\nplay/2\n"));
+ Assertions.assertEquals("walk", stemmer.stem("walks").toString());
+ Assertions.assertEquals("play", stemmer.stem("plays").toString());
+ Assertions.assertEquals("play", stemmer.stem("replay").toString());
+ // walk carries alias 1, the suffix-only run, so the prefix must not apply
+ Assertions.assertEquals("rewalk", stemmer.stem("rewalk").toString());
+ }
+
+ /**
+ * Verifies that an alias reference outside the AF table fails loud with the line
+ * and the table size, instead of silently flagging the entry with nothing, and that
+ * a digit run too large for an alias number fails loud as well.
+ */
+ @Test
+ void testAliasReferenceOutsideTheTableFailsLoud() {
+ IOException e = Assertions.assertThrows(IOException.class, () -> load(
+ "AF 1\nAF S # 1\nSFX S Y 1\nSFX S 0 s .\n",
+ "1\nwalk/2\n"));
+ Assertions.assertEquals(
+ "flag alias 2 at line 2 is outside the AF table of 1 aliases",
+ e.getMessage());
+
+ e = Assertions.assertThrows(IOException.class, () -> load(
+ "AF 1\nAF S # 1\nSFX S Y 1\nSFX S 0 s .\n",
+ "1\nwalk/99999999999999999999\n"));
+ Assertions.assertEquals(
+ "malformed flag alias '99999999999999999999' at line 2",
+ e.getMessage());
+ }
+
+ /**
+ * Verifies that numeric flag fields stay ordinary flags when no AF table exists:
+ * under FLAG num a digit run is a flag value, not an alias reference.
+ *
+ * @throws IOException Thrown if a fixture fails to load.
+ */
+ @Test
+ void testNumericFlagsWithoutAliasTableStayFlags() throws IOException {
+ final HunspellDictionary numbers = load("FLAG num\n", "1\nwalk/39\n");
+ Assertions.assertTrue(HunspellDictionary.hasFlag(numbers.lookup("walk"), 39));
+ }
+
+ /**
+ * Verifies two-part compound decomposition under the general compounding flag: a
+ * word the affix analysis cannot explain splits into two listed parts that both
+ * carry the flag, reported left to right, while a part without the flag blocks the
+ * split and the word stays unanalyzed.
+ *
+ * @throws IOException Thrown if a fixture fails to load.
+ */
+ @Test
+ void testCompoundFlagDecomposesUnanalyzedWords() throws IOException {
+ final HunspellStemmer stemmer = new HunspellStemmer(load(
+ "COMPOUNDFLAG Z\nCOMPOUNDMIN 3\n",
+ "3\ndog/Z\nhouse/Z\ncat\n"));
+ Assertions.assertEquals(List.of("dog", "house"), stemmer.stemAll("doghouse"));
+ // cat is listed without the compounding flag, so no split may use it
+ Assertions.assertEquals(List.of("cathouse"), stemmer.stemAll("cathouse"));
+ // a listed word never decomposes; it is its own analysis
+ Assertions.assertEquals(List.of("dog"), stemmer.stemAll("dog"));
+ }
+
+ /**
+ * Verifies the positional compound flags: the begin flag only opens and the end
+ * flag only closes, so the parts compose in one order and refuse the other.
+ *
+ * @throws IOException Thrown if a fixture fails to load.
+ */
+ @Test
+ void testCompoundBeginAndEndFlagsArePositional() throws IOException {
+ final HunspellStemmer stemmer = new HunspellStemmer(load(
+ "COMPOUNDBEGIN B\nCOMPOUNDEND E\nCOMPOUNDMIN 3\n",
+ "2\ndog/B\nhouse/E\n"));
+ Assertions.assertEquals(List.of("dog", "house"), stemmer.stemAll("doghouse"));
+ Assertions.assertEquals(List.of("housedog"), stemmer.stemAll("housedog"));
+ }
+
+ /**
+ * Verifies the minimum part length: a split leaving a side shorter than
+ * COMPOUNDMIN is never taken, although both sides are listed and flagged.
+ *
+ * @throws IOException Thrown if a fixture fails to load.
+ */
+ @Test
+ void testCompoundMinBoundsThePartLength() throws IOException {
+ final HunspellStemmer stemmer = new HunspellStemmer(load(
+ "COMPOUNDFLAG Z\nCOMPOUNDMIN 4\n",
+ "2\ndog/Z\nhouse/Z\n"));
+ // the left side would be three characters, below the declared minimum of four
+ Assertions.assertEquals(List.of("doghouse"), stemmer.stemAll("doghouse"));
+ }
+
+ /**
+ * Verifies the NEEDAFFIX flag on entries: a virtual stem exists only to be affixed,
+ * the linking forms of the published German dictionary being the model, so its bare
+ * form is no analysis of itself while its affixed forms still reduce to it.
+ *
+ * @throws IOException Thrown if a fixture fails to load.
+ */
+ @Test
+ void testNeedAffixEntryIsNoStandaloneAnalysis() throws IOException {
+ final HunspellStemmer stemmer = new HunspellStemmer(load(
+ "NEEDAFFIX h\nSFX S Y 1\nSFX S 0 s .\nSFX K Y 1\nSFX K 0 k .\n",
+ "2\nlink/hS\nlin/K\n"));
+ // the virtual entry no longer explains the bare form; only the k analysis remains
+ Assertions.assertEquals(List.of("lin"), stemmer.stemAll("link"));
+ // affixed, the virtual stem is exactly what the s removal lands on
+ Assertions.assertEquals(List.of("link"), stemmer.stemAll("links"));
+ }
+
+ /**
+ * Verifies NEEDAFFIX against homonyms: the flag blocks one entry's flag set, not
+ * the word, so a second listing without the flag keeps the bare form valid.
+ *
+ * @throws IOException Thrown if a fixture fails to load.
+ */
+ @Test
+ void testNeedAffixHomonymKeepsTheBareWord() throws IOException {
+ final HunspellStemmer stemmer = new HunspellStemmer(load(
+ "NEEDAFFIX h\nSFX S Y 1\nSFX S 0 s .\n",
+ "2\nlink/hS\nlink\n"));
+ Assertions.assertEquals(List.of("link"), stemmer.stemAll("link"));
+ }
+
+ /**
+ * Verifies the historical PSEUDOROOT alias, the directive's name before hunspell
+ * renamed it to NEEDAFFIX; older dictionaries still declare it.
+ *
+ * @throws IOException Thrown if a fixture fails to load.
+ */
+ @Test
+ void testPseudoRootIsNeedAffixByItsOldName() throws IOException {
+ final HunspellStemmer stemmer = new HunspellStemmer(load(
+ "PSEUDOROOT h\nSFX S Y 1\nSFX S 0 s .\nSFX K Y 1\nSFX K 0 k .\n",
+ "2\nlink/hS\nlin/K\n"));
+ Assertions.assertEquals(List.of("lin"), stemmer.stemAll("link"));
+ }
+
+ /**
+ * Verifies the NEEDAFFIX flag on affix rules: a rule carrying the flag among its
+ * continuation classes makes a form that still needs another affix, so its
+ * single-removal analysis is suppressed while a twofold removal, whose inner affix
+ * is the further one required, still reports the stem.
+ *
+ * @throws IOException Thrown if a fixture fails to load.
+ */
+ @Test
+ void testNeedAffixOnAnAffixRequiresAnotherAffix() throws IOException {
+ final HunspellStemmer stemmer = new HunspellStemmer(load(
+ String.join("\n",
+ "NEEDAFFIX h",
+ "SFX A Y 1",
+ "SFX A 0 er/hB .",
+ "SFX B Y 1",
+ "SFX B 0 s .",
+ ""),
+ "1\nwork/A\n"));
+ // work + er alone is virtual, so worker has no analysis and passes through
+ Assertions.assertEquals(List.of("worker"), stemmer.stemAll("worker"));
+ // work + er + s is complete; the twofold removal reaches the listed stem
+ Assertions.assertEquals(List.of("work"), stemmer.stemAll("workers"));
+ }
+
+ /**
+ * Verifies that a cross-product analysis satisfies an affix's NEEDAFFIX marker:
+ * the prefix is the further affix the marked suffix requires, mirroring how
+ * hunspell accepts a prefix plus a needs-affix suffix together.
+ *
+ * @throws IOException Thrown if a fixture fails to load.
+ */
+ @Test
+ void testCrossProductSatisfiesNeedAffixOnTheSuffix() throws IOException {
+ final HunspellStemmer stemmer = new HunspellStemmer(load(
+ String.join("\n",
+ "NEEDAFFIX h",
+ "PFX P Y 1",
+ "PFX P 0 un .",
+ "SFX A Y 1",
+ "SFX A 0 er/h .",
+ ""),
+ "1\nwork/AP\n"));
+ Assertions.assertEquals(List.of("worker"), stemmer.stemAll("worker"));
+ Assertions.assertEquals(List.of("work"), stemmer.stemAll("unworker"));
+ }
+
+ /**
+ * Verifies the ONLYINCOMPOUND flag: an entry carrying it appears only inside
+ * compounds, the ordinal parts of the published US English dictionary being the
+ * model, so neither its bare form nor its affixed forms are standalone analyses,
+ * while compound decomposition may still use it.
+ *
+ * @throws IOException Thrown if a fixture fails to load.
+ */
+ @Test
+ void testOnlyInCompoundEntrySupportsNoStandaloneAnalyses() throws IOException {
+ final HunspellStemmer stemmer = new HunspellStemmer(load(
+ "ONLYINCOMPOUND c\nCOMPOUNDFLAG Z\nCOMPOUNDMIN 3\nSFX S Y 1\nSFX S 0 s .\n",
+ "3\npart/cSZ\nhouse/Z\nwalk/S\n"));
+ // the affix analysis is suppressed because part's only flag set is compound-only
+ Assertions.assertEquals(List.of("parts"), stemmer.stemAll("parts"));
+ Assertions.assertEquals(List.of("part"), stemmer.stemAll("part"));
+ // inside a compound the entry serves exactly its declared purpose
+ Assertions.assertEquals(List.of("part", "house"), stemmer.stemAll("parthouse"));
+ Assertions.assertEquals(List.of("walk"), stemmer.stemAll("walks"));
+ }
+
+ /**
+ * Verifies the FORBIDDENWORD flag: an entry carrying it is listed to be blocked,
+ * so it supports no analysis and no compound part.
+ *
+ * @throws IOException Thrown if a fixture fails to load.
+ */
+ @Test
+ void testForbiddenWordSupportsNothing() throws IOException {
+ final HunspellStemmer stemmer = new HunspellStemmer(load(
+ "FORBIDDENWORD w\nCOMPOUNDFLAG Z\nCOMPOUNDMIN 3\nSFX S Y 1\nSFX S 0 s .\n",
+ "3\nfoo/wSZ\nhouse/Z\nbar/S\n"));
+ Assertions.assertEquals(List.of("foo"), stemmer.stemAll("foo"));
+ Assertions.assertEquals(List.of("foos"), stemmer.stemAll("foos"));
+ Assertions.assertEquals(List.of("foohouse"), stemmer.stemAll("foohouse"));
+ Assertions.assertEquals(List.of("bar"), stemmer.stemAll("bars"));
+ }
+
+ /** The circumfix fixture: the German {@code ge...t} participle in miniature. */
+ private static final String CIRCUMFIX_AFFIX = String.join("\n",
+ "CIRCUMFIX f",
+ "PFX G Y 1",
+ "PFX G 0 ge/f .",
+ "SFX T Y 1",
+ "SFX T en et/f en",
+ "PFX U Y 1",
+ "PFX U 0 un .",
+ "SFX S Y 1",
+ "SFX S 0 s .",
+ "");
+
+ /**
+ * Verifies the CIRCUMFIX flag: two marked halves analyze together and neither
+ * analyzes alone, so the participle reduces to its verb while the half-applied
+ * forms stay unexplained.
+ *
+ * @throws IOException Thrown if a fixture fails to load.
+ */
+ @Test
+ void testCircumfixHalvesOnlyAnalyzeTogether() throws IOException {
+ final HunspellStemmer stemmer = new HunspellStemmer(load(
+ CIRCUMFIX_AFFIX, "1\narbeiten/GT\n"));
+ Assertions.assertEquals(List.of("arbeiten"), stemmer.stemAll("gearbeitet"));
+ // the suffix half alone is no word, although the stem carries its flag
+ Assertions.assertEquals(List.of("arbeitet"), stemmer.stemAll("arbeitet"));
+ // the prefix half alone is no word either
+ Assertions.assertEquals(List.of("gearbeiten"), stemmer.stemAll("gearbeiten"));
+ }
+
+ /**
+ * Verifies that circumfixing rejects mixed pairs: a marked half never combines
+ * with an unmarked affix of the other kind, in either direction, while a fully
+ * unmarked cross-product in the same dictionary still analyzes.
+ *
+ * @throws IOException Thrown if a fixture fails to load.
+ */
+ @Test
+ void testCircumfixRejectsMixedPairs() throws IOException {
+ final HunspellStemmer stemmer = new HunspellStemmer(load(
+ CIRCUMFIX_AFFIX, "2\narbeiten/GTUS\nlauf/US\n"));
+ // unmarked prefix with the marked suffix half
+ Assertions.assertEquals(List.of("unarbeitet"), stemmer.stemAll("unarbeitet"));
+ // the marked prefix half with an unmarked suffix
+ Assertions.assertEquals(List.of("gearbeitens"), stemmer.stemAll("gearbeitens"));
+ // both halves marked still analyze beside the rejected mixtures
+ Assertions.assertEquals(List.of("arbeiten"), stemmer.stemAll("gearbeitet"));
+ // a fully unmarked cross-product is untouched by the circumfix declaration
+ Assertions.assertEquals(List.of("lauf"), stemmer.stemAll("unlaufs"));
+ }
+
+ /**
+ * Verifies decomposition beyond two parts: the positional flags admit a begin, a
+ * middle, and an end part, a part fit only for the middle neither opens nor closes,
+ * and repeated middles fold into the reported set.
+ *
+ * @throws IOException Thrown if a fixture fails to load.
+ */
+ @Test
+ void testCompoundMiddleAdmitsInnerParts() throws IOException {
+ final HunspellStemmer stemmer = new HunspellStemmer(load(
+ "COMPOUNDBEGIN B\nCOMPOUNDMIDDLE M\nCOMPOUNDEND E\nCOMPOUNDMIN 3\n",
+ "3\ndog/B\ncat/M\nhouse/E\n"));
+ Assertions.assertEquals(List.of("dog", "cat", "house"),
+ stemmer.stemAll("dogcathouse"));
+ Assertions.assertEquals(List.of("dog", "cat", "house"),
+ stemmer.stemAll("dogcatcathouse"));
+ Assertions.assertEquals(List.of("dog", "house"), stemmer.stemAll("doghouse"));
+ // cat holds only the middle flag, so it neither opens nor closes
+ Assertions.assertEquals(List.of("cathouse"), stemmer.stemAll("cathouse"));
+ Assertions.assertEquals(List.of("dogcat"), stemmer.stemAll("dogcat"));
+ }
+
+ /**
+ * Verifies COMPOUNDWORDMAX: a decomposition needing more parts than declared is
+ * rejected while one within the bound still analyzes.
+ *
+ * @throws IOException Thrown if a fixture fails to load.
+ */
+ @Test
+ void testCompoundWordMaxBoundsThePartCount() throws IOException {
+ final HunspellStemmer stemmer = new HunspellStemmer(load(
+ "COMPOUNDBEGIN B\nCOMPOUNDMIDDLE M\nCOMPOUNDEND E\nCOMPOUNDMIN 3\n"
+ + "COMPOUNDWORDMAX 2\n",
+ "3\ndog/B\ncat/M\nhouse/E\n"));
+ Assertions.assertEquals(List.of("dog", "house"), stemmer.stemAll("doghouse"));
+ Assertions.assertEquals(List.of("dogcathouse"), stemmer.stemAll("dogcathouse"));
+ }
+
+ /**
+ * Verifies affixed compound parts, the German linking form being the model: a part
+ * is its entry plus one suffix whose continuation classes position the derived form
+ * and permit it at the internal boundary, and the reported analysis is the entry,
+ * not the linking form. The lowercase interior spelling of a capitalized entry is
+ * found through the part's uppercased retry.
+ *
+ * @throws IOException Thrown if a fixture fails to load.
+ */
+ @Test
+ void testLinkingSuffixJoinsCompoundParts() throws IOException {
+ final HunspellStemmer stemmer = new HunspellStemmer(load(
+ String.join("\n",
+ "COMPOUNDBEGIN x",
+ "COMPOUNDEND z",
+ "COMPOUNDPERMITFLAG c",
+ "COMPOUNDMIN 2",
+ "SFX j Y 1",
+ "SFX j 0 s/xc .",
+ ""),
+ "2\nAbbildung/j\nVerzeichnis/z\n"));
+ Assertions.assertEquals(List.of("Abbildung", "Verzeichnis"),
+ stemmer.stemAll("Abbildungsverzeichnis"));
+ // without the linking s the first part has no admitting reading
+ Assertions.assertEquals(List.of("Abbildungverzeichnis"),
+ stemmer.stemAll("Abbildungverzeichnis"));
+ }
+
+ /**
+ * Verifies zero-suffix part positioning, the pattern the published German
+ * dictionary uses: a virtual stem enters compounds through a rule that adds no
+ * material but whose continuation classes carry the positional and permit flags.
+ *
+ * @throws IOException Thrown if a fixture fails to load.
+ */
+ @Test
+ void testZeroSuffixPositionsAVirtualStemInCompounds() throws IOException {
+ final HunspellStemmer stemmer = new HunspellStemmer(load(
+ String.join("\n",
+ "NEEDAFFIX h",
+ "COMPOUNDBEGIN x",
+ "COMPOUNDEND z",
+ "COMPOUNDPERMITFLAG c",
+ "COMPOUNDMIN 3",
+ "SFX j Y 1",
+ "SFX j 0 0/xc .",
+ ""),
+ "2\nfugen/hj\nwerk/z\n"));
+ Assertions.assertEquals(List.of("fugen", "werk"), stemmer.stemAll("fugenwerk"));
+ // the virtual stem alone is still no word
+ Assertions.assertEquals(List.of("fugen"), stemmer.stemAll("fugen"));
+ }
+
+ /**
+ * Verifies COMPOUNDFORBIDFLAG: an affixed form whose rule carries the flag stays
+ * out of compounds although its positioning otherwise admits it.
+ *
+ * @throws IOException Thrown if a fixture fails to load.
+ */
+ @Test
+ void testCompoundForbidFlagBarsAnAffixedPart() throws IOException {
+ final String words = "2\ndog/ZS\nhouse/Z\n";
+ final HunspellStemmer barred = new HunspellStemmer(load(
+ "COMPOUNDFLAG Z\nCOMPOUNDPERMITFLAG c\nCOMPOUNDFORBIDFLAG F\nCOMPOUNDMIN 3\n"
+ + "SFX S Y 1\nSFX S 0 s/cF .\n",
+ words));
+ Assertions.assertEquals(List.of("dogshouse"), barred.stemAll("dogshouse"));
+ final HunspellStemmer allowed = new HunspellStemmer(load(
+ "COMPOUNDFLAG Z\nCOMPOUNDPERMITFLAG c\nCOMPOUNDMIN 3\n"
+ + "SFX S Y 1\nSFX S 0 s/c .\n",
+ words));
+ Assertions.assertEquals(List.of("dog", "house"), allowed.stemAll("dogshouse"));
+ }
+
+ /**
+ * Verifies that an affix without the permit flag keeps off internal boundaries: a
+ * suffixed reading fits the last part but not an earlier one.
+ *
+ * @throws IOException Thrown if a fixture fails to load.
+ */
+ @Test
+ void testAffixWithoutPermitFlagStaysAtTheEdge() throws IOException {
+ final HunspellStemmer stemmer = new HunspellStemmer(load(
+ "COMPOUNDFLAG Z\nCOMPOUNDMIN 3\nSFX S Y 1\nSFX S 0 s .\n",
+ "2\ndog/ZS\nhouse/ZS\n"));
+ // the suffix closes the word, so the last part may carry it
+ Assertions.assertEquals(List.of("dog", "house"), stemmer.stemAll("doghouses"));
+ // an internal suffix without the permit flag blocks the split
+ Assertions.assertEquals(List.of("dogshouse"), stemmer.stemAll("dogshouse"));
+ }
+
+ /**
+ * Verifies CHECKCOMPOUNDDUP: a part must not repeat its left neighbor, while the
+ * same dictionary without the declaration accepts the repetition.
+ *
+ * @throws IOException Thrown if a fixture fails to load.
+ */
+ @Test
+ void testCheckCompoundDupForbidsRepeatedParts() throws IOException {
+ final String words = "2\ndog/Z\nhouse/Z\n";
+ final HunspellStemmer checked = new HunspellStemmer(load(
+ "COMPOUNDFLAG Z\nCOMPOUNDMIN 3\nCHECKCOMPOUNDDUP\n", words));
+ Assertions.assertEquals(List.of("dogdoghouse"), checked.stemAll("dogdoghouse"));
+ final HunspellStemmer unchecked = new HunspellStemmer(load(
+ "COMPOUNDFLAG Z\nCOMPOUNDMIN 3\n", words));
+ Assertions.assertEquals(List.of("dog", "house"), unchecked.stemAll("dogdoghouse"));
+ }
+
+ /**
+ * Verifies CHECKCOMPOUNDCASE: an uppercase character on either side of a junction
+ * forbids the split, while the same dictionary without the declaration accepts it.
+ *
+ * @throws IOException Thrown if a fixture fails to load.
+ */
+ @Test
+ void testCheckCompoundCaseForbidsUppercaseJunctions() throws IOException {
+ final String words = "2\ndog/Z\nHouse/Z\n";
+ final HunspellStemmer checked = new HunspellStemmer(load(
+ "COMPOUNDFLAG Z\nCOMPOUNDMIN 3\nCHECKCOMPOUNDCASE\n", words));
+ Assertions.assertEquals(List.of("dogHouse"), checked.stemAll("dogHouse"));
+ final HunspellStemmer unchecked = new HunspellStemmer(load(
+ "COMPOUNDFLAG Z\nCOMPOUNDMIN 3\n", words));
+ Assertions.assertEquals(List.of("dog", "House"), unchecked.stemAll("dogHouse"));
+ }
+
+ /**
+ * Verifies CHECKCOMPOUNDTRIPLE: the same character three times in a row across a
+ * junction forbids the split, while the same dictionary without the declaration
+ * accepts it.
+ *
+ * @throws IOException Thrown if a fixture fails to load.
+ */
+ @Test
+ void testCheckCompoundTripleForbidsTripleLetters() throws IOException {
+ final String words = "2\nbell/Z\nlow/Z\n";
+ final HunspellStemmer checked = new HunspellStemmer(load(
+ "COMPOUNDFLAG Z\nCOMPOUNDMIN 3\nCHECKCOMPOUNDTRIPLE\n", words));
+ Assertions.assertEquals(List.of("belllow"), checked.stemAll("belllow"));
+ final HunspellStemmer unchecked = new HunspellStemmer(load(
+ "COMPOUNDFLAG Z\nCOMPOUNDMIN 3\n", words));
+ Assertions.assertEquals(List.of("bell", "low"), unchecked.stemAll("belllow"));
+ }
+
+ /**
+ * Verifies that a listed forbidden word never decomposes: the dictionary blocks
+ * one specific ill-formed compound while its parts stay productive elsewhere.
+ *
+ * @throws IOException Thrown if a fixture fails to load.
+ */
+ @Test
+ void testForbiddenEntryBlocksItsDecomposition() throws IOException {
+ final HunspellStemmer stemmer = new HunspellStemmer(load(
+ "FORBIDDENWORD w\nCOMPOUNDFLAG Z\nCOMPOUNDMIN 3\n",
+ "4\ndog/Z\nhouse/Z\ncat/Z\ndoghouse/w\n"));
+ Assertions.assertEquals(List.of("doghouse"), stemmer.stemAll("doghouse"));
+ Assertions.assertEquals(List.of("cat", "house"), stemmer.stemAll("cathouse"));
+ }
+
+ /**
+ * Verifies that result-altering unsupported affix directives fail at load time.
+ * Ignoring {@code ICONV}, {@code OCONV}, {@code COMPLEXPREFIXES},
+ * {@code COMPOUNDRULE}, {@code IGNORE}, or {@code KEEPCASE} would change stems
+ * with no signal.
+ */
+ @ParameterizedTest
+ @CsvSource({
+ "ICONV, ICONV 1",
+ "OCONV, OCONV 1",
+ "COMPLEXPREFIXES, COMPLEXPREFIXES",
+ "COMPOUNDRULE, COMPOUNDRULE 1",
+ "IGNORE, IGNORE x",
+ "KEEPCASE, KEEPCASE k"
+ })
+ void testResultAlteringUnsupportedDirectiveFailsLoud(String name, String line) {
+ final IOException e = Assertions.assertThrows(IOException.class,
+ () -> load(line + "\n", "0\n"));
+ Assertions.assertEquals("unsupported affix directive '" + name + "' at line 1",
+ e.getMessage());
+ }
+
+ /**
+ * Verifies that a full-strip suffix rule is not applied unless the affix file
+ * declares {@code FULLSTRIP}. Hunspell applies a rule whose strip string consumes
+ * the whole stem only under that declaration; without it, inventing a stem from
+ * such a rule contradicts the fail-closed loader policy.
+ *
+ * @throws IOException Thrown if a fixture fails to load.
+ */
+ @Test
+ void testFullStripRuleRequiresFullStripDirective() throws IOException {
+ final String rule = "SFX A Y 1\nSFX A work ed .\n";
+ final String words = "1\nwork/A\n";
+ Assertions.assertEquals(List.of("work"),
+ new HunspellStemmer(load("FULLSTRIP\n" + rule, words)).stemAll("ed"));
+ Assertions.assertEquals(List.of("ed"),
+ new HunspellStemmer(load(rule, words)).stemAll("ed"));
+ }
+
+ /**
+ * Verifies that {@link HunspellDictionary#load(InputStream, InputStream)} rejects an
+ * affix stream larger than {@link HunspellDictionary#MAX_STREAM_BYTES}.
+ */
+ @Test
+ void testLoadRejectsOversizedAffixStream() {
+ final IOException e = Assertions.assertThrows(IOException.class,
+ () -> HunspellDictionary.load(
+ filledStream(HunspellDictionary.MAX_STREAM_BYTES + 1),
+ new ByteArrayInputStream("1\nlock\n".getBytes(StandardCharsets.UTF_8))));
+ Assertions.assertEquals(
+ "affix stream size exceeds safe limit of " + HunspellDictionary.MAX_STREAM_BYTES,
+ e.getMessage());
+ }
+
+ /**
+ * Verifies that {@link HunspellDictionary#load(InputStream, InputStream)} rejects a
+ * dictionary stream larger than {@link HunspellDictionary#MAX_STREAM_BYTES}.
+ */
+ @Test
+ void testLoadRejectsOversizedDictionaryStream() {
+ final byte[] affix = "SET UTF-8\n".getBytes(StandardCharsets.UTF_8);
+ final IOException e = Assertions.assertThrows(IOException.class,
+ () -> HunspellDictionary.load(
+ new ByteArrayInputStream(affix),
+ filledStream(HunspellDictionary.MAX_STREAM_BYTES + 1)));
+ Assertions.assertEquals(
+ "dictionary stream size exceeds safe limit of "
+ + HunspellDictionary.MAX_STREAM_BYTES,
+ e.getMessage());
+ }
+
+ /**
+ * Pins the inclusive stream-byte ceiling: a stream of exactly {@code limit} bytes
+ * succeeds, and {@code limit + 1} fails. Uses a small limit so the test does not
+ * allocate the production ceiling.
+ *
+ * @throws IOException Thrown if reading the in-bound stream fails.
+ */
+ @Test
+ void testBoundedReadCeilingIsInclusive() throws IOException {
+ final int limit = 64;
+ final byte[] bytes = HunspellDictionary.readBounded(filledStream(limit), limit,
+ "affix stream");
+ Assertions.assertEquals(limit, bytes.length);
+ final IOException e = Assertions.assertThrows(IOException.class,
+ () -> HunspellDictionary.readBounded(filledStream(limit + 1), limit,
+ "affix stream"));
+ Assertions.assertEquals(
+ "affix stream size exceeds safe limit of " + limit, e.getMessage());
+ }
+
+ /**
+ * Pins affix conditions and boundary bucketing to code points, matching FLAG UTF-8:
+ * a condition of two dots needs two code points, so a stem that is one supplementary
+ * character must not match, while a one-dot condition and a supplementary affix
+ * character still analyze.
+ *
+ * @throws IOException Thrown if a fixture fails to load.
+ */
+ @Test
+ void testAffixConditionAndBoundaryUseCodePoints() throws IOException {
+ final HunspellStemmer twoDots = new HunspellStemmer(load(
+ "SFX X Y 1\nSFX X 0 s ..\n",
+ "1\n\uD83D\uDE00/X\n"));
+ Assertions.assertEquals("\uD83D\uDE00s", twoDots.stem("\uD83D\uDE00s").toString());
+
+ final HunspellStemmer oneDot = new HunspellStemmer(load(
+ "SFX X Y 1\nSFX X 0 s .\n",
+ "1\n\uD83D\uDE00/X\n"));
+ Assertions.assertEquals("\uD83D\uDE00", oneDot.stem("\uD83D\uDE00s").toString());
+
+ final HunspellStemmer emojiSuffix = new HunspellStemmer(load(
+ "SFX X Y 1\nSFX X 0 \uD83D\uDE00 .\n",
+ "1\nwalk/X\n"));
+ Assertions.assertEquals("walk", emojiSuffix.stem("walk\uD83D\uDE00").toString());
+
+ final HunspellStemmer classCondition = new HunspellStemmer(load(
+ "SFX X Y 1\nSFX X 0 s [\uD83D\uDE00]\n",
+ "1\nwalk\uD83D\uDE00/X\n"));
+ Assertions.assertEquals("walk\uD83D\uDE00",
+ classCondition.stem("walk\uD83D\uDE00s").toString());
+ }
+
+ /**
+ * Returns a stream of {@code size} zero bytes.
+ *
+ * @param size The number of bytes the stream yields.
+ * @return The stream. Never {@code null}.
+ */
+ private static InputStream filledStream(int size) {
+ return new InputStream() {
+ private int remaining = size;
+
+ @Override
+ public int read() {
+ if (remaining <= 0) {
+ return -1;
+ }
+ remaining--;
+ return 0;
+ }
+
+ @Override
+ public int read(byte[] buffer, int offset, int length) {
+ if (remaining <= 0) {
+ return -1;
+ }
+ final int n = Math.min(length, remaining);
+ Arrays.fill(buffer, offset, offset + n, (byte) 0);
+ remaining -= n;
+ return n;
+ }
+ };
+ }
+
+ /**
+ * Verifies that a cosmetic unsupported directive such as {@code REP} is skipped so
+ * the dictionary still loads.
+ *
+ * @throws IOException Thrown if the fixture fails to load.
+ */
+ @Test
+ void testCosmeticUnsupportedDirectiveIsSkipped() throws IOException {
+ final HunspellDictionary dictionary = load("REP 1\nREP alot a lot\n", "1\nlock\n");
+ Assertions.assertNotNull(dictionary.lookup("lock"));
+ }
+}
diff --git a/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/util/DictionaryCatalogTest.java b/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/util/DictionaryCatalogTest.java
new file mode 100644
index 000000000..9233b297a
--- /dev/null
+++ b/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/util/DictionaryCatalogTest.java
@@ -0,0 +1,154 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package opennlp.tools.util;
+
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.Path;
+
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+
+import opennlp.tools.stemmer.hunspell.HunspellDictionary;
+
+/**
+ * Tests the opt-in dictionary catalog against an in-memory properties file and a
+ * local file URI so no network access is required.
+ */
+public class DictionaryCatalogTest {
+
+ /**
+ * Verifies that a catalog download without the remote-download property fails with
+ * the property name in the message.
+ *
+ * @param dir A scratch directory managed by the test framework.
+ * @throws Exception Thrown if the fixture catalog cannot be prepared.
+ */
+ @Test
+ void testDownloadRequiresRemoteProperty(@TempDir Path dir) throws Exception {
+ final byte[] payload = "payload".getBytes(StandardCharsets.UTF_8);
+ final DictionaryCatalog loaded = demoCatalog(dir, payload);
+
+ final String previous = System.getProperty(DownloadUtil.REMOTE_DOWNLOAD_PROPERTY);
+ System.clearProperty(DownloadUtil.REMOTE_DOWNLOAD_PROPERTY);
+ try {
+ final IOException e = Assertions.assertThrows(IOException.class,
+ () -> loaded.download("demo", dir.resolve("out.bin")));
+ Assertions.assertTrue(e.getMessage().contains(DownloadUtil.REMOTE_DOWNLOAD_PROPERTY));
+ } finally {
+ restore(previous);
+ }
+ }
+
+ /**
+ * Verifies that an enabled catalog download fetches the entry and writes the
+ * digest-verified bytes to the target.
+ *
+ * @param dir A scratch directory managed by the test framework.
+ * @throws Exception Thrown if the fixture catalog cannot be prepared or fetched.
+ */
+ @Test
+ void testDownloadWithRemotePropertyEnabled(@TempDir Path dir) throws Exception {
+ final byte[] payload = "payload".getBytes(StandardCharsets.UTF_8);
+ final DictionaryCatalog loaded = demoCatalog(dir, payload);
+ final Path target = dir.resolve("out.bin");
+
+ final String previous = System.getProperty(DownloadUtil.REMOTE_DOWNLOAD_PROPERTY);
+ System.setProperty(DownloadUtil.REMOTE_DOWNLOAD_PROPERTY, "true");
+ try {
+ loaded.download("demo", target);
+ Assertions.assertArrayEquals(payload, Files.readAllBytes(target));
+ } finally {
+ restore(previous);
+ }
+ }
+
+ /**
+ * Verifies that the example catalog holds the MeCab and Hunspell entries, each with
+ * a full-length SHA-512 digest. Applications load their own catalog through the same
+ * {@link DictionaryCatalog#load(InputStream)} entry point.
+ *
+ * @throws IOException Thrown if the example catalog fails to load.
+ */
+ @Test
+ void testExampleCatalogContainsMecabAndHunspellEntries() throws IOException {
+ final DictionaryCatalog catalog = loadExampleCatalog();
+ Assertions.assertTrue(catalog.ids().contains("mecab.ipadic"));
+ Assertions.assertTrue(catalog.ids().contains("mecab.ko-dic"));
+ Assertions.assertTrue(catalog.ids().contains(
+ "hunspell.en_US" + HunspellDictionary.AFFIX_FILE_SUFFIX));
+ Assertions.assertEquals(128, catalog.get("mecab.ipadic").sha512().length());
+ Assertions.assertEquals(128, catalog.get(
+ "hunspell.en_US"
+ + HunspellDictionary.DICTIONARY_FILE_SUFFIX)
+ .sha512().length());
+ }
+
+ /**
+ * Loads the example catalog from test resources rather than a production classpath
+ * default.
+ *
+ * @return The example catalog. Never {@code null}.
+ * @throws IOException Thrown if the resource is absent or cannot be read.
+ */
+ private static DictionaryCatalog loadExampleCatalog() throws IOException {
+ try (InputStream in = DictionaryCatalogTest.class.getResourceAsStream(
+ "/opennlp/tools/util/dictionary-catalog.properties")) {
+ if (in == null) {
+ throw new IOException("missing example dictionary catalog");
+ }
+ return DictionaryCatalog.load(in);
+ }
+ }
+
+ /**
+ * Builds a one-entry catalog whose URL is a local file holding {@code payload}, so
+ * downloads need no network.
+ *
+ * @param dir The directory to write the payload file into.
+ * @param payload The bytes the catalog entry points at.
+ * @return The loaded catalog. Never {@code null}.
+ * @throws IOException Thrown if the payload file cannot be written.
+ */
+ private static DictionaryCatalog demoCatalog(Path dir, byte[] payload)
+ throws IOException {
+ final Path source = dir.resolve("dict.bin");
+ Files.write(source, payload);
+ final String catalog = "demo.url=" + source.toUri() + "\n"
+ + "demo.sha512=" + DigestTestUtil.sha512(payload) + "\n";
+ return DictionaryCatalog.load(
+ new ByteArrayInputStream(catalog.getBytes(StandardCharsets.UTF_8)));
+ }
+
+ /**
+ * Restores the remote-download property to its value before the test.
+ *
+ * @param previous The saved value, or {@code null} when the property was unset.
+ */
+ private static void restore(String previous) {
+ if (previous == null) {
+ System.clearProperty(DownloadUtil.REMOTE_DOWNLOAD_PROPERTY);
+ } else {
+ System.setProperty(DownloadUtil.REMOTE_DOWNLOAD_PROPERTY, previous);
+ }
+ }
+}
diff --git a/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/util/DigestTestUtil.java b/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/util/DigestTestUtil.java
new file mode 100644
index 000000000..af85b09a8
--- /dev/null
+++ b/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/util/DigestTestUtil.java
@@ -0,0 +1,45 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package opennlp.tools.util;
+
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.util.HexFormat;
+
+/**
+ * Computes SHA-512 digests for test fixtures.
+ */
+public final class DigestTestUtil {
+
+ private DigestTestUtil() {
+ }
+
+ /**
+ * {@return the SHA-512 digest of {@code bytes} as 128 lowercase hex digits}
+ *
+ * @param bytes The content to digest. Must not be {@code null}.
+ */
+ public static String sha512(byte[] bytes) {
+ try {
+ return HexFormat.of().formatHex(
+ MessageDigest.getInstance("SHA-512").digest(bytes));
+ } catch (NoSuchAlgorithmException e) {
+ throw new IllegalStateException(e);
+ }
+ }
+}
diff --git a/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/util/DownloadUtilFileTest.java b/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/util/DownloadUtilFileTest.java
new file mode 100644
index 000000000..493bb2050
--- /dev/null
+++ b/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/util/DownloadUtilFileTest.java
@@ -0,0 +1,177 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package opennlp.tools.util;
+
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.Path;
+
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.ValueSource;
+
+/**
+ * Pins {@link DownloadUtil#download(java.net.URI, Path, String)} against local file URIs
+ * so digest verification and the size ceiling are covered without a network.
+ */
+public class DownloadUtilFileTest {
+
+ /** The fixture bytes the download tests serve and digest. */
+ private static final byte[] PAYLOAD = "dictionary-bytes".getBytes(StandardCharsets.UTF_8);
+
+ /**
+ * Verifies that a download whose bytes match the expected digest lands in the
+ * target file.
+ *
+ * @param dir A scratch directory managed by the test framework.
+ * @throws IOException Thrown if the fixture cannot be written or fetched.
+ */
+ @Test
+ void testDownloadAcceptsMatchingDigest(@TempDir Path dir) throws IOException {
+ final Path source = dir.resolve("source.bin");
+ Files.write(source, PAYLOAD);
+ final Path target = dir.resolve("target.bin");
+
+ DownloadUtil.download(source.toUri(), target, DigestTestUtil.sha512(PAYLOAD));
+
+ Assertions.assertArrayEquals(PAYLOAD, Files.readAllBytes(target));
+ }
+
+ /**
+ * Verifies that a digest mismatch fails the download and leaves no target file
+ * behind.
+ *
+ * @param dir A scratch directory managed by the test framework.
+ * @throws IOException Thrown if the fixture cannot be written.
+ */
+ @Test
+ void testDownloadRejectsMismatchedDigest(@TempDir Path dir) throws IOException {
+ final Path source = dir.resolve("source.bin");
+ Files.write(source, PAYLOAD);
+ final Path target = dir.resolve("target.bin");
+ final String wrong = DigestTestUtil.sha512("other".getBytes(StandardCharsets.UTF_8));
+
+ final IOException e = Assertions.assertThrows(IOException.class,
+ () -> DownloadUtil.download(source.toUri(), target, wrong));
+ Assertions.assertTrue(e.getMessage().contains("SHA512 checksum validation failed"));
+ Assertions.assertTrue(Files.notExists(target));
+ }
+
+ /** Verifies that a {@code null} digest is rejected with the documented exception. */
+ @Test
+ void testDownloadRequiresSha512() {
+ Assertions.assertThrows(IllegalArgumentException.class,
+ () -> DownloadUtil.download(Path.of("x").toUri(), Path.of("y"), null));
+ }
+
+ /**
+ * Verifies that a digest shorter than 128 hex digits is rejected before anything is
+ * fetched.
+ *
+ * @param dir A scratch directory managed by the test framework.
+ * @throws IOException Thrown if the fixture cannot be written.
+ */
+ @Test
+ void testDownloadRejectsMalformedSha512(@TempDir Path dir) throws IOException {
+ final Path source = dir.resolve("source.bin");
+ Files.write(source, PAYLOAD);
+
+ Assertions.assertThrows(IllegalArgumentException.class,
+ () -> DownloadUtil.download(source.toUri(), dir.resolve("target.bin"), "abc123"));
+ }
+
+ /**
+ * Verifies that a source larger than the byte ceiling fails the download and leaves
+ * no target file behind.
+ *
+ * @param dir A scratch directory managed by the test framework.
+ * @throws IOException Thrown if the fixture cannot be written.
+ */
+ @Test
+ void testDownloadRejectsOversizedSource(@TempDir Path dir) throws IOException {
+ final Path source = dir.resolve("source.bin");
+ Files.write(source, PAYLOAD);
+ final Path target = dir.resolve("target.bin");
+
+ final IOException e = Assertions.assertThrows(IOException.class,
+ () -> DownloadUtil.download(source.toUri(), target,
+ DigestTestUtil.sha512(PAYLOAD), PAYLOAD.length - 1));
+ Assertions.assertTrue(e.getMessage().contains("exceeds safe limit"));
+ Assertions.assertTrue(Files.notExists(target));
+ }
+
+ /**
+ * Pins the inclusive byte ceiling: a source of exactly the ceiling's size still
+ * downloads.
+ *
+ * @param dir A scratch directory managed by the test framework.
+ * @throws IOException Thrown if the fixture cannot be written or fetched.
+ */
+ @Test
+ void testDownloadCeilingIsInclusive(@TempDir Path dir) throws IOException {
+ final Path source = dir.resolve("source.bin");
+ Files.write(source, PAYLOAD);
+ final Path target = dir.resolve("target.bin");
+
+ DownloadUtil.download(source.toUri(), target,
+ DigestTestUtil.sha512(PAYLOAD), PAYLOAD.length);
+
+ Assertions.assertArrayEquals(PAYLOAD, Files.readAllBytes(target));
+ }
+
+ /** Verifies that a positive property value overrides the fallback limit. */
+ @Test
+ void testConfiguredLimitOverridesFromProperty() {
+ final String property = "opennlp.test.limit.override";
+ System.setProperty(property, "1024");
+ try {
+ Assertions.assertEquals(1024L, DownloadUtil.configuredLimit(property, 7L));
+ } finally {
+ System.clearProperty(property);
+ }
+ }
+
+ /** Verifies that an unset property falls back to the given default. */
+ @Test
+ void testConfiguredLimitFallsBackWhenAbsent() {
+ Assertions.assertEquals(7L,
+ DownloadUtil.configuredLimit("opennlp.test.limit.absent", 7L));
+ }
+
+ /** Verifies that blank, non-numeric, and non-positive values fall back. */
+ @ParameterizedTest(name = "value \"{0}\" falls back")
+ @ValueSource(strings = {"", " ", "abc", "-1", "0"})
+ void testConfiguredLimitRejectsInvalidValues(String invalid) {
+ final String property = "opennlp.test.limit.invalid";
+ System.setProperty(property, invalid);
+ try {
+ Assertions.assertEquals(7L, DownloadUtil.configuredLimit(property, 7L));
+ } finally {
+ System.clearProperty(property);
+ }
+ }
+
+ /** Pins the default download ceiling of 64 MiB when no override property is set. */
+ @Test
+ void testDefaultBudgetsWithoutOverrides() {
+ Assertions.assertEquals(64L * 1024 * 1024, DownloadUtil.MAX_DOWNLOAD_BYTES);
+ }
+}
diff --git a/opennlp-core/opennlp-runtime/src/test/resources/opennlp/tools/util/dictionary-catalog.properties b/opennlp-core/opennlp-runtime/src/test/resources/opennlp/tools/util/dictionary-catalog.properties
new file mode 100644
index 000000000..748258fe6
--- /dev/null
+++ b/opennlp-core/opennlp-runtime/src/test/resources/opennlp/tools/util/dictionary-catalog.properties
@@ -0,0 +1,57 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# Pinned remote dictionary files used as catalog parsing fixtures. Tests never fetch
+# these URLs. Applications provide and maintain their own catalogs.
+
+# MeCab IPADIC 2.7.0 (EUC-JP). Upstream: MeCab project on SourceForge.
+mecab.ipadic.url=https://downloads.sourceforge.net/project/mecab/\
+mecab-ipadic/2.7.0-20070801/mecab-ipadic-2.7.0-20070801.tar.gz
+mecab.ipadic.sha512=35ea662cb62f1967849f7ed5781bd6dafef0fe20d63e88d9\
+a0057666e57ed23d5a0e6fb8d0701a0cc4da43a1050c1b0246\
+3bb862decc71c36b7fc2acdc158d86
+mecab.ipadic.filename=mecab-ipadic-2.7.0-20070801.tar.gz
+
+# mecab-ko-dic 2.1.1 (UTF-8). Upstream: eunjeon/mecab-ko-dic on Bitbucket.
+mecab.ko-dic.url=https://bitbucket.org/eunjeon/mecab-ko-dic/downloads/\
+mecab-ko-dic-2.1.1-20180720.tar.gz
+mecab.ko-dic.sha512=986f8f9c66c53accd296756bf632c979d2d44b695ada33f3\
+6c662f210dba34cd95d67b61dd8c84a1f7d59f80ee6bc22eb1\
+e9afb5dc6a7f9b6b75b4fbf2f8164f
+mecab.ko-dic.filename=mecab-ko-dic-2.1.1-20180720.tar.gz
+
+# LibreOffice en_US Hunspell pair, pinned to dictionaries commit 208a9fd8.
+hunspell.en_US.aff.url=https://raw.githubusercontent.com/LibreOffice/\
+dictionaries/208a9fd80b2a182fe20f224cd615119c6323ae2e/en/en_US.aff
+hunspell.en_US.aff.sha512=2b4448dfdff03caf300914415f4642f8d2ba5b650c5f024a\
+12355b420a279ffc12146649fce092ba591504476634a3d6\
+fd4c079335a27085b396fa76bfd28b74
+hunspell.en_US.aff.filename=en_US.aff
+
+hunspell.en_US.dic.url=https://raw.githubusercontent.com/LibreOffice/\
+dictionaries/208a9fd80b2a182fe20f224cd615119c6323ae2e/en/en_US.dic
+hunspell.en_US.dic.sha512=4be737249a8a436d20a02be575dcf6cf2f06f5f2abb840ea\
+5ec0ef0ac73a71fa0e4669e527c703d5c6b50ef61713a674\
+1b55bc136d559a54bcdeebcd62027988
+hunspell.en_US.dic.filename=en_US.dic
+
+hunspell.en_US.readme.url=https://raw.githubusercontent.com/LibreOffice/\
+dictionaries/208a9fd80b2a182fe20f224cd615119c6323ae2e/en/README_en_US.txt
+hunspell.en_US.readme.sha512=aa23ebc8adc0649b540264c7bf98cef5b6e383fec0e4a1a7\
+dd49d1c887cfeefd8edf6a568afc8a651521a3864c5b1ab5\
+0748ad16230d386809080b5b09135082
+hunspell.en_US.readme.filename=README_en_US.txt
diff --git a/opennlp-docs/src/docbkx/stemmer.xml b/opennlp-docs/src/docbkx/stemmer.xml
index 248b310c1..d777842fd 100644
--- a/opennlp-docs/src/docbkx/stemmer.xml
+++ b/opennlp-docs/src/docbkx/stemmer.xml
@@ -69,4 +69,42 @@ new CachingStemmer(factory).stem("running"); // "run"]]>
longer uses a sharing or caching stemmer.
</para>
</section>
+
+ <section xml:id="tools.stemmer.hunspell">
+ <title>Hunspell dictionaries</title>
+ <para>
+ <code>opennlp.tools.stemmer.hunspell</code> implements the documented
+ Hunspell dictionary format: a user-supplied
+ <code>.aff</code> affix file and its <code>.dic</code> word list. OpenNLP
+ bundles no dictionary data; dictionaries are downloaded separately, and
+ each states its own license. The dictionary is immutable and safe to share;
+ <code>HunspellStemmerFactory</code> hands out a fresh stemmer per call.
+ <code>HunspellManualExampleTest</code> asserts the behavior shown here.
+ <programlisting language="java"><![CDATA[
+HunspellDictionary dictionary = HunspellDictionary.load(
+ Path.of("dictionary.aff"), Path.of("dictionary.dic"));
+Stemmer stemmer = new HunspellStemmerFactory(dictionary).newStemmer();
+
+stemmer.stem("workers"); // "work"
+stemmer.stem("table"); // "table" (unknown vocabulary is unchanged)]]>
+ </programlisting>
+ The stems above are those of the project-authored miniature dictionary the
+ test loads, which lists <code>work</code> with an agentive and a plural
+ suffix; the test asserts the same stem for <code>worker</code>. Which stem a
+ published dictionary yields for a given form is decided by that dictionary.
+ Acquisition helpers and the supported affix feature set live in
+ <code>dev/README-hunspell-dictionaries.md</code>.
+ An opt-in catalog download accepts an application-supplied
+ <code>DictionaryCatalog</code>, needs
+ <code>-Dopennlp.download.remote=true</code>, and verifies SHA-512 digests.
+ Directives that would change stems when ignored
+ (<code>ICONV</code>, <code>OCONV</code>, <code>COMPLEXPREFIXES</code>,
+ <code>COMPOUNDRULE</code>, <code>IGNORE</code>, <code>KEEPCASE</code>)
+ fail at load time; cosmetic tables such as <code>REP</code> are skipped.
+ A rule that strips a whole stem applies only when the affix file
+ declares <code>FULLSTRIP</code>, as in Hunspell itself.
+ Each affix or dictionary stream is rejected when it exceeds
+ <code>HunspellDictionary.MAX_STREAM_BYTES</code> (64 MiB).
+ </para>
+ </section>
</chapter>