You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@opennlp.apache.org by rz...@apache.org on 2026/09/29 17:55:48 UTC
(opennlp) branch main updated: OPENNLP-1927: Expand Hunspell dictionary support (#1270)
This is an automated email from the ASF dual-hosted git repository.
rzo1 pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/opennlp.git
The following commit(s) were added to refs/heads/main by this push:
new 7b1dccfb6 OPENNLP-1927: Expand Hunspell dictionary support (#1270)
7b1dccfb6 is described below
commit 7b1dccfb6aecfad8f2c85da69e7463078356aef3
Author: Kristian Rickert <kr...@apache.org>
AuthorDate: Tue Sep 29 13:55:41 2026 -0400
OPENNLP-1927: Expand Hunspell dictionary support (#1270)
Extend the Hunspell stemmer to cover more affix file directives and fix
cases where stemming differed from Hunspell: zero-material affixes,
compound restrictions, forbidden word ordering, hidden capitalized forms,
Turkic and mixed case input, the Hungarian hyphen rule, ICONV/OCONV and
BREAK.
Loading is strict by default: HunspellDictionary.load(...) throws an
IOException for an unsupported affix directive. Callers can allow partial
loading and inspect which directives were skipped. In 3.0.0-M6 unsupported
directives were ignored.
Handle a byte order mark and CR-only line endings in dictionary files,
accept NONGRAMSUGGEST and CHECKNUM under strict loading, and read an
underscore in ICONV, OCONV and REP patterns as a space. Bound the word-pair
check for long tokens under compound dictionaries.
Reduce allocations in the stemming path: int[] overloads for the flag
checks, word lists frozen at load, and BREAK anchors parsed once at load.
Assisted-by: Claude Code 2.1.281
Reviewed-by: Kristian Rickert <kr...@apache.org>
Signed-off-by: Kristian Rickert <kr...@apache.org>
Co-authored-by: Claude <no...@anthropic.com>
---
dev/README-hunspell-dictionaries.md | 170 +-
.../tools/stemmer/hunspell/HunspellAffixTable.java | 84 +
.../stemmer/hunspell/HunspellCompoundRule.java | 157 ++
.../tools/stemmer/hunspell/HunspellConversion.java | 204 ++
.../tools/stemmer/hunspell/HunspellDictionary.java | 2061 +++++++++++++++++---
.../tools/stemmer/hunspell/HunspellStemmer.java | 1421 ++++++++++++--
.../hunspell/HunspellCompatibilityTest.java | 480 +++++
.../stemmer/hunspell/HunspellCompletionTest.java | 410 ++++
.../stemmer/hunspell/HunspellConversionTest.java | 65 +
.../hunspell/HunspellDictionaryLoadTest.java | 538 +++++
.../hunspell/HunspellManualExampleTest.java | 28 +
.../hunspell/HunspellRealDictionaryTest.java | 126 --
.../stemmer/hunspell/HunspellStemmerTest.java | 174 +-
opennlp-docs/src/docbkx/stemmer.xml | 147 +-
.../tools/eval/HunspellCompatibilityEval.java | 401 ++++
15 files changed, 5817 insertions(+), 649 deletions(-)
diff --git a/dev/README-hunspell-dictionaries.md b/dev/README-hunspell-dictionaries.md
index 711435d94..9dc0ea417 100644
--- a/dev/README-hunspell-dictionaries.md
+++ b/dev/README-hunspell-dictionaries.md
@@ -19,7 +19,8 @@
The Hunspell stemmer (`opennlp.tools.stemmer.hunspell`) reads a user-supplied
`.dic` word list and its `.aff` affix file. Apache OpenNLP bundles no dictionary
-data. The dictionary's readme states its license.
+data. Retain the upstream copyright notices and full license text with downloaded
+files. A dictionary's license is separate from OpenNLP's Apache License.
## Where dictionaries come from
@@ -78,7 +79,7 @@ CharSequence stem = stemmer.stem("workers");
```
The result depends on the loaded dictionary. The in-tree manual example uses a
-small dictionary and checks that `workers` stems to `worker`.
+small dictionary and checks that `workers` stems to `work`.
The dictionary is immutable and safe to share between threads. The factory creates a
new stemmer for each call, so each thread can use its own instance. A dictionary that
@@ -87,22 +88,165 @@ accordingly; no conversion is required.
## Testing against real dictionaries
-The in-tree tests use project-authored fixtures only. An opt-in test class, `HunspellRealDictionaryTest`, also checks everyday morphology with the LibreOffice `en_US`, `de_DE_frami`, and `hu_HU` dictionaries. Point it at one directory containing all listed `<name>.aff` and `<name>.dic` files. A missing dictionary skips the associated test; a dictionary that cannot be loaded fails it.
+The runtime tests use project-authored fixtures. `HunspellCompatibilityEval` in `opennlp-eval-tests` extends `AbstractEvalTest` and loads the LibreOffice `en_US`, `de_DE_frami`, and `hu_HU` dictionaries from the `hunspell/` directory of `OPENNLP_DATA_DIR`, the shared `opennlp-data.zip` archive every evaluation uses. It checks strict loading, expected inflections, compounds, concurrent stemming, and the stems and recognition recorded from Hunspell as described below.
+
+The dictionary revision is
+[`32b006a2c22a4ac7e8ed3f03346f7b3d85a970a4`](https://github.com/LibreOffice/dictionaries/tree/32b006a2c22a4ac7e8ed3f03346f7b3d85a970a4).
+The archive holds `<name>.aff`, `<name>.dic`, and `README_<name>.txt` for each of
+the three dictionaries under `hunspell/`. The evaluation verifies the MD5 digests of
+the affix and word-list files before loading them, as the other evaluations do, and
+fails when a file is missing or changed. These checks cover selected examples, not
+all possible words or dictionaries.
+
+The English dictionary's `README_en_US.txt` contains the SCOWL and Ispell
+copyright and license notices. The German dictionary is GPL-licensed and must
+not be bundled in an Apache release. The Hungarian dictionary offers MPL-2.0
+or LGPL-3.0-or-later; select MPL-2.0 and retain that license text with the README.
+They are evaluation inputs, not redistributed OpenNLP resources.
+See the [ASF third-party license policy](https://www.apache.org/legal/resolved.html)
+before proposing to bundle any dictionary.
```
-./mvnw test -pl opennlp-core/opennlp-runtime -am \
- -Dtest=HunspellRealDictionaryTest -Dsurefire.failIfNoSpecifiedTests=false \
- -Dopennlp.hunspell.dict.dir=/tmp/hunspell-dicts
+./mvnw test -pl opennlp-eval-tests -am -Peval-tests \
+ -Dtest=HunspellCompatibilityEval -Dsurefire.failIfNoSpecifiedTests=false \
+ -Dopennlp.forkCount=1 -DOPENNLP_DATA_DIR=/path/to/opennlp-data
```
-## What the engine supports
+The evaluation compares 49 input forms with the stems and the recognition recorded
+from Hunspell. Morphological analyses are compared on the fixtures in
+`HunspellCompletionTest` only. It reports exact result-set matches,
+expected differences, unknown-input identity fallbacks, and unexpected results
+separately, and fails on an unexpected result. Expected differences specify the
+complete OpenNLP output for inputs whose recorded reference output differs, so a
+change on either side fails. Concurrency checks compare repeated results with a
+single-threaded reference. They do not measure throughput, and compatibility
+counts are not accuracy scores.
+
+### How the reference results were recorded
-The engine applies `PFX` and `SFX` rules with strip strings and character-class conditions. It supports a prefix and suffix cross-product, a double suffix sequence connected by continuation classes, identity rules in continuation paths, file-wide `FLAG` modes, file-wide `AF` aliases, and the `SET` encoding declaration. Numeric flags range from 1 through 65000.
+The reference outcomes recorded in `HunspellCompatibilityTest`, `HunspellCompletionTest`, and `HunspellCompatibilityEval` come from Hunspell revision [`e184e22c51fe213f4490e9b36998f0ad3e5e606b`](https://github.com/hunspell/hunspell/commit/e184e22c51fe213f4490e9b36998f0ad3e5e606b), built from source and driven through its C API. The project contains no native source, and no test forks a native process; the fixtures and the recorded outputs are what is committed.
+
+The driver used for recording is about thirty lines of C++ against `hunspell.h`: it calls `Hunspell_create(affixPath, dictionaryPath)`, reads one input per line from standard input in the encoding the affix file declares with `SET`, and for each line calls `Hunspell_spell`, `Hunspell_stem`, or `Hunspell_analyze` as selected by a command-line argument, printing one output line per input with multiple stems or analyses joined by a tab, then releases each result list with `Hunspell_free_lis [...]
+
+```sh
+g++ -std=c++17 -O2 -DHUNSPELL_STATIC \
+ -I/path/to/hunspell/src/hunspell \
+ /path/to/hunspell/src/hunspell/*.cxx driver.cc -o hunspell-reference
+```
-Compound decomposition supports `COMPOUNDFLAG`, `COMPOUNDBEGIN`, `COMPOUNDMIDDLE`, `COMPOUNDEND`, `COMPOUNDMIN`, `COMPOUNDWORDMAX`, `COMPOUNDPERMITFLAG`, `COMPOUNDFORBIDFLAG`, `CHECKCOMPOUNDDUP`, `CHECKCOMPOUNDCASE`, and `CHECKCOMPOUNDTRIPLE`. Compound boundaries and minimum lengths use Unicode code points. `NEEDAFFIX` (also named `PSEUDOROOT`), `ONLYINCOMPOUND`, `FORBIDDENWORD`, `CIRCUMFIX`, and `FULLSTRIP` control whether an analysis is accepted.
+Whitespace inside recorded analyses is normalized to single spaces. The fixture tests assert the OpenNLP results and, where recognition deliberately deviates from the recorded reference outcome, name the deviation from the manual; a fixture whose deviation disappears fails, so the recorded outcomes stay honest. To re-record after a reference upgrade, rebuild the driver from the new revision, run the fixtures and the evaluation inputs through it, and update the recorded values.
-Other directives are skipped. Their conversion, suggestion, or advanced compound behavior is not applied by this affix stemmer. Comments and unused metadata may contain legacy-encoded bytes even when the file uses UTF-8. Parsed rules and dictionary text are decoded strictly. Default and `long` flag modes preserve raw one-byte flag values used by published UTF-8 dictionaries. Invalid rule counts, aliases, flags, and compound limits fail during loading. Each affix or dictionary stream is r [...]
+## What the engine supports
+
+The engine applies `PFX` and `SFX` rules with strip strings and character-class conditions. It supports a prefix and suffix cross-product, a double suffix sequence connected by continuation classes, rules that add and strip no material both on their own and in continuation paths, file-wide `FLAG` modes, file-wide `AF` aliases, and the `SET` encoding declaration. Numeric flags range from 1 through 65535, the full range the reference accepts. A number sign starts a comment at the beginning [...]
+
+`COMPLEXPREFIXES` selects 2 prefix levels and 1 suffix level instead of 1
+prefix and 2 suffixes. `ICONV` and `OCONV` use longest-match conversions, and `_` in an `ICONV`, `OCONV` or `REP` pattern means a space, as in Hunspell;
+`IGNORE` removes configured characters from input, entries, and affix material.
+`KEEPCASE`, `CHECKSHARPS`, `LANG`, `WARN`, and `FORBIDWARN` control case variants
+and warning-marked entries. A capitalized word with a further inner capital is also
+tried with a lowercase initial, and the Turkic `LANG` values map the dotted and
+dotless `i` in both case directions. All-uppercase input also matches mixed-case
+entries and flagged all-uppercase entries in their capitalized form, as the
+reference does through hidden capitalized homonyms, so `IPODS` stems to `Ipod`
+while `Ipods` stays unrecognized; these forms take no part in compounds. An
+all-uppercase word with an apostrophe is also tried with the part after the
+apostrophe capitalized, so `L'AFRIQUE` finds an elided article rule. Trailing
+periods are removed before lookup, and one period is restored when only an entry
+listed with it matches, so `texts.` stems to `text` and `etc.` stays `etc.`.
+Under `LANG hu`, the part of a word before a hyphen follows the reference's
+moving rule: it may be a compound whose opening entry carries one of the
+hardwired flags `F`, `G`, or `H`, ignoring compound-forbid and size limits.
+
+Compound decomposition supports positional flags and independent `COMPOUNDRULE`
+patterns, including optional and repeated flags. It applies compound permit and
+forbid flags, word-count limits, `COMPOUNDROOT`, `COMPOUNDSYLLABLE`, duplicate,
+case, triple-letter and pattern restrictions, simplified junctions,
+`COMPOUNDMORESUFFIXES`, and `FORCEUCASE`. `CHECKCOMPOUNDREP` checks both `REP`
+entries and dictionary `ph:` replacements. Compound boundaries and minimum
+lengths use Unicode code points. `BREAK` splits recognized parts recursively;
+the default separators are `-`, `^-`, and `-$`, and `BREAK 0` disables them.
+
+The compound restrictions follow Hunspell in detail. `CHECKCOMPOUNDDUP` compares the two parts joined at each level, so only a repeated closing part rejects a compound. The `CHECKCOMPOUNDREP` and word-pair checks apply to the complete input and to every remainder a further level splits. A junction restored from a `CHECKCOMPOUNDPATTERN` replacement is exempt from the other patterns. A listed spelling whose first homonym carries `COMPOUNDFORBIDFLAG` is barred from every position but the la [...]
+
+`NEEDAFFIX` (also named `PSEUDOROOT`), `ONLYINCOMPOUND`, `FORBIDDENWORD`,
+`CIRCUMFIX`, and `FULLSTRIP` control whether an analysis is accepted. As in
+Hunspell, the first listed homonym decides whether a spelling is
+forbidden, and a forbidden direct or affixed reading also blocks the compound and
+`BREAK` readings of that input. Morphology
+aliases use `AM`; `st:` supplies an explicit stem, `sp:` prepends surface
+material, and `ds:` makes the form derived by the entry's suffixes the stem.
+
+`SYLLABLENUM` supports Hungarian compound syllable adjustments. The deprecated
+`LEMMA_PRESENT` directive is validated but has no effect. Obsolete
+`COMPOUNDFIRST`, `COMPOUNDLAST`, `ONLYROOT`, `HU_KOTOHANGZO`, and `GENERATE`
+metadata have no effect on stemming or analysis in the pinned reference and
+are ignored. The active compound and affix directives remain applicable.
+
+`HunspellStemmer.analyze` is package-private in 3.0. It returns each analysis as
+Hunspell fields separated by spaces, in the order Hunspell prints them, and
+`HunspellCompletionTest` compares that output with results recorded from Hunspell.
+The public API returns stems. The shared `Stemmer` interface is unchanged.
+
+Comments and unused metadata may contain legacy-encoded bytes even when the file uses UTF-8. Parsed rules and dictionary text are decoded strictly. Default and `long` flag modes preserve raw one-byte flag values used by published UTF-8 dictionaries. Invalid rule counts, aliases, flags, and compound limits fail during loading in both modes. Each affix or dictionary stream is rejected when it exceeds `HunspellDictionary.MAX_STREAM_BYTES` (64 MiB).
+
+## Loading policy
+
+`HunspellDictionary.load(...)` defaults to `LoadMode.STRICT`. Unsupported affix
+directives cause an `IOException` identifying the directive and source line.
+Path-based loading includes the affix path. Valid Hunspell dictionaries using
+unsupported features require an explicit choice to load partially.
+
+Unknown directive names cause rejection. Recognized metadata and settings outside stemming,
+such as `NAME`, `TRY`, and `WORDCHARS`, are ignored. `REP` is parsed when
+`CHECKCOMPOUNDREP` makes replacements affect compound recognition; otherwise it
+is unused suggestion data.
+
+Use `ALLOW_PARTIAL` to skip unsupported directives and inspect the diagnostics:
+
+```java
+HunspellDictionary partial = HunspellDictionary.load(
+ Path.of("dictionary.aff"), Path.of("dictionary.dic"),
+ HunspellDictionary.LoadMode.ALLOW_PARTIAL);
+for (HunspellDictionary.UnsupportedDirective diagnostic : partial.getUnsupportedDirectives()) {
+ System.err.println(diagnostic.directive() + " at "
+ + diagnostic.source() + ":" + diagnostic.lineNumber());
+}
+```
-Skipped directives include `ICONV`, `OCONV`, `COMPLEXPREFIXES`, `COMPOUNDRULE`,
-`IGNORE`, and `KEEPCASE`. Loading a dictionary does not apply these rules;
-results can differ from Hunspell for words that need them.
+`getUnsupportedDirectives()` returns an immutable list containing the first
+source location of each unsupported directive in file order. Recognized settings
+outside stemming are excluded from the list. File paths identify file-based
+loads; stream-based loads use `affix stream` as the source description.
+
+Partial loading does not apply skipped behavior. Strict loading does not
+establish complete Hunspell compatibility. The engine does not generate
+inflected forms or spelling suggestions.
+
+Compound search permits at most 64 parts and 2048 candidate checks per spelling
+variant. Recursive word-break search has the same depth and candidate limits.
+Sharp-s case expansion permits at most 64 variants. Compound-rule patterns are
+limited to 4096 flag elements. Results are limited to 2048 distinct stems or
+analyses. These limits can exclude valid analyses; all included candidates
+must pass validation. Simplified triple letters can be restored at multiple
+junctions. `CHECKCOMPOUNDPATTERN` replacement applies at one junction.
+
+Hunspell's `stem()` and `spell()` do not have equivalent acceptance rules.
+The Hunspell stemmer can return a stem for KEEPCASE or FORBIDWARN input rejected by
+the spell checker, or return no stem for accepted complex-prefix and simplified
+compound forms. OpenNLP applies the dictionary restrictions and returns recognized
+compound part stems separately. Tests record Hunspell's stemming and recognition
+results independently. Compatibility requires checking recognition and output.
+
+The German comparisons also distinguish standalone entries from compound-only
+readings. For example, OpenNLP returns `Kind` for `Kinder`; Hunspell's stemmer also
+returns the compound-only `kind` and an identity-affixed `kinder` reading.
+For `Vorschläge`, Hunspell recognizes the input but returns no stem. OpenNLP
+returns the component stems. These are documented differences, not exact matches
+or a general accuracy claim.
+
+Hunspell returns no stem for `well-known` and for German compounds such as
+`Haustür`, while OpenNLP returns the part stems. The evaluation records each of
+these as an expected difference with the complete OpenNLP output, without treating
+the additional output as a general correctness advantage. Prefix-only forms such
+as `undone` stem the same way in both and are exact matches.
diff --git a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/hunspell/HunspellAffixTable.java b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/hunspell/HunspellAffixTable.java
new file mode 100644
index 000000000..0915ba24d
--- /dev/null
+++ b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/hunspell/HunspellAffixTable.java
@@ -0,0 +1,84 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package opennlp.tools.stemmer.hunspell;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+
+/** Affix tables with entry counts and source locations for validation errors. */
+final class HunspellAffixTable {
+
+ /** Prevents construction. */
+ private HunspellAffixTable() { }
+
+ /**
+ * An entry including the directive name.
+ *
+ * @param fields The tokenized entry.
+ * @param line The one-based source line.
+ */
+ record Entry(String[] fields, int line) { }
+
+ /**
+ * Checks a table header, entry count, and field counts.
+ *
+ * @param lines The tokenized affix content.
+ * @param tag The table directive.
+ * @param minimumFields The minimum fields in an entry, including the directive.
+ * @param maximumFields The maximum fields in an entry, including the directive.
+ * @return The entries, or null if the file has no declaration for this table.
+ * @throws IOException Thrown if the header, entries, or count is invalid.
+ */
+ static List<Entry> read(String[][] lines, String tag, int minimumFields, int maximumFields)
+ throws IOException {
+ final List<Entry> entries = new ArrayList<>();
+ int count = -1;
+ for (int i = 0; i < lines.length; i++) {
+ final String[] fields = lines[i];
+ if (fields.length == 0 || !tag.equals(fields[0])) {
+ continue;
+ }
+ if (count < 0) {
+ try {
+ if (fields.length != 2) {
+ throw new NumberFormatException();
+ }
+ count = Integer.parseInt(fields[1]);
+ if (count < 0) {
+ throw new NumberFormatException();
+ }
+ } catch (NumberFormatException e) {
+ throw new IOException("invalid " + tag + " count at line " + (i + 1), e);
+ }
+ } else {
+ if (fields.length < minimumFields || fields.length > maximumFields || entries.size() == count) {
+ throw new IOException("invalid " + tag + " entry at line " + (i + 1));
+ }
+ entries.add(new Entry(fields, i + 1));
+ }
+ }
+ if (count < 0) {
+ return null;
+ }
+ if (entries.size() != count) {
+ throw new IOException(tag + " header specifies " + count + " entries but found " + entries.size());
+ }
+ return entries;
+ }
+}
diff --git a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/hunspell/HunspellCompoundRule.java b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/hunspell/HunspellCompoundRule.java
new file mode 100644
index 000000000..2ec138c34
--- /dev/null
+++ b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/hunspell/HunspellCompoundRule.java
@@ -0,0 +1,157 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package opennlp.tools.stemmer.hunspell;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+
+/** A compound flag sequence with optional and repeated elements. */
+final class HunspellCompoundRule {
+
+ /** Interprets a flag according to the affix file's FLAG setting. */
+ @FunctionalInterface
+ interface FlagReader {
+ /**
+ * Interprets one flag.
+ *
+ * @param text The encoded flag.
+ * @param line The source line.
+ * @return The flag value.
+ * @throws IOException Thrown if the flag is malformed.
+ */
+ int read(String text, int line) throws IOException;
+ }
+
+ /** Maximum elements in one compound pattern. */
+ private static final int MAX_ELEMENTS = 4096;
+
+ private static final String INVALID_PATTERN = "invalid COMPOUNDRULE at line ";
+
+ private final int[] flags;
+ private final char[] repetition;
+
+ /**
+ * Initializes a parsed compound pattern.
+ *
+ * @param flags The required flag at each position.
+ * @param repetition The position's repetition operator or a space.
+ */
+ private HunspellCompoundRule(int[] flags, char[] repetition) {
+ this.flags = flags;
+ this.repetition = repetition;
+ }
+
+ /**
+ * Parses a compound rule.
+ *
+ * @param pattern The rule text.
+ * @param line The source line.
+ * @param reader The flag decoder.
+ * @return The parsed rule.
+ * @throws IOException Thrown if the pattern or a flag is malformed.
+ */
+ static HunspellCompoundRule parse(String pattern, int line, FlagReader reader) throws IOException {
+ final List<Integer> flags = new ArrayList<>();
+ final StringBuilder repetitions = new StringBuilder();
+ for (int at = 0; at < pattern.length();) {
+ final String flag;
+ if (pattern.charAt(at) == '(') {
+ final int end = pattern.indexOf(')', at + 1);
+ if (end <= at + 1) {
+ throw new IOException(INVALID_PATTERN + line);
+ }
+ flag = pattern.substring(at + 1, end);
+ at = end + 1;
+ } else {
+ final int point = pattern.codePointAt(at);
+ if (point == '*' || point == '?' || point == ')') {
+ throw new IOException(INVALID_PATTERN + line);
+ }
+ final int end = at + Character.charCount(point);
+ flag = pattern.substring(at, end);
+ at = end;
+ }
+ flags.add(reader.read(flag, line));
+ if (flags.size() > MAX_ELEMENTS) {
+ throw new IOException("COMPOUNDRULE exceeds " + MAX_ELEMENTS + " elements at line " + line);
+ }
+ if (at < pattern.length() && (pattern.charAt(at) == '*' || pattern.charAt(at) == '?')) {
+ repetitions.append(pattern.charAt(at++));
+ } else {
+ repetitions.append(' ');
+ }
+ }
+ if (flags.isEmpty()) {
+ throw new IOException("empty COMPOUNDRULE at line " + line);
+ }
+ final int[] values = new int[flags.size()];
+ final char[] repetition = new char[flags.size()];
+ for (int i = 0; i < values.length; i++) {
+ values[i] = flags.get(i);
+ repetition[i] = repetitions.charAt(i);
+ }
+ return new HunspellCompoundRule(values, repetition);
+ }
+
+ /**
+ * Tests a sequence of selected homonyms. Each part consumes one flag position.
+ *
+ * @param parts The flags of one selected entry per compound part.
+ * @param complete Whether the sequence must complete the rule.
+ * @return Whether the sequence is permitted by this rule.
+ */
+ boolean matches(List<int[]> parts, boolean complete) {
+ boolean[] states = new boolean[flags.length + 1];
+ states[0] = true;
+ skipOptional(states);
+ for (int[] part : parts) {
+ final boolean[] next = new boolean[states.length];
+ for (int i = 0; i < flags.length; i++) {
+ if (states[i] && HunspellDictionary.contains(part, flags[i])) {
+ next[repetition[i] == '*' ? i : i + 1] = true;
+ }
+ }
+ skipOptional(next);
+ states = next;
+ }
+ if (complete) {
+ return states[flags.length];
+ }
+ for (boolean state : states) {
+ if (state) {
+ return true;
+ }
+ }
+ return false;
+ }
+
+ /**
+ * Advances states through optional pattern elements.
+ *
+ * @param states The active pattern positions.
+ */
+ private void skipOptional(boolean[] states) {
+ for (int i = 0; i < flags.length; i++) {
+ if (states[i] && repetition[i] != ' ') {
+ states[i + 1] = true;
+ }
+ }
+ }
+
+}
diff --git a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/hunspell/HunspellConversion.java b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/hunspell/HunspellConversion.java
new file mode 100644
index 000000000..eb4a9c057
--- /dev/null
+++ b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/hunspell/HunspellConversion.java
@@ -0,0 +1,204 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package opennlp.tools.stemmer.hunspell;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.function.Predicate;
+
+import opennlp.tools.util.StringUtil;
+
+/** Non-recursive longest-match input and output conversions. */
+final class HunspellConversion {
+
+ /** An empty conversion table. */
+ static final HunspellConversion NONE = new HunspellConversion(List.of());
+
+ /** Marks a {@code REP} pattern that must begin the word. */
+ private static final String START_ANCHOR = "^";
+ /** Marks a {@code REP} pattern that must end the word. */
+ private static final String END_ANCHOR = "$";
+ /** Stands for a space in {@code ICONV}, {@code OCONV} and {@code REP} fields. */
+ private static final char SPACE_MARK_CHAR = '_';
+ /** {@link #SPACE_MARK_CHAR} as text, for the anchor checks. */
+ private static final String SPACE_MARK = String.valueOf(SPACE_MARK_CHAR);
+ /** Separates the input and output of a {@code ph:} field. */
+ private static final String PHONETIC_ARROW = "->";
+ /** Ends a {@code ph:} field whose last character is dropped from both sides. */
+ private static final String PHONETIC_WILDCARD = "*";
+
+ /**
+ * A conversion with optional word-boundary requirements.
+ *
+ * @param from The input text.
+ * @param to The output text.
+ * @param initial Whether the match requires the start of the input.
+ * @param terminal Whether the match requires the end of the input.
+ */
+ private record Rule(String from, String to, boolean initial, boolean terminal) { }
+
+ private final List<Rule> rules;
+
+ /**
+ * Constructs an immutable table.
+ *
+ * @param rules The conversion definitions.
+ */
+ private HunspellConversion(List<Rule> rules) {
+ this.rules = List.copyOf(rules);
+ }
+
+ /**
+ * Parses an affix conversion table and validates the entry count.
+ *
+ * @param lines The affix fields indexed by source line.
+ * @param directive The table name.
+ * @return The conversion table.
+ * @throws IOException Thrown if the table is malformed.
+ */
+ static HunspellConversion parse(String[][] lines, String directive) throws IOException {
+ final List<Rule> rules = new ArrayList<>();
+ final boolean replacement = HunspellDictionary.REPLACEMENT_TAG.equals(directive);
+ final List<HunspellAffixTable.Entry> entries = HunspellAffixTable.read(lines, directive,
+ 3, replacement ? Integer.MAX_VALUE : 3);
+ if (entries == null) {
+ return NONE;
+ }
+ for (HunspellAffixTable.Entry entry : entries) {
+ final String[] fields = entry.fields();
+ final boolean initial = fields[1].startsWith(replacement ? START_ANCHOR : SPACE_MARK);
+ final boolean terminal = fields[1].endsWith(replacement ? END_ANCHOR : SPACE_MARK);
+ final int start = initial ? 1 : 0;
+ final int end = fields[1].length() - (terminal ? 1 : 0);
+ if (end <= start) {
+ throw new IOException("empty " + directive + " pattern at line " + entry.line());
+ }
+ rules.add(new Rule(fields[1].substring(start, end).replace(SPACE_MARK_CHAR, ' '),
+ HunspellDictionary.NO_MATERIAL.equals(fields[2])
+ ? "" : fields[2].replace(SPACE_MARK_CHAR, ' '),
+ initial, terminal));
+ }
+ return rules.isEmpty() ? NONE : new HunspellConversion(rules);
+ }
+
+ /**
+ * Adds dictionary transliterations to the replacement table.
+ *
+ * @param entries The dictionary entries and their flags.
+ * @param morphology The fields of each selected entry.
+ * @return A table containing REP and ph: replacements.
+ */
+ HunspellConversion withPhoneticFields(Map<String, List<int[]>> entries,
+ Map<int[], List<String>> morphology) {
+ final List<Rule> extended = new ArrayList<>(rules);
+ for (Map.Entry<String, List<int[]>> word : entries.entrySet()) {
+ for (int[] flags : word.getValue()) {
+ for (String field : morphology.getOrDefault(flags, List.of())) {
+ if (!field.startsWith(HunspellDictionary.PHONETIC_FIELD)) {
+ continue;
+ }
+ String from = field.substring(HunspellDictionary.PHONETIC_FIELD.length());
+ String to = word.getKey();
+ final int arrow = from.indexOf(PHONETIC_ARROW);
+ if (arrow >= 0) {
+ to = from.substring(arrow + PHONETIC_ARROW.length());
+ from = from.substring(0, arrow);
+ } else if (from.endsWith(PHONETIC_WILDCARD)) {
+ from = from.substring(0, from.length() - PHONETIC_WILDCARD.length());
+ if (!from.isEmpty() && !to.isEmpty()) {
+ from = from.substring(0, from.offsetByCodePoints(from.length(), -1));
+ to = to.substring(0, to.offsetByCodePoints(to.length(), -1));
+ }
+ }
+ if (!from.isEmpty()) {
+ extended.add(new Rule(from, to, false, false));
+ final int first = Character.charCount(from.codePointAt(0));
+ final String upper = StringUtil.toUpperCase(from.substring(0, first)) + from.substring(first);
+ if (!from.equals(upper)) {
+ extended.add(new Rule(upper, to, false, false));
+ }
+ }
+ }
+ }
+ }
+ return new HunspellConversion(extended);
+ }
+
+ /**
+ * Replaces the longest matching pattern at each input position without rescanning
+ * replacement text.
+ *
+ * @param input The input text.
+ * @return The converted text, or the input when no conversion applies.
+ */
+ String apply(String input) {
+ if (rules.isEmpty() || input.isEmpty()) {
+ return input;
+ }
+ final StringBuilder output = new StringBuilder(input.length());
+ boolean changed = false;
+ for (int offset = 0; offset < input.length();) {
+ Rule selected = null;
+ for (Rule rule : rules) {
+ if ((!rule.initial() || offset == 0)
+ && (!rule.terminal() || offset + rule.from().length() == input.length())
+ && input.startsWith(rule.from(), offset)
+ && (selected == null || rule.from().length() > selected.from().length())) {
+ selected = rule;
+ }
+ }
+ if (selected != null) {
+ output.append(selected.to());
+ offset += selected.from().length();
+ changed = true;
+ } else {
+ final int point = input.codePointAt(offset);
+ output.appendCodePoint(point);
+ offset += Character.charCount(point);
+ }
+ }
+ return changed ? output.toString() : input;
+ }
+
+ /**
+ * Tests individual replacements without combining separate corrections.
+ *
+ * @param input The candidate compound.
+ * @param accepted Tests whether a replacement is a recognized non-compound word.
+ * @return Whether a replacement satisfies the test.
+ */
+ boolean anyReplacement(String input, Predicate<String> accepted) {
+ for (int offset = 0; offset < input.length();) {
+ for (Rule rule : rules) {
+ if ((!rule.initial() || offset == 0)
+ && (!rule.terminal() || offset + rule.from().length() == input.length())
+ && input.startsWith(rule.from(), offset)) {
+ final String replacement = input.substring(0, offset) + rule.to()
+ + input.substring(offset + rule.from().length());
+ if (!replacement.equals(input) && accepted.test(replacement)) {
+ return true;
+ }
+ }
+ }
+ offset += Character.charCount(input.codePointAt(offset));
+ }
+ return false;
+ }
+}
diff --git a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/hunspell/HunspellDictionary.java b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/hunspell/HunspellDictionary.java
index 2bb06935d..94aa308a8 100644
--- a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/hunspell/HunspellDictionary.java
+++ b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/hunspell/HunspellDictionary.java
@@ -32,8 +32,11 @@ import java.nio.file.Path;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.HashMap;
+import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
+import java.util.Set;
+import java.util.function.Predicate;
import opennlp.tools.commons.ThreadSafe;
import opennlp.tools.util.StringUtil;
@@ -43,21 +46,24 @@ import opennlp.tools.util.StringUtil;
* {@code .aff} and {@code .dic} files. OpenNLP includes no dictionary data.
*
* <p>Supported affix features are {@code PFX} and {@code SFX} rules with strip
- * strings, character-class conditions, cross-product combinations, and a double suffix
- * connected by continuation classes; {@code FLAG} modes {@code char}, {@code UTF-8},
- * {@code long}, and {@code num}; the {@code AF} alias table; and the {@code SET}
- * encoding declaration. Compound decomposition supports {@code COMPOUNDFLAG},
- * {@code COMPOUNDBEGIN}, {@code COMPOUNDMIDDLE}, {@code COMPOUNDEND},
- * {@code COMPOUNDMIN}, {@code COMPOUNDWORDMAX}, {@code COMPOUNDPERMITFLAG},
- * {@code COMPOUNDFORBIDFLAG}, {@code CHECKCOMPOUNDDUP},
- * {@code CHECKCOMPOUNDCASE}, and {@code CHECKCOMPOUNDTRIPLE}. The blocking flags
+ * strings, character-class conditions, cross-product combinations, and continuation
+ * classes. {@code COMPLEXPREFIXES} selects 2 prefixes and 1 suffix instead of
+ * 1 prefix and 2 suffixes. {@code FLAG}, {@code AF}, {@code AM}, and {@code SET}
+ * declarations apply throughout the file. Input and output conversions, ignored
+ * characters, case restrictions, compound flags and patterns, and word breaks are
+ * applied during stemming. The blocking flags
* {@code NEEDAFFIX} (also named {@code PSEUDOROOT}), {@code ONLYINCOMPOUND}, and
* {@code FORBIDDENWORD}, plus {@code CIRCUMFIX} and {@code FULLSTRIP}, are also
* applied.</p>
*
- * <p>Other directives are skipped. Their spelling, conversion, suggestion, or
- * advanced compound behavior is not applied by this affix stemmer. Dictionary
- * morphology fields are also ignored.</p>
+ * <p>Loading defaults to {@link LoadMode#STRICT}: unsupported affix directives
+ * cause an {@link IOException}. Recognized metadata and suggestion settings that
+ * do not affect stemming are ignored. {@link LoadMode#ALLOW_PARTIAL} skips other
+ * directives and reports them through {@link #getUnsupportedDirectives()}.
+ * Morphological stems use {@code st:}, {@code sp:}, and {@code ds:} fields.
+ * {@code SYLLABLENUM} supports Hungarian compound syllable adjustments.
+ * {@code LEMMA_PRESENT} is accepted and has no effect. Strict loading validates
+ * declarations; it does not establish that every result matches Hunspell.</p>
*
* <p>Instances are immutable and safe to share between threads.</p>
*
@@ -81,6 +87,48 @@ public final class HunspellDictionary {
*/
public static final int MAX_STREAM_BYTES = 64 * 1024 * 1024;
+ /** Controls whether unsupported affix directives prevent dictionary loading. */
+ public enum LoadMode {
+ /** Reject unsupported directives, including unknown directive names. */
+ STRICT,
+ /** Skip unsupported directives and record their first source locations. */
+ ALLOW_PARTIAL
+ }
+
+ /**
+ * The first occurrence of an unsupported affix directive.
+ *
+ * @param directive The directive name.
+ * @param source The affix file path, or {@code affix stream} for stream loading.
+ * @param lineNumber The one-based source line number.
+ */
+ public record UnsupportedDirective(String directive, String source, int lineNumber) {
+
+ /**
+ * Creates a diagnostic with a directive name and source location.
+ *
+ * @param directive The nonblank directive name.
+ * @param source The nonblank source description.
+ * @param lineNumber The positive source line number.
+ * @throws IllegalArgumentException Thrown if a name is null or blank, or the line
+ * number is not positive.
+ */
+ public UnsupportedDirective {
+ if (directive == null || directive.isBlank()) {
+ throw new IllegalArgumentException("directive must not be null or blank");
+ }
+ if (source == null || source.isBlank()) {
+ throw new IllegalArgumentException("source must not be null or blank");
+ }
+ if (lineNumber < 1) {
+ throw new IllegalArgumentException("lineNumber must be positive");
+ }
+ }
+ }
+
+ private static final String AFFIX_STREAM = "affix stream";
+ private static final String DICTIONARY_STREAM = "dictionary stream";
+
/**
* One parsed affix rule of a {@code PFX} or {@code SFX} block.
*
@@ -90,9 +138,19 @@ public final class HunspellDictionary {
* @param affix The surface material the rule adds to the stem.
* @param condition The condition the stem must satisfy for the rule to apply.
* @param continuation The flags of the further affixes that may stack on this one.
+ * @param suffix Whether the rule adds a suffix.
+ * @param morphology The expanded morphological fields, empty when the rule declares none.
+ * @param flagText The flag as written in the rule, reported as the {@code fl:} field of
+ * a rule without morphological fields.
*/
record Affix(int flag, boolean crossProduct, String strip, String affix,
- AffixCondition condition, int[] continuation) {
+ AffixCondition condition, int[] continuation, boolean suffix, List<String> morphology,
+ String flagText) {
+
+ /** {@return the morphological fields, or the {@code fl:} flag field when none are declared} */
+ List<String> analysisFields() {
+ return morphology.isEmpty() ? List.of(FLAG_FIELD + flagText) : morphology;
+ }
/**
* Checks whether a further affix may stack on this one.
@@ -101,12 +159,37 @@ public final class HunspellDictionary {
* @return {@code true} if this affix's continuation classes allow it.
*/
boolean allowsContinuation(int otherFlag) {
- for (final int candidate : continuation) {
- if (candidate == otherFlag) {
- return true;
- }
+ return contains(continuation, otherFlag);
+ }
+ }
+
+ /**
+ * One {@code BREAK} declaration: a separator and the anchors that bind it to the start
+ * or end of the word.
+ *
+ * @param separator The separator text without anchors. Never empty.
+ * @param atStart Whether the separator must begin the word.
+ * @param atEnd Whether the separator must end the word.
+ */
+ record WordBreak(String separator, boolean atStart, boolean atEnd) {
+
+ /**
+ * Parses a declaration such as {@code -}, {@code ^-}, or {@code -$}.
+ *
+ * @param declaration The declared separator with optional anchors.
+ * @param line The one-based source line, for diagnostics.
+ * @return The parsed declaration.
+ * @throws IOException Thrown if the declaration consists of anchors only.
+ */
+ static WordBreak parse(String declaration, int line) throws IOException {
+ final boolean atStart = declaration.startsWith(BREAK_START_ANCHOR);
+ final boolean atEnd = declaration.endsWith(BREAK_END_ANCHOR);
+ final String separator = declaration.substring(atStart ? 1 : 0,
+ declaration.length() - (atEnd ? 1 : 0));
+ if (separator.isEmpty()) {
+ throw new IOException("invalid BREAK at line " + line);
}
- return false;
+ return new WordBreak(separator, atStart, atEnd);
}
}
@@ -135,6 +218,62 @@ public final class HunspellDictionary {
/** The directive that defines the file-wide flag alias table. */
private static final String ALIAS_TAG = "AF";
+ private static final String INPUT_CONVERSION_TAG = "ICONV";
+ private static final String OUTPUT_CONVERSION_TAG = "OCONV";
+ private static final String IGNORE_TAG = "IGNORE";
+ private static final String KEEP_CASE_TAG = "KEEPCASE";
+ private static final String COMPLEX_PREFIXES_TAG = "COMPLEXPREFIXES";
+ private static final String COMPOUND_RULE_TAG = "COMPOUNDRULE";
+ private static final String COMPOUND_PATTERN_TAG = "CHECKCOMPOUNDPATTERN";
+ private static final String MORPHOLOGY_ALIAS_TAG = "AM";
+ private static final String BREAK_TAG = "BREAK";
+ private static final String LANGUAGE_TAG = "LANG";
+ private static final String WARNING_TAG = "WARN";
+ private static final String FORBID_WARNING_TAG = "FORBIDWARN";
+ private static final String CHECK_SHARPS_TAG = "CHECKSHARPS";
+ private static final String COMPOUND_ROOT_TAG = "COMPOUNDROOT";
+ private static final String FORCE_UPPER_CASE_TAG = "FORCEUCASE";
+ private static final String COMPOUND_MORE_SUFFIXES_TAG = "COMPOUNDMORESUFFIXES";
+ private static final String SIMPLIFIED_TRIPLE_TAG = "SIMPLIFIEDTRIPLE";
+ private static final String SYLLABLE_NUMBER_TAG = "SYLLABLENUM";
+ private static final String LEMMA_PRESENT_TAG = "LEMMA_PRESENT";
+ private static final String COMPOUND_SYLLABLE_TAG = "COMPOUNDSYLLABLE";
+ private static final String CHECK_COMPOUND_REP_TAG = "CHECKCOMPOUNDREP";
+ /** The suggestion table whose replacements {@code CHECKCOMPOUNDREP} applies to compounds. */
+ static final String REPLACEMENT_TAG = "REP";
+
+ /** The word-list field that lists a misspelling of its entry. */
+ static final String PHONETIC_FIELD = "ph:";
+ /** Field prefix of a compound part in an analysis. */
+ static final String COMPOUND_PART_FIELD = "pa:";
+ /** The sharp s, which {@code CHECKSHARPS} relates to a double s. */
+ static final String SHARP_S = "ß";
+ /** The lowercase double s that a sharp s expands to. */
+ static final String DOUBLE_S = "ss";
+ /** The uppercase double s that an all-uppercase sharp s entry is written with. */
+ static final String DOUBLE_CAPITAL_S = "SS";
+
+ /** The {@code BREAK} anchor that binds a separator to the start of the word. */
+ static final String BREAK_START_ANCHOR = "^";
+
+ /** The {@code BREAK} anchor that binds a separator to the end of the word. */
+ static final String BREAK_END_ANCHOR = "$";
+
+ /** The dotted capital I, the uppercase of {@code i} in the Turkic languages. */
+ static final String DOTTED_CAPITAL_I = "İ";
+ private static final String STEM_FIELD = "st:";
+ private static final String SURFACE_PREFIX_FIELD = "sp:";
+ private static final String DERIVATIONAL_SUFFIX_FIELD = "ds:";
+ private static final String FLAG_FIELD = "fl:";
+
+ /**
+ * The separators used when the affix file declares no {@code BREAK} table: a hyphen
+ * inside the word, at its start, and at its end.
+ */
+ private static final List<WordBreak> DEFAULT_WORD_BREAKS = List.of(
+ new WordBreak("-", false, false), new WordBreak("-", true, false),
+ new WordBreak("-", false, true));
+
/** Prefix used by comment lines. */
private static final String COMMENT_PREFIX = "#";
@@ -148,19 +287,32 @@ public final class HunspellDictionary {
private static final String SET_TAB_PREFIX = SET_TAG + "\t";
/** The affix format's marker for absent strip or affix material. */
- private static final String NO_MATERIAL = "0";
+ static final String NO_MATERIAL = "0";
/**
- * Largest flag value permitted by {@code FLAG num}, as specified by the
- * <a href="https://github.com/hunspell/hunspell/blob/e184e22c51fe213f4490e9b36998f0ad3e5e606b/man/hunspell.5#L133-L139">
- * Hunspell format manual</a>.
+ * Largest flag value permitted by {@code FLAG num}: the full unsigned 16-bit range.
+ * The <a href="https://github.com/hunspell/hunspell/blob/master/man/hunspell.5">format
+ * manual</a> names 65000, but published dictionaries use values above it and Hunspell
+ * loads them.
*/
- private static final int MAX_NUMERIC_FLAG = 65_000;
+ private static final int MAX_NUMERIC_FLAG = 65_535;
+
+ /**
+ * The flags Hungarian dictionaries reserve for the hyphen rule: an entry carrying one
+ * of them may open a compound written before a hyphen.
+ */
+ private static final int[] HUNGARIAN_HYPHEN_FLAGS = {'F', 'G', 'H'};
/** Largest {@code COMPOUNDMIN} value that can be doubled without overflow. */
private static final int MAX_COMPOUND_MIN = Integer.MAX_VALUE / 2;
private final Map<String, List<int[]>> entries;
+ /**
+ * The capitalized forms of mixed-case and flagged all-uppercase entries, kept as
+ * hidden homonyms so that all-uppercase input matches them. Keyed like
+ * {@link #entries}, sharing the entry flag arrays.
+ */
+ private final Map<String, List<int[]>> hiddenEntries;
private final BoundaryIndex suffixesByLast;
private final List<Affix> suffixesWithoutMaterial;
private final BoundaryIndex prefixesByFirst;
@@ -181,14 +333,41 @@ public final class HunspellDictionary {
private final boolean checkCompoundCase;
private final boolean checkCompoundTriple;
private final boolean fullStrip;
+ private final String ignoredCharacters;
+ private final HunspellConversion inputConversion;
+ private final HunspellConversion outputConversion;
+ private final Map<int[], List<String>> morphology;
+ private final boolean complexPrefixes;
+ private final int keepCase;
+ private final int warningFlag;
+ private final boolean forbidWarn;
+ private final boolean checkSharps;
+ private final boolean turkicCase;
+ private final boolean hungarian;
+ private final boolean syllableNumber;
+ private final List<HunspellCompoundRule> compoundRules;
+ private final List<CompoundPattern> compoundPatterns;
+ private final int compoundRoot;
+ private final int forceUpperCase;
+ private final boolean compoundMoreSuffixes;
+ private final boolean simplifiedTriple;
+ private final boolean checkCompoundRep;
+ private final HunspellConversion replacements;
+ private final int maxCompoundSyllables;
+ private final String compoundVowels;
+ private final List<WordBreak> wordBreaks;
+ private final List<UnsupportedDirective> unsupportedDirectives;
+ private final int longestSpacedForm;
/**
* Initializes the dictionary from the two parsed files.
*
* @param entries The words mapped to the flag sets of their entries.
* @param affix The parsed affix file.
+ * @param unsupportedDirectives The skipped directives in encounter order.
*/
- private HunspellDictionary(Map<String, List<int[]>> entries, AffixFile affix) {
+ private HunspellDictionary(Map<String, List<int[]>> entries, AffixFile affix,
+ List<UnsupportedDirective> unsupportedDirectives) {
this.compoundFlag = affix.compoundFlag;
this.compoundBegin = affix.compoundBegin;
this.compoundEnd = affix.compoundEnd;
@@ -205,7 +384,37 @@ public final class HunspellDictionary {
this.checkCompoundCase = affix.checkCompoundCase;
this.checkCompoundTriple = affix.checkCompoundTriple;
this.fullStrip = affix.fullStrip;
- this.entries = entries;
+ this.ignoredCharacters = affix.ignoredCharacters;
+ this.inputConversion = affix.inputConversion;
+ this.outputConversion = affix.outputConversion;
+ this.morphology = Map.copyOf(affix.entryMorphology);
+ this.complexPrefixes = affix.complexPrefixes;
+ this.keepCase = affix.keepCase;
+ this.warningFlag = affix.warningFlag;
+ this.forbidWarn = affix.forbidWarn;
+ this.checkSharps = affix.checkSharps;
+ this.compoundRules = List.copyOf(affix.compoundRules);
+ this.compoundPatterns = List.copyOf(affix.compoundPatterns);
+ this.compoundRoot = affix.compoundRoot;
+ this.forceUpperCase = affix.forceUpperCase;
+ this.compoundMoreSuffixes = affix.compoundMoreSuffixes;
+ this.simplifiedTriple = affix.simplifiedTriple;
+ this.checkCompoundRep = affix.checkCompoundRep;
+ this.replacements = affix.checkCompoundRep
+ ? affix.replacements.withPhoneticFields(entries, affix.entryMorphology) : affix.replacements;
+ this.maxCompoundSyllables = affix.maxCompoundSyllables;
+ this.compoundVowels = affix.compoundVowels;
+ this.wordBreaks = List.copyOf(affix.wordBreaks);
+ this.turkicCase = affix.language.equals("tr") || affix.language.startsWith("tr_")
+ || affix.language.equals("az") || affix.language.startsWith("az_")
+ || affix.language.equals("crh") || affix.language.startsWith("crh_");
+ this.hungarian = affix.language.equals("hu") || affix.language.startsWith("hu_");
+ this.syllableNumber = affix.syllableNumber;
+ this.entries = immutableFlagSets(entries);
+ this.hiddenEntries = immutableFlagSets(hiddenCapitalizedEntries(entries));
+ this.unsupportedDirectives = List.copyOf(unsupportedDirectives);
+ this.longestSpacedForm = longestSpacedForm(this.entries.keySet(), affix.prefixes,
+ affix.suffixes);
// A material-bearing rule can only be undone from a word whose boundary
// character matches its affix material, so bucketing by that character
// narrows each scan to one bucket plus the strip-only rules.
@@ -217,6 +426,54 @@ public final class HunspellDictionary {
this.prefixesWithoutMaterial = List.copyOf(prefixesWithout);
}
+ /**
+ * Bounds the length of a word containing a space that can have a reading. Such a
+ * reading needs an entry or affix material with a space, and it is found by undoing
+ * at most two prefixes and two suffixes, each adding at most the longest affix
+ * material; a case variant is at least half as long as its surface form, because
+ * the sharp s variant replaces two letters with one.
+ *
+ * @param words The listed spellings.
+ * @param prefixes The prefix rules.
+ * @param suffixes The suffix rules.
+ * @return The longest possible UTF-16 length, or -1 when no spaced form has a reading.
+ */
+ private static int longestSpacedForm(Set<String> words, List<Affix> prefixes,
+ List<Affix> suffixes) {
+ boolean spacedAffix = false;
+ int prefixMaterial = 0;
+ for (Affix prefix : prefixes) {
+ prefixMaterial = Math.max(prefixMaterial, prefix.affix().length());
+ spacedAffix |= prefix.affix().indexOf(' ') >= 0;
+ }
+ int suffixMaterial = 0;
+ for (Affix suffix : suffixes) {
+ suffixMaterial = Math.max(suffixMaterial, suffix.affix().length());
+ spacedAffix |= suffix.affix().indexOf(' ') >= 0;
+ }
+ int longestWord = -1;
+ for (String word : words) {
+ if (spacedAffix || word.indexOf(' ') >= 0) {
+ longestWord = Math.max(longestWord, word.length());
+ }
+ }
+ if (longestWord < 0) {
+ return -1;
+ }
+ return 2 * (longestWord + 2 * prefixMaterial + 2 * suffixMaterial);
+ }
+
+ /**
+ * Tests whether a word containing a space can have a reading, so that the compound
+ * word-pair check can skip texts no reading can match.
+ *
+ * @param length The UTF-16 length of the spaced word.
+ * @return {@code false} if no word of this length containing a space has a reading.
+ */
+ boolean mayReadSpacedForm(int length) {
+ return length <= longestSpacedForm;
+ }
+
/**
* An immutable index of affix rules keyed by the boundary code point of their affix
* material, answering each lookup by binary search so the per-word scans in
@@ -287,30 +544,52 @@ public final class HunspellDictionary {
}
/**
- * Loads a dictionary from its two files.
+ * Loads affix and dictionary files using {@link LoadMode#STRICT}.
*
* @param affixFile The {@code .aff} affix file. Must not be {@code null}.
* @param dictionaryFile The {@code .dic} word list. Must not be {@code null}.
* @return The loaded dictionary. Never {@code null}.
- * @throws IOException Thrown if reading fails or a file is malformed.
+ * @throws IOException Thrown if reading fails, a file is malformed, or an affix
+ * directive is unsupported.
* @throws IllegalArgumentException Thrown if a parameter is {@code null}.
*/
public static HunspellDictionary load(Path affixFile, Path dictionaryFile)
throws IOException {
+ return load(affixFile, dictionaryFile, LoadMode.STRICT);
+ }
+
+ /**
+ * Loads affix and dictionary files with the selected directive policy.
+ *
+ * @param affixFile The {@code .aff} affix file. Must not be {@code null}.
+ * @param dictionaryFile The {@code .dic} word list. Must not be {@code null}.
+ * @param mode The loading policy. Must not be {@code null}.
+ * @return The loaded dictionary. Never {@code null}.
+ * @throws IOException Thrown if reading fails, a file exceeds
+ * {@link #MAX_STREAM_BYTES}, a file is malformed, or {@link LoadMode#STRICT}
+ * encounters an unsupported affix directive.
+ * @throws IllegalArgumentException Thrown if a parameter is {@code null}.
+ */
+ public static HunspellDictionary load(Path affixFile, Path dictionaryFile,
+ LoadMode mode) throws IOException {
if (affixFile == null) {
throw new IllegalArgumentException("affixFile must not be null");
}
if (dictionaryFile == null) {
throw new IllegalArgumentException("dictionaryFile must not be null");
}
+ if (mode == null) {
+ throw new IllegalArgumentException("mode must not be null");
+ }
try (InputStream affix = Files.newInputStream(affixFile);
InputStream dictionary = Files.newInputStream(dictionaryFile)) {
- return load(affix, dictionary);
+ return loadStreams(affix, dictionary, mode, affixFile.toString());
}
}
/**
- * Loads a dictionary from its two streams. Each stream is buffered up to
+ * Loads affix and dictionary streams using {@link LoadMode#STRICT}.
+ * Each stream is buffered up to
* {@link #MAX_STREAM_BYTES} bytes; a larger stream fails with {@link IOException}.
*
* @param affixStream The {@code .aff} affix content. Must not be {@code null}. Not
@@ -319,33 +598,85 @@ public final class HunspellDictionary {
* {@code null}. Not closed.
* @return The loaded dictionary. Never {@code null}.
* @throws IOException Thrown if reading fails, a stream exceeds
- * {@link #MAX_STREAM_BYTES}, or the content is malformed.
+ * {@link #MAX_STREAM_BYTES}, the content is malformed, or an affix directive
+ * is unsupported.
* @throws IllegalArgumentException Thrown if a parameter is {@code null}.
*/
public static HunspellDictionary load(InputStream affixStream,
InputStream dictionaryStream) throws IOException {
+ return load(affixStream, dictionaryStream, LoadMode.STRICT);
+ }
+
+ /**
+ * Loads affix and dictionary streams with the selected directive policy.
+ * Each stream is buffered up to {@link #MAX_STREAM_BYTES} bytes.
+ *
+ * @param affixStream The {@code .aff} affix content. Must not be {@code null}. Not
+ * closed.
+ * @param dictionaryStream The {@code .dic} word list content. Must not be
+ * {@code null}. Not closed.
+ * @param mode The loading policy. Must not be {@code null}.
+ * @return The loaded dictionary. Never {@code null}. Under
+ * {@link LoadMode#ALLOW_PARTIAL} it reports skipped directives through
+ * {@link #getUnsupportedDirectives()}.
+ * @throws IOException Thrown if reading fails, a stream exceeds
+ * {@link #MAX_STREAM_BYTES}, the content is malformed, or
+ * {@link LoadMode#STRICT} encounters an unsupported affix directive.
+ * @throws IllegalArgumentException Thrown if a parameter is {@code null}.
+ */
+ public static HunspellDictionary load(InputStream affixStream,
+ InputStream dictionaryStream, LoadMode mode) throws IOException {
if (affixStream == null) {
throw new IllegalArgumentException("affixStream must not be null");
}
if (dictionaryStream == null) {
throw new IllegalArgumentException("dictionaryStream must not be null");
}
- byte[] affixBytes = readBounded(affixStream, MAX_STREAM_BYTES, "affix stream");
+ if (mode == null) {
+ throw new IllegalArgumentException("mode must not be null");
+ }
+ return loadStreams(affixStream, dictionaryStream, mode, AFFIX_STREAM);
+ }
+
+ /**
+ * Returns skipped unsupported directives, one entry per name in encounter order.
+ * Recognized settings outside stemming, such as suggestion tables, are excluded.
+ *
+ * @return An immutable list; empty for a dictionary loaded in strict mode.
+ */
+ public List<UnsupportedDirective> getUnsupportedDirectives() {
+ return unsupportedDirectives;
+ }
+
+ /**
+ * Loads validated streams without closing them.
+ *
+ * @param affixStream The affix input.
+ * @param dictionaryStream The word-list input.
+ * @param mode The directive policy.
+ * @param source The affix source used in unsupported-directive diagnostics.
+ * @return The parsed dictionary.
+ * @throws IOException Thrown if reading, validation, or parsing fails.
+ */
+ private static HunspellDictionary loadStreams(InputStream affixStream,
+ InputStream dictionaryStream, LoadMode mode, String source) throws IOException {
+ byte[] affixBytes = readBounded(affixStream, MAX_STREAM_BYTES, AFFIX_STREAM);
final Charset charset = declaredCharset(affixBytes);
- maskIgnoredAffixLines(affixBytes);
+ final List<UnsupportedDirective> unsupported = maskIgnoredAffixLines(
+ affixBytes, mode, source);
final boolean rawUtf8Flags = StandardCharsets.UTF_8.equals(charset)
&& !usesUnicodeOrNumericFlags(affixBytes);
affixBytes = normalizeUtf8ByteFlags(affixBytes, charset);
- final AffixFile affix = parseAffix(decode(affixBytes, charset, "affix stream"));
+ final AffixFile affix = parseAffix(decode(affixBytes, charset, AFFIX_STREAM));
byte[] dictionaryBytes = readBounded(dictionaryStream, MAX_STREAM_BYTES,
- "dictionary stream");
+ DICTIONARY_STREAM);
if (rawUtf8Flags) {
dictionaryBytes = normalizeDictionaryByteFlags(dictionaryBytes);
}
final Map<String, List<int[]>> entries = parseWordList(
- decode(dictionaryBytes, charset, "dictionary stream"),
- affix.flagMode, affix.flagAliases);
- return new HunspellDictionary(entries, affix);
+ decode(dictionaryBytes, charset, DICTIONARY_STREAM),
+ affix);
+ return new HunspellDictionary(entries, affix, unsupported);
}
/**
@@ -355,12 +686,20 @@ public final class HunspellDictionary {
* while malformed bytes in parsed directives are still reported.
*
* @param bytes The buffered affix file, modified in place.
+ * @param mode The directive policy applied before masking.
+ * @param source The affix source description.
+ * @return Unsupported directives skipped in partial mode, in encounter order.
+ * @throws IOException Thrown if strict loading encounters an unsupported directive.
*/
- private static void maskIgnoredAffixLines(byte[] bytes) {
+ private static List<UnsupportedDirective> maskIgnoredAffixLines(byte[] bytes,
+ LoadMode mode, String source) throws IOException {
+ final Map<String, UnsupportedDirective> unsupported = new LinkedHashMap<>();
+ final boolean useReplacements = hasAffixDirective(bytes, CHECK_COMPOUND_REP_TAG);
int lineStart = 0;
+ int lineNumber = 1;
for (int i = 0; i <= bytes.length; i++) {
if (i == bytes.length || bytes[i] == '\n' || bytes[i] == '\r') {
- int fieldStart = lineStart;
+ int fieldStart = lineStart == 0 ? Math.min(byteOrderMarkLength(bytes), i) : lineStart;
while (fieldStart < i && isAsciiFieldSpace(bytes[fieldStart])) {
fieldStart++;
}
@@ -368,42 +707,181 @@ public final class HunspellDictionary {
while (fieldEnd < i && !isAsciiFieldSpace(bytes[fieldEnd])) {
fieldEnd++;
}
+ boolean parsed = false;
if (fieldStart < fieldEnd && bytes[fieldStart] != '#') {
final String directive = new String(bytes, fieldStart,
fieldEnd - fieldStart, StandardCharsets.US_ASCII);
- if (isParsedAffixDirective(directive)) {
- maskInlineComment(bytes, fieldEnd, i);
- lineStart = i + 1;
- continue;
+ if (isParsedAffixDirective(directive) && (!REPLACEMENT_TAG.equals(directive) || useReplacements)) {
+ maskInlineComment(bytes, fieldStart, i, directive);
+ parsed = true;
+ } else if (!isIgnoredAffixDirective(directive)) {
+ if (mode == LoadMode.STRICT) {
+ throw new IOException("unsupported affix directive " + directive
+ + " in " + source + " at line " + lineNumber
+ + "; use LoadMode.ALLOW_PARTIAL to load without this behavior");
+ }
+ if (!unsupported.containsKey(directive)) {
+ unsupported.put(directive,
+ new UnsupportedDirective(directive, source, lineNumber));
+ }
}
}
- Arrays.fill(bytes, lineStart, i, (byte) ' ');
+ if (!parsed) {
+ Arrays.fill(bytes, lineStart, i, (byte) ' ');
+ }
+ if (i < bytes.length && bytes[i] == '\r'
+ && i + 1 < bytes.length && bytes[i + 1] == '\n') {
+ i++;
+ }
lineStart = i + 1;
+ lineNumber++;
}
}
+ return List.copyOf(unsupported.values());
+ }
+
+ /**
+ * Measures a leading UTF-8 byte order mark, which the byte-level scans skip so that
+ * the first line's directive is read without it.
+ *
+ * @param bytes The affix content.
+ * @return 3 when the content starts with a UTF-8 byte order mark, otherwise 0.
+ */
+ private static int byteOrderMarkLength(byte[] bytes) {
+ return bytes.length >= 3 && bytes[0] == (byte) 0xef
+ && bytes[1] == (byte) 0xbb && bytes[2] == (byte) 0xbf ? 3 : 0;
+ }
+
+ /**
+ * Finds a directive before metadata is removed or decoded.
+ *
+ * @param bytes The affix content.
+ * @param directive The requested first field.
+ * @return Whether the directive occurs outside a comment.
+ */
+ private static boolean hasAffixDirective(byte[] bytes, String directive) {
+ final int[] starts = new int[1];
+ final int[] ends = new int[1];
+ int from = byteOrderMarkLength(bytes);
+ for (int to = from; to <= bytes.length; to++) {
+ if (to == bytes.length || bytes[to] == '\r' || bytes[to] == '\n') {
+ if (findAsciiFields(bytes, from, to, starts, ends) > 0
+ && directive.equals(asciiField(bytes, starts[0], ends[0]))) {
+ return true;
+ }
+ from = to + 1;
+ }
+ }
+ return false;
+ }
+
+ /**
+ * Identifies metadata, suggestion and command-line settings unused by stemming.
+ * REP is parsed separately when CHECKCOMPOUNDREP makes it affect recognition.
+ *
+ * @param directive The affix directive name.
+ * @return Whether the directive can be ignored by the affix stemmer.
+ */
+ private static boolean isIgnoredAffixDirective(String directive) {
+ return switch (directive) {
+ case "NAME", "HOME", "VERSION", "KEY", "TRY", REPLACEMENT_TAG, "MAP", "PHONE",
+ "NOSUGGEST", "MAXCPDSUGS", "MAXNGRAMSUGS", "MAXDIFF", "ONLYMAXDIFF",
+ "NOSPLITSUGS", "SUGSWITHDOTS", "SUBSTANDARD", "WORDCHARS", "NONGRAMSUGGEST",
+ "CHECKNUM",
+ "COMPOUNDFIRST", "COMPOUNDLAST", "ONLYROOT", "HU_KOTOHANGZO", "GENERATE" -> true;
+ default -> false;
+ };
}
- /** {@return whether a byte separates fields in an affix line} */
+ /**
+ * Tests an affix field separator.
+ *
+ * @param value The input byte.
+ * @return Whether the byte separates fields.
+ */
private static boolean isAsciiFieldSpace(byte value) {
return value == ' ' || value == '\t' || value == '\f';
}
- /** Replaces an inline comment that starts after an affix field separator. */
- private static void maskInlineComment(byte[] bytes, int from, int to) {
- boolean fieldStart = false;
- for (int i = from; i < to; i++) {
- if (isAsciiFieldSpace(bytes[i])) {
- fieldStart = true;
- } else if (fieldStart && bytes[i] == '#') {
- Arrays.fill(bytes, i, to, (byte) ' ');
+ /**
+ * Replaces a trailing comment with spaces. A number sign starts a comment only in a
+ * field the directive does not consume, because the format allows {@code #} as a
+ * flag, a separator, affix material, and conversion text. The fields after the
+ * ones a directive reads are not interpreted.
+ *
+ * @param bytes The mutable file content.
+ * @param from The first byte of the directive.
+ * @param to The exclusive end of the line.
+ * @param directive The directive name.
+ */
+ private static void maskInlineComment(byte[] bytes, int from, int to, String directive) {
+ final List<int[]> fields = new ArrayList<>();
+ for (int i = from; i < to;) {
+ while (i < to && isAsciiFieldSpace(bytes[i])) {
+ i++;
+ }
+ final int start = i;
+ while (i < to && !isAsciiFieldSpace(bytes[i])) {
+ i++;
+ }
+ if (i > start) {
+ fields.add(new int[] {start, i});
+ }
+ }
+ final int firstComment = firstCommentField(bytes, fields, directive);
+ for (int index = firstComment; index < fields.size(); index++) {
+ if (bytes[fields.get(index)[0]] == '#') {
+ Arrays.fill(bytes, fields.get(index)[0], to, (byte) ' ');
return;
- } else {
- fieldStart = false;
}
}
}
- /** {@return whether this implementation parses a directive's fields} */
+ /**
+ * Finds the first field index a trailing comment may occupy.
+ *
+ * @param bytes The file content.
+ * @param fields The field boundaries of the line.
+ * @param directive The directive name.
+ * @return The index after the fields the directive consumes.
+ */
+ private static int firstCommentField(byte[] bytes, List<int[]> fields, String directive) {
+ return switch (directive) {
+ case PREFIX_TAG, SUFFIX_TAG -> isAffixHeader(bytes, fields) ? 4 : 5;
+ case REPLACEMENT_TAG, INPUT_CONVERSION_TAG, OUTPUT_CONVERSION_TAG,
+ COMPOUND_PATTERN_TAG, MORPHOLOGY_ALIAS_TAG -> Integer.MAX_VALUE;
+ case COMPOUND_SYLLABLE_TAG -> 3;
+ default -> 2;
+ };
+ }
+
+ /**
+ * Distinguishes an affix block header, which carries a cross-product marker and a
+ * rule count, from a rule line.
+ *
+ * @param bytes The file content.
+ * @param fields The field boundaries of the line.
+ * @return Whether the line is a block header.
+ */
+ private static boolean isAffixHeader(byte[] bytes, List<int[]> fields) {
+ if (fields.size() < 4 || fields.get(2)[1] - fields.get(2)[0] != 1
+ || (bytes[fields.get(2)[0]] != 'Y' && bytes[fields.get(2)[0]] != 'N')) {
+ return false;
+ }
+ for (int i = fields.get(3)[0]; i < fields.get(3)[1]; i++) {
+ if (bytes[i] < '0' || bytes[i] > '9') {
+ return false;
+ }
+ }
+ return true;
+ }
+
+ /**
+ * Tests whether a directive is supported by the parser.
+ *
+ * @param directive The directive name.
+ * @return Whether the parser processes the directive's fields.
+ */
private static boolean isParsedAffixDirective(String directive) {
return switch (directive) {
case SET_TAG, FLAG_TAG, ALIAS_TAG, PREFIX_TAG, SUFFIX_TAG,
@@ -411,7 +889,12 @@ public final class HunspellDictionary {
"COMPOUNDPERMITFLAG", "COMPOUNDFORBIDFLAG", "NEEDAFFIX", "PSEUDOROOT",
"ONLYINCOMPOUND", "FORBIDDENWORD", "CIRCUMFIX", "COMPOUNDMIN",
"COMPOUNDWORDMAX", "CHECKCOMPOUNDDUP", "CHECKCOMPOUNDCASE",
- "CHECKCOMPOUNDTRIPLE", "FULLSTRIP" -> true;
+ "CHECKCOMPOUNDTRIPLE", "FULLSTRIP", INPUT_CONVERSION_TAG, OUTPUT_CONVERSION_TAG,
+ IGNORE_TAG, MORPHOLOGY_ALIAS_TAG, COMPLEX_PREFIXES_TAG, KEEP_CASE_TAG, WARNING_TAG,
+ FORBID_WARNING_TAG, LANGUAGE_TAG, CHECK_SHARPS_TAG,
+ COMPOUND_RULE_TAG, COMPOUND_ROOT_TAG, FORCE_UPPER_CASE_TAG, COMPOUND_MORE_SUFFIXES_TAG,
+ SIMPLIFIED_TRIPLE_TAG, CHECK_COMPOUND_REP_TAG, COMPOUND_PATTERN_TAG,
+ COMPOUND_SYLLABLE_TAG, SYLLABLE_NUMBER_TAG, LEMMA_PRESENT_TAG, REPLACEMENT_TAG, BREAK_TAG -> true;
default -> false;
};
}
@@ -432,8 +915,10 @@ public final class HunspellDictionary {
return bytes;
}
final ByteArrayOutputStream normalized = new ByteArrayOutputStream(bytes.length);
- int lineStart = 0;
- for (int i = 0; i <= bytes.length; i++) {
+ // the byte order mark is copied as it is, so the first line's fields start after it
+ int lineStart = byteOrderMarkLength(bytes);
+ normalized.write(bytes, 0, lineStart);
+ for (int i = lineStart; i <= bytes.length; i++) {
if (i == bytes.length || bytes[i] == '\n' || bytes[i] == '\r') {
writeNormalizedFlagLine(normalized, bytes, lineStart, i);
if (i < bytes.length) {
@@ -445,12 +930,17 @@ public final class HunspellDictionary {
return normalized.toByteArray();
}
- /** {@return whether {@code FLAG UTF-8} or {@code FLAG num} selects non-byte flags} */
+ /**
+ * Tests for UTF-8 or numeric flag encoding.
+ *
+ * @param bytes The affix content.
+ * @return Whether {@code FLAG UTF-8} or {@code FLAG num} is configured.
+ */
private static boolean usesUnicodeOrNumericFlags(byte[] bytes) {
final int[] starts = new int[5];
final int[] fieldEnds = new int[5];
- int lineStart = 0;
- for (int i = 0; i <= bytes.length; i++) {
+ int lineStart = byteOrderMarkLength(bytes);
+ for (int i = lineStart; i <= bytes.length; i++) {
if (i == bytes.length || bytes[i] == '\n' || bytes[i] == '\r') {
final int count = findAsciiFields(bytes, lineStart, i, starts, fieldEnds);
if (count >= 2 && FLAG_TAG.equals(
@@ -464,7 +954,14 @@ public final class HunspellDictionary {
return false;
}
- /** Writes one affix line, converting high bytes only within raw flag fields. */
+ /**
+ * Writes one affix line, converting high bytes only within raw flag fields.
+ *
+ * @param target The converted content destination.
+ * @param bytes The source file content.
+ * @param lineStart The first byte of the line.
+ * @param lineEnd The exclusive end of the line.
+ */
private static void writeNormalizedFlagLine(ByteArrayOutputStream target,
byte[] bytes, int lineStart, int lineEnd) {
final int[] starts = new int[5];
@@ -476,7 +973,8 @@ public final class HunspellDictionary {
int continuationEnd = -1;
if (count >= 2) {
final String directive = asciiField(bytes, starts[0], fieldEnds[0]);
- if (ALIAS_TAG.equals(directive) || PREFIX_TAG.equals(directive)
+ if (ALIAS_TAG.equals(directive) || COMPOUND_RULE_TAG.equals(directive)
+ || PREFIX_TAG.equals(directive)
|| SUFFIX_TAG.equals(directive) || isSingleFlagDirective(directive)) {
firstFlagStart = starts[1];
firstFlagEnd = fieldEnds[1];
@@ -490,6 +988,22 @@ public final class HunspellDictionary {
}
}
}
+ if (count >= 3 && COMPOUND_PATTERN_TAG.equals(directive)) {
+ for (int i = starts[1]; i < fieldEnds[1]; i++) {
+ if (bytes[i] == '/') {
+ firstFlagStart = i + 1;
+ firstFlagEnd = fieldEnds[1];
+ break;
+ }
+ }
+ for (int i = starts[2]; i < fieldEnds[2]; i++) {
+ if (bytes[i] == '/') {
+ continuationStart = i + 1;
+ continuationEnd = fieldEnds[2];
+ break;
+ }
+ }
+ }
}
for (int i = lineStart; i < lineEnd; i++) {
final boolean flagByte = i >= firstFlagStart && i < firstFlagEnd
@@ -535,7 +1049,13 @@ public final class HunspellDictionary {
return normalized.toByteArray();
}
- /** Writes a raw byte, converting a high flag byte to the matching UTF-8 code point. */
+ /**
+ * Writes a raw byte, converting a high flag byte to a UTF-8 code point.
+ *
+ * @param target The converted content destination.
+ * @param source The source byte.
+ * @param flagByte Whether this byte represents a flag.
+ */
private static void writeNormalizedByte(ByteArrayOutputStream target, byte source,
boolean flagByte) {
final int value = source & 0xff;
@@ -547,17 +1067,32 @@ public final class HunspellDictionary {
}
}
- /** {@return whether the directive value is one Hunspell flag} */
+ /**
+ * Tests for a directive with one flag argument.
+ *
+ * @param directive The directive name.
+ * @return Whether the value is one Hunspell flag.
+ */
private static boolean isSingleFlagDirective(String directive) {
return switch (directive) {
case "COMPOUNDFLAG", "COMPOUNDBEGIN", "COMPOUNDMIDDLE", "COMPOUNDEND",
"COMPOUNDPERMITFLAG", "COMPOUNDFORBIDFLAG", "NEEDAFFIX", "PSEUDOROOT",
- "ONLYINCOMPOUND", "FORBIDDENWORD", "CIRCUMFIX" -> true;
+ "ONLYINCOMPOUND", "FORBIDDENWORD", "CIRCUMFIX", KEEP_CASE_TAG, WARNING_TAG,
+ COMPOUND_ROOT_TAG, FORCE_UPPER_CASE_TAG, LEMMA_PRESENT_TAG -> true;
default -> false;
};
}
- /** Finds the fields needed to classify one raw line. */
+ /**
+ * Finds the fields needed to classify one raw line.
+ *
+ * @param bytes The source content.
+ * @param from The first byte of the line.
+ * @param to The exclusive end of the line.
+ * @param starts The destination for field start offsets.
+ * @param fieldEnds The destination for exclusive field end offsets.
+ * @return The number of fields found, limited to the destination array length.
+ */
private static int findAsciiFields(byte[] bytes, int from, int to,
int[] starts, int[] fieldEnds) {
int count = 0;
@@ -579,7 +1114,14 @@ public final class HunspellDictionary {
return count;
}
- /** Returns one raw ASCII field. */
+ /**
+ * Extracts one ASCII field.
+ *
+ * @param bytes The source content.
+ * @param from The first byte of the field.
+ * @param to The exclusive end of the field.
+ * @return The field text.
+ */
private static String asciiField(byte[] bytes, int from, int to) {
return new String(bytes, from, to - from, StandardCharsets.US_ASCII);
}
@@ -636,21 +1178,143 @@ public final class HunspellDictionary {
}
/**
- * Looks up a word's flag sets.
+ * Looks up a word's flag sets among the listed entries.
*
* @param word The word exactly as listed.
* @return The flag sets of all matching entries, or {@code null} when absent.
*/
List<int[]> lookup(String word) {
+ return lookup(word, false);
+ }
+
+ /**
+ * Looks up a word's flag sets, optionally including the hidden capitalized forms
+ * that all-uppercase input outside compounds may match. The stored lists are
+ * immutable and are returned as they are; a list is built only when a word has
+ * both listed and hidden entries. Callers do not modify the flag arrays.
+ *
+ * @param word The word exactly as listed or in hidden capitalized form.
+ * @param includeHidden Whether hidden capitalized forms count.
+ * @return The flag sets of all matching entries, or {@code null} when absent.
+ */
+ List<int[]> lookup(String word, boolean includeHidden) {
final List<int[]> found = entries.get(word);
+ final List<int[]> hidden = includeHidden ? hiddenEntries.get(word) : null;
+ if (hidden == null) {
+ return found;
+ }
if (found == null) {
- return null;
+ return hidden;
}
- final List<int[]> copy = new ArrayList<>(found.size());
- for (final int[] flags : found) {
- copy.add(flags.clone());
+ final List<int[]> both = new ArrayList<>(found.size() + hidden.size());
+ both.addAll(found);
+ both.addAll(hidden);
+ return both;
+ }
+
+ /**
+ * Replaces every flag-set list with an immutable copy so that lookups can return
+ * the stored lists directly.
+ *
+ * @param flagSets The words mapped to mutable flag-set lists.
+ * @return The same words mapped to immutable lists.
+ */
+ private static Map<String, List<int[]>> immutableFlagSets(Map<String, List<int[]>> flagSets) {
+ final Map<String, List<int[]>> frozen = new HashMap<>(flagSets.size() * 2);
+ for (final Map.Entry<String, List<int[]>> entry : flagSets.entrySet()) {
+ frozen.put(entry.getKey(), List.copyOf(entry.getValue()));
}
- return copy;
+ return frozen;
+ }
+
+ /**
+ * The capitalization classes of a word.
+ */
+ enum CaseType {
+ /** No uppercase letter. */
+ NOCAP,
+ /** Exactly one uppercase letter, at the start. */
+ INITCAP,
+ /** Every letter uppercase. */
+ ALLCAP,
+ /** An uppercase start and a further uppercase letter among lowercase ones. */
+ HUHINITCAP,
+ /** Uppercase letters after a lowercase start or among lowercase ones. */
+ HUHCAP
+ }
+
+ /**
+ * Classifies a word's capitalization, judging the initial by the first character.
+ *
+ * @param word The word to classify.
+ * @return The capitalization class.
+ */
+ static CaseType caseType(String word) {
+ int letters = 0;
+ int uppers = 0;
+ boolean firstUpper = false;
+ for (int i = 0; i < word.length();) {
+ final int point = word.codePointAt(i);
+ if (Character.isLowerCase(point)) {
+ letters++;
+ } else if (Character.isUpperCase(point) || Character.isTitleCase(point)) {
+ firstUpper |= i == 0;
+ uppers++;
+ letters++;
+ }
+ i += Character.charCount(point);
+ }
+ if (uppers == 0) {
+ return CaseType.NOCAP;
+ }
+ if (uppers == 1 && firstUpper) {
+ return CaseType.INITCAP;
+ }
+ if (uppers == letters) {
+ return CaseType.ALLCAP;
+ }
+ return firstUpper ? CaseType.HUHINITCAP : CaseType.HUHCAP;
+ }
+
+ /**
+ * Derives the hidden capitalized forms of mixed-case entries and of all-uppercase
+ * entries carrying flags, so that all-uppercase input such as {@code EBOOKS} for
+ * {@code eBook} or {@code ACME'S} for {@code ACME} matches them. A form that is
+ * itself listed is not added, and forbidden entries contribute none.
+ *
+ * @param listed The words mapped to the flag sets of their entries.
+ * @return The capitalized forms mapped to the flag sets they share with their entries.
+ */
+ private Map<String, List<int[]>> hiddenCapitalizedEntries(Map<String, List<int[]>> listed) {
+ final Map<String, List<int[]>> hidden = new HashMap<>();
+ for (final Map.Entry<String, List<int[]>> entry : listed.entrySet()) {
+ final CaseType type = caseType(entry.getKey());
+ if (type != CaseType.HUHCAP && type != CaseType.HUHINITCAP && type != CaseType.ALLCAP) {
+ continue;
+ }
+ final String capitalized = upperCaseInitial(lowerCase(entry.getKey()));
+ if (listed.containsKey(capitalized)) {
+ continue;
+ }
+ for (final int[] flags : entry.getValue()) {
+ if ((type == CaseType.ALLCAP && flags.length == 0) || contains(flags, forbiddenWord)) {
+ continue;
+ }
+ hidden.computeIfAbsent(capitalized, key -> new ArrayList<>(1)).add(flags);
+ }
+ }
+ return hidden;
+ }
+
+ /**
+ * The flag sets of a root's entries, listed or hidden capitalized.
+ *
+ * @param root The entry spelling a reading selected.
+ * @return The flag sets in list order. Never {@code null}.
+ */
+ private List<int[]> homonyms(String root) {
+ final List<int[]> found = entries.get(root);
+ return found != null ? found : hiddenEntries.getOrDefault(root, List.of());
}
/**
@@ -670,55 +1334,552 @@ public final class HunspellDictionary {
}
/**
- * The prefix rules whose affix material starts with the given code point, which are
- * the only material-bearing rules that can be undone from a word starting with it.
+ * The prefix rules whose affix material starts with the given code point, which are
+ * the only material-bearing rules that can be undone from a word starting with it.
+ *
+ * @param first The word's first code point.
+ * @return The bucket, possibly empty. Never {@code null}.
+ */
+ List<Affix> prefixesStartingWith(int first) {
+ return prefixesByFirst.bucket(first);
+ }
+
+ /** {@return the strip-only prefix rules, applicable to any word} Never {@code null}. */
+ List<Affix> prefixesWithoutMaterial() {
+ return prefixesWithoutMaterial;
+ }
+
+ /** {@return whether the affix file declares any compounding flag at all} */
+ boolean compoundsDeclared() {
+ return compoundFlag != 0 || compoundBegin != 0 || compoundEnd != 0
+ || compoundMiddle != 0 || !compoundRules.isEmpty();
+ }
+
+ /** {@return the parsed independent compound flag patterns} */
+ List<HunspellCompoundRule> compoundRules() {
+ return compoundRules;
+ }
+
+ /** {@return the compound-boundary patterns} */
+ List<CompoundPattern> compoundPatterns() {
+ return compoundPatterns;
+ }
+
+ /** {@return whether simplified triple-letter junctions are enabled} */
+ boolean simplifiedTriple() {
+ return simplifiedTriple;
+ }
+
+ /** {@return whether compounds permit an additional suffix level} */
+ boolean compoundMoreSuffixes() {
+ return compoundMoreSuffixes;
+ }
+
+ /** {@return the word separators with their start and end anchors} */
+ List<WordBreak> wordBreaks() {
+ return wordBreaks;
+ }
+
+ /**
+ * Counts an entry toward COMPOUNDWORDMAX.
+ *
+ * @param flags The selected entry flags.
+ * @param affixes The applied rules.
+ * @return One unit, or an additional unit for COMPOUNDROOT.
+ */
+ int compoundUnits(int[] flags, List<Affix> affixes) {
+ int units = contains(flags, compoundRoot) ? 2 : 1;
+ if (hungarian) {
+ for (Affix affix : affixes) {
+ if (!affix.suffix() && countSyllables(affix.affix()) > 1) {
+ units++;
+ }
+ }
+ }
+ return units;
+ }
+
+ /**
+ * Checks word and syllable limits on a completed compound.
+ *
+ * @param syllables The syllable count after language adjustments.
+ * @param units The compound-word count.
+ * @return Whether the limits permit the compound.
+ */
+ boolean compoundSizeAllowed(int syllables, int units) {
+ if (compoundWordMax == 0 || units <= compoundWordMax) {
+ return true;
+ }
+ if (maxCompoundSyllables == 0) {
+ return false;
+ }
+ return syllables <= maxCompoundSyllables;
+ }
+
+ /**
+ * Calculates a component's contribution to the compound syllable limit.
+ *
+ * @param part The component text.
+ * @param flags The selected entry flags.
+ * @param affixes The applied rules in order.
+ * @param last Whether this is the closing component.
+ * @return The adjusted vowel count.
+ */
+ int compoundSyllables(String part, int[] flags, List<Affix> affixes, boolean last) {
+ if (!hungarian) {
+ return last ? countSyllables(part) : 0;
+ }
+ int result = countSyllables(part);
+ if (!last) {
+ return result;
+ }
+ Affix suffix = null;
+ for (Affix affix : affixes) {
+ if (affix.suffix()) {
+ suffix = affix;
+ }
+ }
+ if (affixes.isEmpty() && contains(flags, 'I') && !contains(flags, 'J')) {
+ return result - 1;
+ }
+ if (suffix != null) {
+ if (suffix.continuation().length == 0) {
+ result -= countSyllables(suffix.affix());
+ } else if (suffix.affix().endsWith("i") && !suffix.affix().endsWith("yi")
+ && !suffix.affix().endsWith("ti")) {
+ result--;
+ }
+ if (syllableNumber) {
+ if (suffix.flag() == 'c') {
+ result += 2;
+ } else if (suffix.flag() == 'J' || (suffix.flag() == 'I' && contains(flags, 'J'))) {
+ result++;
+ }
+ }
+ }
+ return result;
+ }
+
+ /**
+ * Counts code points listed by COMPOUNDSYLLABLE.
+ *
+ * @param text The text to count.
+ * @return The vowel count.
+ */
+ private int countSyllables(String text) {
+ int syllables = 0;
+ for (int at = 0; at < text.length();) {
+ final int point = text.codePointAt(at);
+ if (compoundVowels.indexOf(point) >= 0) {
+ syllables++;
+ }
+ at += Character.charCount(point);
+ }
+ return syllables;
+ }
+
+ /**
+ * {@return whether the Hungarian moving rule applies} The part of a Hungarian word
+ * before a hyphen is checked as a compound with relaxed rules: the opening part may
+ * qualify through the reserved flags {@code F}, {@code G}, and {@code H}, a
+ * compound-forbidden opening entry is allowed, and the size limits do not apply.
+ */
+ boolean hyphenMovingRule() {
+ return hungarian;
+ }
+
+ /**
+ * Checks whether an entry may open a compound under the Hungarian moving rule.
+ *
+ * @param flags One entry's flag set.
+ * @return {@code true} if the entry carries one of the hardwired opening flags and is
+ * not forbidden.
+ */
+ boolean opensHyphenatedCompound(int[] flags) {
+ if (contains(flags, forbiddenWord)) {
+ return false;
+ }
+ for (final int flag : HUNGARIAN_HYPHEN_FLAGS) {
+ if (contains(flags, flag)) {
+ return true;
+ }
+ }
+ return false;
+ }
+
+ /**
+ * Checks a closing entry's capitalization requirement.
+ *
+ * @param flags The closing entry flags.
+ * @param word The input compound.
+ * @return Whether FORCEUCASE permits the input.
+ */
+ boolean compoundCaseAllowed(int[] flags, String word) {
+ return !contains(flags, forceUpperCase) || Character.isUpperCase(word.codePointAt(0))
+ || Character.isTitleCase(word.codePointAt(0));
+ }
+
+ /**
+ * Tests CHECKCOMPOUNDREP with the configured replacement table.
+ *
+ * @param word The candidate compound.
+ * @param recognized Tests a non-compound form.
+ * @return Whether a replacement invalidates the compound reading.
+ */
+ boolean rejectsCompoundReplacement(String word, Predicate<String> recognized) {
+ return checkCompoundRep && replacements.anyReplacement(word, recognized);
+ }
+
+ /**
+ * A compound-boundary restriction and optional spelling replacement.
+ *
+ * @param end The left-side ending.
+ * @param endFlag The required left flag or zero.
+ * @param begin The right-side beginning.
+ * @param beginFlag The required right flag or zero.
+ * @param unaffixed Whether the left side must have no nonzero affix.
+ * @param replacement The simplified junction, or null.
+ */
+ record CompoundPattern(String end, int endFlag, String begin, int beginFlag,
+ boolean unaffixed, String replacement) {
+ /**
+ * Checks the selected readings at a junction.
+ *
+ * @param left The left part spelling.
+ * @param leftFlags The left entry flags.
+ * @param right The right part spelling.
+ * @param rightFlags The right entry flags.
+ * @param leftAffixes The left part's affixes.
+ * @param rightAffixes The right part's affixes.
+ * @return Whether this pattern applies.
+ */
+ boolean matches(String left, int[] leftFlags, String right, int[] rightFlags,
+ List<Affix> leftAffixes, List<Affix> rightAffixes) {
+ if (!left.endsWith(end) || !right.startsWith(begin)
+ || !hasBoundaryFlag(leftFlags, leftAffixes, endFlag)
+ || !hasBoundaryFlag(rightFlags, rightAffixes, beginFlag)) {
+ return false;
+ }
+ if (unaffixed) {
+ for (Affix affix : leftAffixes) {
+ if (!affix.affix().isEmpty() || !affix.strip().isEmpty()) {
+ return false;
+ }
+ }
+ }
+ return true;
+ }
+
+ /**
+ * Tests entry and continuation flags for a boundary condition.
+ *
+ * @param flags The entry flags.
+ * @param affixes The component's rules.
+ * @param required The required flag, or zero for an unrestricted condition.
+ * @return Whether the selected reading satisfies the condition.
+ */
+ private boolean hasBoundaryFlag(int[] flags, List<Affix> affixes, int required) {
+ if (required == 0 || contains(flags, required)) {
+ return true;
+ }
+ for (Affix affix : affixes) {
+ if (affix.allowsContinuation(required)) {
+ return true;
+ }
+ }
+ return false;
+ }
+ }
+
+ /** {@return whether positional or general compound flags are configured} */
+ boolean positionalCompoundsDeclared() {
+ return compoundFlag != 0 || compoundBegin != 0 || compoundMiddle != 0 || compoundEnd != 0;
+ }
+
+ /**
+ * Checks blocking flags on a compound-rule part.
+ *
+ * @param flags The selected entry flags.
+ * @param last Whether this is the closing part.
+ * @param affixes The applied suffixes.
+ * @return Whether the selected entry and affixes can form a compound part.
+ */
+ boolean acceptsRulePart(int[] flags, boolean last, Affix... affixes) {
+ if (contains(flags, forbiddenWord) || (!last && contains(flags, compoundForbid))
+ || (affixes.length == 0 && contains(flags, needAffix))) {
+ return false;
+ }
+ for (Affix affix : affixes) {
+ if (forbidsInCompound(affix) || circumfixOnly(affix)) {
+ return false;
+ }
+ }
+ return affixes.length != 1 || !needsFurtherAffix(affixes[0]);
+ }
+
+ /** {@return the smallest length a compound part may have} At least {@code 1}. */
+ int compoundMin() {
+ return compoundMin;
+ }
+
+ /** {@return the largest number of parts a compound may have} {@code 0} is unbounded. */
+ int compoundWordMax() {
+ return compoundWordMax;
+ }
+
+ /** {@return whether {@code CHECKCOMPOUNDDUP} forbids a part repeating its neighbor} */
+ boolean checkCompoundDup() {
+ return checkCompoundDup;
+ }
+
+ /** {@return whether {@code CHECKCOMPOUNDCASE} forbids uppercase at part boundaries} */
+ boolean checkCompoundCase() {
+ return checkCompoundCase;
+ }
+
+ /** {@return whether {@code CHECKCOMPOUNDTRIPLE} forbids triple letters at boundaries} */
+ boolean checkCompoundTriple() {
+ return checkCompoundTriple;
+ }
+
+ /** {@return whether {@code FULLSTRIP} allows an affix rule to strip a whole stem} */
+ boolean fullStrip() {
+ return fullStrip;
+ }
+
+ /**
+ * Applies input conversion followed by ignored-character removal.
+ *
+ * @param word The supplied word.
+ * @return The dictionary lookup form.
+ */
+ String inputForm(String word) {
+ return removeIgnored(inputConversion.apply(word), ignoredCharacters);
+ }
+
+ /**
+ * Applies output conversion to a recognized stem.
+ *
+ * @param stem The dictionary stem.
+ * @return The converted output.
+ */
+ String outputForm(String stem) {
+ return outputConversion.apply(stem);
+ }
+
+ /** {@return whether prefix continuation allows an additional prefix level} */
+ boolean complexPrefixes() {
+ return complexPrefixes;
+ }
+
+ /** {@return whether uppercase SS may represent a German sharp s} */
+ boolean checkSharps() {
+ return checkSharps;
+ }
+
+ /**
+ * Applies the dictionary's lowercase mapping.
+ *
+ * @param text The input text.
+ * @return The lowercase text.
+ */
+ String lowerCase(String text) {
+ if (!turkicCase) {
+ return StringUtil.toLowerCase(text);
+ }
+ return StringUtil.toLowerCase(text.replace('I', 'ı').replace('İ', 'i'));
+ }
+
+ /**
+ * Checks case-sensitive and warning restrictions for a selected entry.
+ *
+ * @param flags The homonym's flags.
+ * @param surface The input after conversion.
+ * @param variant The case variant under analysis.
+ * @param affixes The applied affixes.
+ * @return Whether this reading is permitted.
+ */
+ boolean acceptsCase(int[] flags, String surface, String variant, Affix... affixes) {
+ if (forbidWarn && contains(flags, warningFlag)) {
+ return false;
+ }
+ boolean preserveCase = contains(flags, keepCase);
+ for (Affix affix : affixes) {
+ if (affix.allowsContinuation(forbiddenWord)
+ || (forbidWarn && affix.allowsContinuation(warningFlag))) {
+ return false;
+ }
+ preserveCase |= keepCase != 0 && affix.allowsContinuation(keepCase);
+ }
+ if (!preserveCase || surface.equals(variant)) {
+ return true;
+ }
+ // CHECKSHARPS lets a case-preserving entry with a sharp s be written all-uppercase
+ // with SS, or capitalized; an all-uppercase form with a capital sharp s stays rejected
+ return checkSharps && variant.contains(SHARP_S)
+ && (surface.contains(DOUBLE_CAPITAL_S) || (caseType(surface) != CaseType.ALLCAP
+ && Character.isUpperCase(surface.codePointAt(0))));
+ }
+
+ /**
+ * Returns a stem after applying entry and affix morphology.
+ *
+ * @param root The dictionary entry spelling.
+ * @param flags The selected homonym's flags.
+ * @param affixes The affixes in application order.
+ * @return The stems of matching homonyms.
+ */
+ List<String> morphologicalStems(String root, int[] flags, Affix... affixes) {
+ final List<String> stems = new ArrayList<>();
+ for (int[] entryFlags : homonyms(root)) {
+ if (Arrays.equals(entryFlags, flags)) {
+ stems.add(morphologicalStem(root, morphology.getOrDefault(entryFlags, List.of()), affixes));
+ }
+ }
+ return stems;
+ }
+
+ /**
+ * Returns morphological fields for matching dictionary entries and applied rules:
+ * prefix fields, then entry fields, then suffix fields. A prefix without morphological fields
+ * contributes its affix text when no suffix follows and its {@code fl:} flag field
+ * otherwise; after the entry fields, an entry without fields contributes that prefix's
+ * {@code fl:} field. A bare closing compound part without entry fields contributes
+ * no stem field.
+ *
+ * @param root The dictionary entry text.
+ * @param flags The selected homonym flags.
+ * @param compoundEnd Whether the reading closes a compound.
+ * @param affixes The applied rules in order.
+ * @return The complete field strings, possibly empty strings.
+ */
+ List<String> morphologicalAnalyses(String root, int[] flags, boolean compoundEnd,
+ Affix... affixes) {
+ final List<String> result = new ArrayList<>();
+ boolean hasSuffix = false;
+ Affix prefixOnly = null;
+ for (Affix affix : affixes) {
+ hasSuffix |= affix.suffix();
+ }
+ for (int[] entryFlags : homonyms(root)) {
+ if (!Arrays.equals(entryFlags, flags)) {
+ continue;
+ }
+ final List<String> fields = new ArrayList<>();
+ for (int i = affixes.length - 1; i >= 0; i--) {
+ final Affix prefix = affixes[i];
+ if (prefix.suffix()) {
+ continue;
+ }
+ if (hasSuffix || !prefix.morphology().isEmpty()) {
+ fields.addAll(prefix.analysisFields());
+ } else {
+ fields.add(prefix.affix());
+ }
+ if (!hasSuffix) {
+ prefixOnly = prefix;
+ }
+ }
+ final List<String> entry = morphology.getOrDefault(entryFlags, List.of());
+ if (compoundEnd && affixes.length == 0 && entry.isEmpty()) {
+ result.add("");
+ continue;
+ }
+ if (!hasField(entry, STEM_FIELD)) {
+ fields.add(STEM_FIELD + root);
+ }
+ fields.addAll(entry);
+ if (entry.isEmpty() && prefixOnly != null) {
+ fields.add(FLAG_FIELD + prefixOnly.flagText());
+ }
+ for (Affix affix : affixes) {
+ if (affix.suffix()) {
+ fields.addAll(affix.analysisFields());
+ }
+ }
+ result.add(String.join(" ", fields));
+ }
+ return result;
+ }
+
+ /**
+ * Applies morphological stem and affix fields to one dictionary entry.
*
- * @param first The word's first code point.
- * @return The bucket, possibly empty. Never {@code null}.
+ * @param root The entry spelling.
+ * @param fields The entry's morphological fields.
+ * @param affixes The affixes in application order.
+ * @return The morphological stem.
*/
- List<Affix> prefixesStartingWith(int first) {
- return prefixesByFirst.bucket(first);
- }
-
- /** {@return the strip-only prefix rules, applicable to any word} Never {@code null}. */
- List<Affix> prefixesWithoutMaterial() {
- return prefixesWithoutMaterial;
- }
-
- /** {@return whether the affix file declares any compounding flag at all} */
- boolean compoundsDeclared() {
- return compoundFlag != 0 || compoundBegin != 0 || compoundEnd != 0
- || compoundMiddle != 0;
- }
-
- /** {@return the smallest length a compound part may have} At least {@code 1}. */
- int compoundMin() {
- return compoundMin;
- }
-
- /** {@return the largest number of parts a compound may have} {@code 0} is unbounded. */
- int compoundWordMax() {
- return compoundWordMax;
- }
-
- /** {@return whether {@code CHECKCOMPOUNDDUP} forbids a part repeating its neighbor} */
- boolean checkCompoundDup() {
- return checkCompoundDup;
+ private String morphologicalStem(String root, List<String> fields, Affix... affixes) {
+ String result = fieldValue(fields, STEM_FIELD, root);
+ if (hasField(fields, DERIVATIONAL_SUFFIX_FIELD)) {
+ result = root;
+ }
+ // A derivational suffix makes the derived form the stem. That form is built from
+ // the entry and its suffixes alone; prefix material appears in the stem only
+ // through a surface prefix field.
+ String derived = root;
+ final StringBuilder surfacePrefix = new StringBuilder(fieldValue(fields, SURFACE_PREFIX_FIELD, ""));
+ for (Affix affix : affixes) {
+ if (affix.suffix()) {
+ derived = derived.substring(0, derived.length() - affix.strip().length()) + affix.affix();
+ if (hasField(affix.morphology(), DERIVATIONAL_SUFFIX_FIELD)) {
+ result = derived;
+ }
+ }
+ surfacePrefix.append(fieldValue(affix.morphology(), SURFACE_PREFIX_FIELD, ""));
+ }
+ return surfacePrefix.append(result).toString();
}
- /** {@return whether {@code CHECKCOMPOUNDCASE} forbids uppercase at part boundaries} */
- boolean checkCompoundCase() {
- return checkCompoundCase;
+ /**
+ * Finds a morphological field value.
+ *
+ * @param fields The expanded fields.
+ * @param tag The tag including the colon.
+ * @param fallback The value used when the tag is not present.
+ * @return The first field value or fallback.
+ */
+ private String fieldValue(List<String> fields, String tag, String fallback) {
+ for (String field : fields) {
+ if (field.startsWith(tag)) {
+ return field.substring(tag.length());
+ }
+ }
+ return fallback;
}
- /** {@return whether {@code CHECKCOMPOUNDTRIPLE} forbids triple letters at boundaries} */
- boolean checkCompoundTriple() {
- return checkCompoundTriple;
+ /**
+ * Checks for a morphological field.
+ *
+ * @param fields The expanded fields.
+ * @param tag The tag including the colon.
+ * @return Whether the tag is present.
+ */
+ private boolean hasField(List<String> fields, String tag) {
+ return fieldValue(fields, tag, null) != null;
}
- /** {@return whether {@code FULLSTRIP} allows an affix rule to strip a whole stem} */
- boolean fullStrip() {
- return fullStrip;
+ /**
+ * Removes characters listed in the affix file's IGNORE setting.
+ *
+ * @param text The word or affix material.
+ * @param ignored The characters to remove.
+ * @return Text without the ignored characters.
+ */
+ private static String removeIgnored(String text, String ignored) {
+ if (ignored.isEmpty()) {
+ return text;
+ }
+ final StringBuilder result = new StringBuilder(text.length());
+ for (int i = 0; i < text.length();) {
+ final int point = text.codePointAt(i);
+ if (ignored.indexOf(point) < 0) {
+ result.appendCodePoint(point);
+ }
+ i += Character.charCount(point);
+ }
+ return result.length() == text.length() ? text : result.toString();
}
/**
@@ -747,17 +1908,27 @@ public final class HunspellDictionary {
* @return {@code true} if the word may stand at the position.
*/
boolean mayStand(List<int[]> flagSets, CompoundPosition position) {
- final int positional = positionalFlag(position);
for (final int[] flags : flagSets) {
- if ((contains(flags, compoundFlag) || contains(flags, positional))
- && !contains(flags, forbiddenWord) && !contains(flags, needAffix)
- && !forbiddenAtCompoundPosition(flags, position)) {
+ if (mayStand(flags, position)) {
return true;
}
}
return false;
}
+ /**
+ * Checks whether one homonym may stand as a bare compound part at a position.
+ *
+ * @param flags One entry's flag set.
+ * @param position The part's place in the compound.
+ * @return {@code true} if the entry may stand at the position.
+ */
+ boolean mayStand(int[] flags, CompoundPosition position) {
+ return (contains(flags, compoundFlag) || contains(flags, positionalFlag(position)))
+ && !contains(flags, forbiddenWord) && !contains(flags, needAffix)
+ && !forbiddenAtCompoundPosition(flags, position);
+ }
+
/**
* Checks whether some homonym supports an affixed compound part. The flag set
* contains the removed affix flag, is not forbidden, and either the affix permits
@@ -772,18 +1943,32 @@ public final class HunspellDictionary {
*/
boolean supportsPart(List<int[]> flagSets, int affixFlag, CompoundPosition position,
boolean affixAdmits) {
- final int positional = positionalFlag(position);
for (final int[] flags : flagSets) {
- if (contains(flags, affixFlag) && !contains(flags, forbiddenWord)
- && !forbiddenAtCompoundPosition(flags, position)
- && (affixAdmits || contains(flags, compoundFlag)
- || contains(flags, positional))) {
+ if (supportsPart(flags, affixFlag, position, affixAdmits)) {
return true;
}
}
return false;
}
+ /**
+ * Checks whether one homonym supports an affixed compound part, under the conditions
+ * of {@link #supportsPart(List, int, CompoundPosition, boolean)}.
+ *
+ * @param flags One entry's flag set.
+ * @param affixFlag The removed affix's flag.
+ * @param position The part's place in the compound.
+ * @param affixAdmits Whether the affix's continuation classes admit the position.
+ * @return {@code true} if the entry permits the affixed form at the position.
+ */
+ boolean supportsPart(int[] flags, int affixFlag, CompoundPosition position,
+ boolean affixAdmits) {
+ return contains(flags, affixFlag) && !contains(flags, forbiddenWord)
+ && !forbiddenAtCompoundPosition(flags, position)
+ && (affixAdmits || contains(flags, compoundFlag)
+ || contains(flags, positionalFlag(position)));
+ }
+
/**
* Checks whether an affix admits its derived form at a compound position: its
* continuation classes carry the general compounding flag or the position's
@@ -826,12 +2011,7 @@ public final class HunspellDictionary {
/**
* Checks whether an entry marked with {@code COMPOUNDFORBIDFLAG} is barred from
- * this compound position. Hunspell permits such an entry only as the last part, as
- * specified by the
- * <a href="https://github.com/hunspell/hunspell/blob/e184e22c51fe213f4490e9b36998f0ad3e5e606b/man/hunspell.5#L502-L506">
- * format manual</a> and the
- * <a href="https://github.com/hunspell/hunspell/blob/e184e22c51fe213f4490e9b36998f0ad3e5e606b/tests/compoundforbid.aff">
- * regression fixture</a>.
+ * this compound position. Such an entry may stand only as the last part.
*
* @param flags One entry's flag set.
* @param position The part's place in the compound.
@@ -842,14 +2022,81 @@ public final class HunspellDictionary {
}
/**
- * Checks whether any of a word's flag sets is forbidden, which a dictionary uses
- * to block one specific ill-formed compound while its parts stay productive.
+ * Checks whether a spelling is forbidden as a whole. Only the first listed homonym
+ * decides, so a dictionary can list a valid word first and a forbidden
+ * compound-only homonym after it.
*
- * @param flagSets The word's flag sets from {@link #lookup(String)}.
- * @return {@code true} if some homonym carries the forbidden-word flag.
+ * @param flagSets The word's flag sets from {@link #lookup(String)}, in list order.
+ * @return {@code true} if the first homonym carries the forbidden-word flag.
+ */
+ boolean firstForbidden(List<int[]> flagSets) {
+ return !flagSets.isEmpty() && contains(flagSets.get(0), forbiddenWord);
+ }
+
+ /**
+ * Checks whether an affix analysis reaches a forbidden entry, which forbids the
+ * affixed spelling and thereby its compound and break readings.
+ *
+ * @param flagSets The stem's flag sets from {@link #lookup(String)}.
+ * @param flag The removed affix's flag.
+ * @return {@code true} if a homonym carries both the affix flag and the forbidden flag.
+ */
+ boolean forbidsAffixed(List<int[]> flagSets, int flag) {
+ for (final int[] flags : flagSets) {
+ if (forbidsAffixed(flags, flag)) {
+ return true;
+ }
+ }
+ return false;
+ }
+
+ /**
+ * Checks whether one homonym carries both an affix flag and the forbidden flag.
+ *
+ * @param flags One entry's flag set.
+ * @param flag The removed affix's flag.
+ * @return {@code true} if the entry forbids the affixed spelling.
+ */
+ boolean forbidsAffixed(int[] flags, int flag) {
+ return contains(flags, flag) && contains(flags, forbiddenWord);
+ }
+
+ /**
+ * Checks whether a spelling may not open a compound part sequence: its first listed
+ * homonym carries {@code COMPOUNDFORBIDFLAG}, which applies to every part but the
+ * last and overrides affixed readings of the same spelling.
+ *
+ * @param flagSets The part's flag sets from {@link #lookup(String)}, in list order.
+ * @return {@code true} if the spelling is barred from non-final compound positions.
+ */
+ boolean forbidsCompoundStart(List<int[]> flagSets) {
+ return !flagSets.isEmpty() && contains(flagSets.get(0), compoundForbid);
+ }
+
+ /**
+ * Converts the initial code point to uppercase with the dictionary's case mapping.
+ *
+ * @param word The nonempty word.
+ * @return The word with an uppercase initial.
+ */
+ String upperCaseInitial(String word) {
+ final int first = word.codePointAt(0);
+ final String initial = word.substring(0, Character.charCount(first));
+ final String upper = turkicCase && first == 'i' ? DOTTED_CAPITAL_I
+ : turkicCase && first == 'ı' ? "I" : StringUtil.toUpperCase(initial);
+ return upper + word.substring(initial.length());
+ }
+
+ /**
+ * Converts the initial code point to lowercase with the dictionary's case mapping.
+ *
+ * @param word The nonempty word.
+ * @return The word with a lowercase initial.
*/
- boolean anyForbidden(List<int[]> flagSets) {
- return hasFlag(flagSets, forbiddenWord);
+ String lowerCaseInitial(String word) {
+ final int first = word.codePointAt(0);
+ final String initial = word.substring(0, Character.charCount(first));
+ return lowerCase(initial) + word.substring(initial.length());
}
/**
@@ -876,7 +2123,7 @@ public final class HunspellDictionary {
* @param flag The flag to look for.
* @return {@code true} if the set contains the flag.
*/
- private static boolean contains(int[] flags, int flag) {
+ static boolean contains(int[] flags, int flag) {
if (flag == 0) {
return false;
}
@@ -900,14 +2147,25 @@ public final class HunspellDictionary {
*/
boolean validStandalone(List<int[]> flagSets) {
for (final int[] flags : flagSets) {
- if (!contains(flags, needAffix) && !contains(flags, onlyInCompound)
- && !contains(flags, forbiddenWord)) {
+ if (validStandalone(flags)) {
return true;
}
}
return false;
}
+ /**
+ * Checks whether one homonym stands on its own: its flag set carries none of the
+ * blocking flags named in {@link #validStandalone(List)}.
+ *
+ * @param flags One entry's flag set.
+ * @return {@code true} if the entry is a word by itself.
+ */
+ boolean validStandalone(int[] flags) {
+ return !contains(flags, needAffix) && !contains(flags, onlyInCompound)
+ && !contains(flags, forbiddenWord);
+ }
+
/**
* Checks whether some homonym supports an affix analysis: its flag set carries the
* affix's flag and is neither compound-only nor forbidden. A {@code NEEDAFFIX} set
@@ -920,14 +2178,26 @@ public final class HunspellDictionary {
*/
boolean supports(List<int[]> flagSets, int flag) {
for (final int[] flags : flagSets) {
- if (contains(flags, flag) && !contains(flags, onlyInCompound)
- && !contains(flags, forbiddenWord)) {
+ if (supports(flags, flag)) {
return true;
}
}
return false;
}
+ /**
+ * Checks whether one homonym supports an affix analysis, under the conditions of
+ * {@link #supports(List, int)}.
+ *
+ * @param flags One entry's flag set.
+ * @param flag The removed affix's flag.
+ * @return {@code true} if the entry carries the flag and may stand affixed.
+ */
+ boolean supports(int[] flags, int flag) {
+ return contains(flags, flag) && !contains(flags, onlyInCompound)
+ && !contains(flags, forbiddenWord);
+ }
+
/**
* Checks whether some homonym supports a cross-product analysis. The root can contain
* both affix flags, or one root flag can select an affix with continuation flags that
@@ -940,19 +2210,63 @@ public final class HunspellDictionary {
*/
boolean supportsCrossProduct(List<int[]> flagSets, Affix prefix, Affix suffix) {
for (final int[] flags : flagSets) {
- final boolean rootHasPrefix = contains(flags, prefix.flag());
- final boolean rootHasSuffix = contains(flags, suffix.flag());
- final boolean licensesBoth = (rootHasPrefix
- && (rootHasSuffix || prefix.allowsContinuation(suffix.flag())))
- || (rootHasSuffix && suffix.allowsContinuation(prefix.flag()));
- if (licensesBoth && !contains(flags, onlyInCompound)
- && !contains(flags, forbiddenWord)) {
+ if (supportsCrossProduct(flags, prefix, suffix)) {
return true;
}
}
return false;
}
+ /**
+ * Checks whether one homonym supports a cross-product analysis, under the conditions
+ * of {@link #supportsCrossProduct(List, Affix, Affix)}.
+ *
+ * @param flags One entry's flag set.
+ * @param prefix The removed prefix.
+ * @param suffix The removed suffix.
+ * @return {@code true} if the entry licenses both affixes and may stand affixed.
+ */
+ boolean supportsCrossProduct(int[] flags, Affix prefix, Affix suffix) {
+ return licensesCrossProduct(flags, prefix, suffix) && !contains(flags, onlyInCompound)
+ && !contains(flags, forbiddenWord);
+ }
+
+ /**
+ * Checks cross-product flags within a single entry.
+ *
+ * @param flags The selected entry flags.
+ * @param prefix The prefix rule.
+ * @param suffix The suffix rule.
+ * @return Whether the entry and continuation classes permit both rules.
+ */
+ boolean licensesCrossProduct(int[] flags, Affix prefix, Affix suffix) {
+ return (contains(flags, prefix.flag())
+ && (contains(flags, suffix.flag()) || prefix.allowsContinuation(suffix.flag())))
+ || (contains(flags, suffix.flag()) && suffix.allowsContinuation(prefix.flag()));
+ }
+
+ /**
+ * Checks a compound cross-product reading.
+ *
+ * @param flags The selected entry flags.
+ * @param position The component position.
+ * @param prefix The prefix rule.
+ * @param suffix The suffix rule.
+ * @param additional Additional continuation-linked rules.
+ * @return Whether the entry and rules permit this component.
+ */
+ boolean supportsCompoundCrossProduct(int[] flags, CompoundPosition position,
+ Affix prefix, Affix suffix, Affix... additional) {
+ boolean positionAllowed = contains(flags, compoundFlag) || contains(flags, positionalFlag(position))
+ || affixAdmits(prefix, position) || affixAdmits(suffix, position);
+ for (Affix affix : additional) {
+ positionAllowed |= affixAdmits(affix, position);
+ }
+ return licensesCrossProduct(flags, prefix, suffix) && !contains(flags, forbiddenWord)
+ && !forbiddenAtCompoundPosition(flags, position)
+ && positionAllowed;
+ }
+
/**
* Checks whether a form made with this affix alone is still a virtual stem: the
* affix carries the {@code NEEDAFFIX} flag among its continuation classes, so a
@@ -999,11 +2313,14 @@ public final class HunspellDictionary {
* @throws IOException Thrown if the declared encoding name is not supported.
*/
private static Charset declaredCharset(byte[] affixBytes) throws IOException {
- final String ascii = new String(affixBytes, StandardCharsets.US_ASCII);
+ final int start = byteOrderMarkLength(affixBytes);
+ final String ascii = new String(affixBytes, start, affixBytes.length - start,
+ StandardCharsets.US_ASCII);
for (final String line : splitLines(ascii)) {
final String trimmed = trim(line);
if (trimmed.startsWith(SET_PREFIX) || trimmed.startsWith(SET_TAB_PREFIX)) {
- final String name = trim(trimmed.substring(SET_PREFIX.length()));
+ // the encoding is the first field after the directive; later fields are ignored
+ final String name = split(trimmed.substring(SET_PREFIX.length()))[0];
try {
return Charset.forName(name);
} catch (IllegalCharsetNameException | UnsupportedCharsetException e) {
@@ -1050,6 +2367,29 @@ public final class HunspellDictionary {
private boolean checkCompoundCase;
private boolean checkCompoundTriple;
private boolean fullStrip;
+ private String ignoredCharacters = "";
+ private HunspellConversion inputConversion = HunspellConversion.NONE;
+ private HunspellConversion outputConversion = HunspellConversion.NONE;
+ private final List<List<String>> morphologyAliases = new ArrayList<>();
+ private final Map<int[], List<String>> entryMorphology = new HashMap<>();
+ private boolean complexPrefixes;
+ private int keepCase;
+ private int warningFlag;
+ private boolean forbidWarn;
+ private boolean checkSharps;
+ private String language = "";
+ private boolean syllableNumber;
+ private final List<HunspellCompoundRule> compoundRules = new ArrayList<>();
+ private final List<CompoundPattern> compoundPatterns = new ArrayList<>();
+ private int compoundRoot;
+ private int forceUpperCase;
+ private boolean compoundMoreSuffixes;
+ private boolean simplifiedTriple;
+ private boolean checkCompoundRep;
+ private HunspellConversion replacements = HunspellConversion.NONE;
+ private int maxCompoundSyllables;
+ private String compoundVowels = "";
+ private List<WordBreak> wordBreaks = DEFAULT_WORD_BREAKS;
}
/**
@@ -1071,102 +2411,168 @@ public final class HunspellDictionary {
}
result.flagMode = readFlagMode(fieldsByLine);
result.flagAliases.addAll(readFlagAliases(fieldsByLine, result.flagMode));
+ result.inputConversion = HunspellConversion.parse(fieldsByLine, INPUT_CONVERSION_TAG);
+ result.outputConversion = HunspellConversion.parse(fieldsByLine, OUTPUT_CONVERSION_TAG);
+ result.morphologyAliases.addAll(readMorphologyAliases(fieldsByLine));
+ result.compoundRules.addAll(readCompoundRules(fieldsByLine, result.flagMode));
+ result.compoundPatterns.addAll(readCompoundPatterns(fieldsByLine, result.flagMode));
+ result.replacements = HunspellConversion.parse(fieldsByLine, REPLACEMENT_TAG);
+ result.wordBreaks = readWordBreaks(fieldsByLine);
+ for (int line = 0; line < fieldsByLine.length; line++) {
+ final String[] fields = fieldsByLine[line];
+ if (fields.length > 0 && IGNORE_TAG.equals(fields[0])) {
+ if (fields.length != 2) {
+ throw new IOException("invalid IGNORE at line " + (line + 1));
+ }
+ result.ignoredCharacters = fields[1];
+ }
+ }
int i = 0;
while (i < lines.length) {
final String[] fields = fieldsByLine[i];
if (fields.length == 0 || fields[0].startsWith(COMMENT_PREFIX)) {
i++;
- continue;
- }
- switch (fields[0]) {
- case FLAG_TAG:
- // The file-wide declaration was parsed before any rule fields.
- i++;
- break;
- case "COMPOUNDFLAG":
- case "COMPOUNDBEGIN":
- case "COMPOUNDMIDDLE":
- case "COMPOUNDEND":
- case "COMPOUNDPERMITFLAG":
- case "COMPOUNDFORBIDFLAG":
- case "NEEDAFFIX":
- case "PSEUDOROOT":
- case "ONLYINCOMPOUND":
- case "FORBIDDENWORD":
- case "CIRCUMFIX":
- if (fields.length < 2) {
- throw new IOException(fields[0] + " line without a flag at line " + (i + 1));
- }
- final int declared = parseFlag(fields[1], result.flagMode, i + 1);
- switch (fields[0]) {
- case "COMPOUNDFLAG" -> result.compoundFlag = declared;
- case "COMPOUNDBEGIN" -> result.compoundBegin = declared;
- case "COMPOUNDMIDDLE" -> result.compoundMiddle = declared;
- case "COMPOUNDEND" -> result.compoundEnd = declared;
- case "COMPOUNDPERMITFLAG" -> result.compoundPermit = declared;
- case "COMPOUNDFORBIDFLAG" -> result.compoundForbid = declared;
- // PSEUDOROOT is the directive's name before hunspell renamed it
- case "NEEDAFFIX", "PSEUDOROOT" -> result.needAffix = declared;
- case "ONLYINCOMPOUND" -> result.onlyInCompound = declared;
- case "CIRCUMFIX" -> result.circumfix = declared;
- case "FORBIDDENWORD" -> result.forbiddenWord = declared;
- default -> throw new IOException(
- "unhandled flag directive " + fields[0] + " at line " + (i + 1));
- }
- i++;
- break;
- case "COMPOUNDMIN":
- final int compoundMin = parseValue(fields, i + 1);
- if (compoundMin < 0) {
- throw new IOException("negative COMPOUNDMIN at line " + (i + 1));
- }
- if (compoundMin > MAX_COMPOUND_MIN) {
- throw new IOException("COMPOUNDMIN exceeds " + MAX_COMPOUND_MIN
- + " at line " + (i + 1));
- }
- result.compoundMin = Math.max(1, compoundMin);
- i++;
- break;
- case "COMPOUNDWORDMAX":
- final int compoundWordMax = parseValue(fields, i + 1);
- if (compoundWordMax < 0) {
- throw new IOException("negative COMPOUNDWORDMAX at line " + (i + 1));
- }
- result.compoundWordMax = compoundWordMax;
- i++;
- break;
- case "CHECKCOMPOUNDDUP":
- result.checkCompoundDup = true;
- i++;
- break;
- case "CHECKCOMPOUNDCASE":
- result.checkCompoundCase = true;
- i++;
- break;
- case "CHECKCOMPOUNDTRIPLE":
- result.checkCompoundTriple = true;
- i++;
- break;
- case "FULLSTRIP":
- result.fullStrip = true;
- i++;
- break;
- case ALIAS_TAG:
- // The file-wide table was parsed before continuation and entry flags.
- i++;
- break;
- case PREFIX_TAG:
- case SUFFIX_TAG:
- i = parseAffixBlock(fieldsByLine, i, fields, result);
- break;
- default:
- i++;
- break;
+ } else if (PREFIX_TAG.equals(fields[0]) || SUFFIX_TAG.equals(fields[0])) {
+ i = parseAffixBlock(fieldsByLine, i, fields, result);
+ } else {
+ // FLAG, AF, and the tables were read before this pass; every other directive
+ // is a flag declaration, a compound setting, a general setting, or ignored
+ final int line = i + 1;
+ if (!readFlagDeclaration(fields, line, result)
+ && !readCompoundSetting(fields, line, result)) {
+ readGeneralSetting(fields, line, result);
+ }
+ i++;
}
}
return result;
}
+ /**
+ * Reads a directive that declares one flag, such as {@code COMPOUNDFLAG} or
+ * {@code NEEDAFFIX}.
+ *
+ * @param fields The line's fields.
+ * @param line The one-based source line.
+ * @param result The affix file under construction.
+ * @return {@code true} if the line was a flag declaration.
+ * @throws IOException Thrown if the declaration has no flag or the flag is malformed.
+ */
+ private static boolean readFlagDeclaration(String[] fields, int line, AffixFile result)
+ throws IOException {
+ if (!isSingleFlagDirective(fields[0])) {
+ return false;
+ }
+ if (fields.length < 2) {
+ throw new IOException(fields[0] + " line without a flag at line " + line);
+ }
+ final int declared = parseFlag(fields[1], result.flagMode, line);
+ switch (fields[0]) {
+ case "COMPOUNDFLAG" -> result.compoundFlag = declared;
+ case "COMPOUNDBEGIN" -> result.compoundBegin = declared;
+ case "COMPOUNDMIDDLE" -> result.compoundMiddle = declared;
+ case "COMPOUNDEND" -> result.compoundEnd = declared;
+ case "COMPOUNDPERMITFLAG" -> result.compoundPermit = declared;
+ case "COMPOUNDFORBIDFLAG" -> result.compoundForbid = declared;
+ // PSEUDOROOT is the directive's name before hunspell renamed it
+ case "NEEDAFFIX", "PSEUDOROOT" -> result.needAffix = declared;
+ case "ONLYINCOMPOUND" -> result.onlyInCompound = declared;
+ case "CIRCUMFIX" -> result.circumfix = declared;
+ case "FORBIDDENWORD" -> result.forbiddenWord = declared;
+ case KEEP_CASE_TAG -> result.keepCase = declared;
+ case WARNING_TAG -> result.warningFlag = declared;
+ case COMPOUND_ROOT_TAG -> result.compoundRoot = declared;
+ case FORCE_UPPER_CASE_TAG -> result.forceUpperCase = declared;
+ case LEMMA_PRESENT_TAG -> { }
+ default -> throw new IOException(
+ "unhandled flag directive " + fields[0] + " at line " + line);
+ }
+ return true;
+ }
+
+ /**
+ * Reads a compound setting: the size limits, the boundary checks, the syllable
+ * rules, and the compound switches.
+ *
+ * @param fields The line's fields.
+ * @param line The one-based source line.
+ * @param result The affix file under construction.
+ * @return {@code true} if the line was a compound setting.
+ * @throws IOException Thrown if a value is missing, negative, or out of range.
+ */
+ private static boolean readCompoundSetting(String[] fields, int line, AffixFile result)
+ throws IOException {
+ switch (fields[0]) {
+ case "COMPOUNDMIN" -> {
+ final int compoundMin = parseValue(fields, line);
+ if (compoundMin < 0) {
+ throw new IOException("negative COMPOUNDMIN at line " + line);
+ }
+ if (compoundMin > MAX_COMPOUND_MIN) {
+ throw new IOException("COMPOUNDMIN exceeds " + MAX_COMPOUND_MIN + " at line " + line);
+ }
+ result.compoundMin = Math.max(1, compoundMin);
+ }
+ case "COMPOUNDWORDMAX" -> {
+ final int compoundWordMax = parseValue(fields, line);
+ if (compoundWordMax < 0) {
+ throw new IOException("negative COMPOUNDWORDMAX at line " + line);
+ }
+ result.compoundWordMax = compoundWordMax;
+ }
+ case COMPOUND_SYLLABLE_TAG -> {
+ result.maxCompoundSyllables = parseValue(fields, line);
+ if (fields.length != 3 || result.maxCompoundSyllables < 0) {
+ throw new IOException("invalid COMPOUNDSYLLABLE at line " + line);
+ }
+ result.compoundVowels = fields[2];
+ }
+ case SYLLABLE_NUMBER_TAG -> {
+ if (fields.length != 2) {
+ throw new IOException("invalid SYLLABLENUM at line " + line);
+ }
+ result.syllableNumber = true;
+ }
+ case "CHECKCOMPOUNDDUP" -> result.checkCompoundDup = true;
+ case "CHECKCOMPOUNDCASE" -> result.checkCompoundCase = true;
+ case "CHECKCOMPOUNDTRIPLE" -> result.checkCompoundTriple = true;
+ case CHECK_COMPOUND_REP_TAG -> result.checkCompoundRep = true;
+ case COMPOUND_MORE_SUFFIXES_TAG -> result.compoundMoreSuffixes = true;
+ case SIMPLIFIED_TRIPLE_TAG -> result.simplifiedTriple = true;
+ default -> {
+ return false;
+ }
+ }
+ return true;
+ }
+
+ /**
+ * Reads a general setting outside compounding: the affix switches, the sharp-s and
+ * warning rules, and the language. Directives read before this pass and ignored
+ * directives are left alone.
+ *
+ * @param fields The line's fields.
+ * @param line The one-based source line.
+ * @param result The affix file under construction.
+ * @throws IOException Thrown if the language declaration is malformed.
+ */
+ private static void readGeneralSetting(String[] fields, int line, AffixFile result)
+ throws IOException {
+ switch (fields[0]) {
+ case "FULLSTRIP" -> result.fullStrip = true;
+ case COMPLEX_PREFIXES_TAG -> result.complexPrefixes = true;
+ case FORBID_WARNING_TAG -> result.forbidWarn = true;
+ case CHECK_SHARPS_TAG -> result.checkSharps = true;
+ case LANGUAGE_TAG -> {
+ if (fields.length != 2) {
+ throw new IOException("invalid LANG at line " + line);
+ }
+ result.language = fields[1];
+ }
+ default -> { }
+ }
+ }
+
/**
* Finds the file-wide flag mode before parsing directives that contain flags.
*
@@ -1298,13 +2704,14 @@ public final class HunspellDictionary {
throw new IOException("affix block truncated at line " + (line + 1));
}
final String[] fields = fieldsByLine[line];
- if (fields.length < 5 || !fields[0].equals(header[0])) {
+ if (fields.length < 4 || !fields[0].equals(header[0])) {
throw new IOException("malformed affix rule at line " + (line + 1));
}
if (parseFlag(fields[1], result.flagMode, line + 1) != flag) {
throw new IOException("affix rule flag does not match header at line " + (line + 1));
}
- final String strip = NO_MATERIAL.equals(fields[2]) ? "" : fields[2];
+ final String strip = NO_MATERIAL.equals(fields[2]) ? ""
+ : removeIgnored(fields[2], result.ignoredCharacters);
String affixText = fields[3];
int[] continuation = new int[0];
final int slash = affixText.indexOf('/');
@@ -1316,8 +2723,14 @@ public final class HunspellDictionary {
if (NO_MATERIAL.equals(affixText)) {
affixText = "";
}
+ affixText = removeIgnored(affixText, result.ignoredCharacters);
+ final boolean hasCondition = fields.length > 4;
+ final List<String> morphology = parseMorphology(
+ Arrays.copyOfRange(fields, hasCondition ? 5 : 4, fields.length),
+ result.morphologyAliases, line + 1);
final Affix affix = new Affix(flag, crossProduct, strip, affixText,
- AffixCondition.parse(fields[4], suffix, line + 1), continuation);
+ AffixCondition.parse(hasCondition ? fields[4] : ".", suffix, line + 1), continuation, suffix,
+ morphology, fields[1]);
if (suffix) {
result.suffixes.add(affix);
} else {
@@ -1330,22 +2743,19 @@ public final class HunspellDictionary {
/**
* Parses the word list: an optional leading entry count, then one entry per line
* consisting of the word, an optional {@code /flags} run, and optional trailing
- * morphological fields, which are ignored. The morphological fields are cut off
+ * morphological fields. The morphological fields are separated
* first, because the flag separator is only meaningful in what precedes them; a word
* may itself contain spaces. A slash escaped as {@code \/} belongs to the word itself
* and is unescaped in the stored key.
*
* @param content The decoded word-list content.
- * @param flagMode The flag encoding declared by the affix file.
- * @param flagAliases The affix file's {@code AF} alias table, possibly empty. When
- * it is not empty, a purely numeric flag field is a 1-based
- * reference into it rather than a flag run of its own.
+ * @param affix The affix settings and destination for entry morphology.
* @return The words mapped to the flag sets of their entries. Never {@code null}.
* @throws IOException Thrown if a flag run is malformed or an alias reference is
* out of range.
*/
private static Map<String, List<int[]>> parseWordList(String content,
- FlagMode flagMode, List<int[]> flagAliases) throws IOException {
+ AffixFile affix) throws IOException {
final String[] lines = splitLines(withoutByteOrderMark(content));
final Map<String, List<int[]>> entries = new HashMap<>();
int start = 0;
@@ -1357,7 +2767,13 @@ public final class HunspellDictionary {
if (line.isEmpty()) {
continue;
}
- final int morphology = morphologyIndex(line);
+ int morphology = morphologyIndex(line);
+ if (morphology < 0 && !affix.morphologyAliases.isEmpty()) {
+ final int separator = line.lastIndexOf(' ');
+ if (separator > 0 && isCount(line.substring(separator + 1))) {
+ morphology = separator;
+ }
+ }
final String entry = morphology < 0 ? line : trim(line.substring(0, morphology));
String word = entry;
int[] flags = new int[0];
@@ -1374,14 +2790,140 @@ public final class HunspellDictionary {
break;
}
}
- flags = parseAliasedFlags(flagRun, flagMode, flagAliases, i + 1);
+ flags = parseAliasedFlags(flagRun, affix.flagMode, affix.flagAliases, i + 1).clone();
}
- entries.computeIfAbsent(word.replace("\\/", "/"), key -> new ArrayList<>(1))
+ if (morphology >= 0) {
+ affix.entryMorphology.put(flags, parseMorphology(split(line.substring(morphology)),
+ affix.morphologyAliases, i + 1));
+ }
+ entries.computeIfAbsent(removeIgnored(word.replace("\\/", "/"), affix.ignoredCharacters),
+ key -> new ArrayList<>(1))
.add(flags);
}
return entries;
}
+ /**
+ * Parses the complete AM alias table before entries and affixes.
+ *
+ * @param lines The affix fields indexed by source line.
+ * @return The aliases in reference order.
+ * @throws IOException Thrown if a count or table entry is malformed.
+ */
+ private static List<List<String>> readMorphologyAliases(String[][] lines) throws IOException {
+ final List<List<String>> aliases = new ArrayList<>();
+ final List<HunspellAffixTable.Entry> entries =
+ HunspellAffixTable.read(lines, MORPHOLOGY_ALIAS_TAG, 2, Integer.MAX_VALUE);
+ if (entries != null) {
+ for (HunspellAffixTable.Entry entry : entries) {
+ final String[] fields = entry.fields();
+ aliases.add(List.of(Arrays.copyOfRange(fields, 1, fields.length)));
+ }
+ }
+ return aliases;
+ }
+
+ /**
+ * Parses and validates the COMPOUNDRULE table.
+ *
+ * @param lines The affix fields indexed by line.
+ * @param mode The flag encoding.
+ * @return The parsed patterns.
+ * @throws IOException Thrown if a count, pattern, or flag is malformed.
+ */
+ private static List<HunspellCompoundRule> readCompoundRules(String[][] lines, FlagMode mode)
+ throws IOException {
+ final List<HunspellCompoundRule> rules = new ArrayList<>();
+ final List<HunspellAffixTable.Entry> entries = HunspellAffixTable.read(lines, COMPOUND_RULE_TAG, 2, 2);
+ if (entries != null) {
+ for (HunspellAffixTable.Entry entry : entries) {
+ rules.add(HunspellCompoundRule.parse(entry.fields()[1], entry.line(),
+ (text, line) -> parseFlag(text, mode, line)));
+ }
+ }
+ return rules;
+ }
+
+ /**
+ * Parses CHECKCOMPOUNDPATTERN restrictions and replacements.
+ *
+ * @param lines The affix fields.
+ * @param mode The flag encoding.
+ * @return The boundary patterns.
+ * @throws IOException Thrown if a declaration is malformed.
+ */
+ private static List<CompoundPattern> readCompoundPatterns(String[][] lines, FlagMode mode)
+ throws IOException {
+ final List<CompoundPattern> patterns = new ArrayList<>();
+ final List<HunspellAffixTable.Entry> entries =
+ HunspellAffixTable.read(lines, COMPOUND_PATTERN_TAG, 3, 4);
+ if (entries == null) {
+ return patterns;
+ }
+ for (HunspellAffixTable.Entry entry : entries) {
+ final String[] fields = entry.fields();
+ final int leftSlash = fields[1].indexOf('/');
+ final int rightSlash = fields[2].indexOf('/');
+ final String left = leftSlash < 0 ? fields[1] : fields[1].substring(0, leftSlash);
+ final String right = rightSlash < 0 ? fields[2] : fields[2].substring(0, rightSlash);
+ patterns.add(new CompoundPattern(NO_MATERIAL.equals(left) ? "" : left,
+ leftSlash < 0 ? 0 : parseFlag(fields[1].substring(leftSlash + 1), mode, entry.line()),
+ right, rightSlash < 0 ? 0 : parseFlag(fields[2].substring(rightSlash + 1), mode, entry.line()),
+ NO_MATERIAL.equals(left), fields.length == 4 ? fields[3] : null));
+ }
+ return patterns;
+ }
+
+ /**
+ * Parses BREAK declarations, using hyphen rules when no table is configured.
+ *
+ * @param lines The affix fields.
+ * @return The separators and anchors.
+ * @throws IOException Thrown if the table or a separator is malformed.
+ */
+ private static List<WordBreak> readWordBreaks(String[][] lines) throws IOException {
+ final List<HunspellAffixTable.Entry> entries = HunspellAffixTable.read(lines, BREAK_TAG, 2, 2);
+ if (entries == null) {
+ return DEFAULT_WORD_BREAKS;
+ }
+ final List<WordBreak> result = new ArrayList<>();
+ for (HunspellAffixTable.Entry entry : entries) {
+ result.add(WordBreak.parse(entry.fields()[1], entry.line()));
+ }
+ return result;
+ }
+
+ /**
+ * Expands an optional morphology alias.
+ *
+ * @param fields The morphological fields or alias reference.
+ * @param aliases The AM table.
+ * @param line The source line.
+ * @return The expanded immutable fields.
+ * @throws IOException Thrown if an alias reference is invalid.
+ */
+ private static List<String> parseMorphology(String[] fields, List<List<String>> aliases,
+ int line) throws IOException {
+ if (fields.length == 0) {
+ return List.of();
+ }
+ if (!aliases.isEmpty()) {
+ try {
+ if (fields.length != 1) {
+ throw new NumberFormatException();
+ }
+ final int index = Integer.parseInt(fields[0]);
+ if (index < 1 || index > aliases.size()) {
+ throw new NumberFormatException();
+ }
+ return aliases.get(index - 1);
+ } catch (NumberFormatException e) {
+ throw new IOException("invalid AM alias at line " + line, e);
+ }
+ }
+ return List.of(fields);
+ }
+
/**
* Removes a Unicode byte-order mark decoded at the start of a file.
*
@@ -1465,9 +3007,9 @@ public final class HunspellDictionary {
* a tabulator, the older separator, or written as a two-letter tag followed by
* {@code :} and preceded by a separator, such as {@code po:verb}. A separator that
* is not followed by such a tag belongs to the word, because a word-list entry may
- * name several words. The separators are the space and the tabulator, exactly the
- * two characters the reference implementation's {@code hashmgr.cxx} splits on; they
- * are format delimiters of the word-list grammar, not a whitespace judgment, so
+ * name several words. The separators are the space and the tabulator, the two
+ * characters the word-list format splits on; they are format delimiters of the
+ * word-list grammar, not a whitespace judgment, so
* wider whitespace such as a no-break space stays part of the word by design.
*
* @param line The trimmed word-list line to scan.
@@ -1496,8 +3038,7 @@ public final class HunspellDictionary {
/**
* Checks one character against the word-list format's field separators, space and
- * tabulator, the exact set the reference implementation splits morphological fields
- * on.
+ * tabulator.
*
* @param c The character to test.
* @return {@code true} if {@code c} separates fields in the word-list format.
@@ -1611,7 +3152,9 @@ public final class HunspellDictionary {
}
/**
- * Splits text into lines with a single character scan, tolerating CRLF endings.
+ * Splits text into lines with a single character scan. A line ends at a line feed, a
+ * carriage return, or a carriage return followed by a line feed, as in the directive
+ * masking, so line numbers agree.
*
* @param content The text to split.
* @return The lines without their terminators. Never {@code null}.
@@ -1620,12 +3163,12 @@ public final class HunspellDictionary {
final List<String> lines = new ArrayList<>();
int start = 0;
for (int i = 0; i <= content.length(); i++) {
- if (i == content.length() || content.charAt(i) == '\n') {
- int end = i;
- if (end > start && content.charAt(end - 1) == '\r') {
- end--;
+ if (i == content.length() || content.charAt(i) == '\n' || content.charAt(i) == '\r') {
+ lines.add(content.substring(start, i));
+ if (i + 1 < content.length() && content.charAt(i) == '\r'
+ && content.charAt(i + 1) == '\n') {
+ i++;
}
- lines.add(content.substring(start, end));
start = i + 1;
}
}
diff --git a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/hunspell/HunspellStemmer.java b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/hunspell/HunspellStemmer.java
index c5699fa14..b9418fe15 100644
--- a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/hunspell/HunspellStemmer.java
+++ b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/hunspell/HunspellStemmer.java
@@ -18,33 +18,38 @@
package opennlp.tools.stemmer.hunspell;
import java.util.ArrayList;
+import java.util.HashMap;
import java.util.LinkedHashSet;
import java.util.List;
+import java.util.Map;
import java.util.Set;
import opennlp.tools.commons.ThreadSafe;
import opennlp.tools.stemmer.Stemmer;
import opennlp.tools.stemmer.hunspell.HunspellDictionary.Affix;
+import opennlp.tools.stemmer.hunspell.HunspellDictionary.CompoundPattern;
import opennlp.tools.stemmer.hunspell.HunspellDictionary.CompoundPosition;
-import opennlp.tools.util.StringUtil;
/**
* A dictionary-backed {@link Stemmer} over a {@link HunspellDictionary}: a surface form
* is reduced to the dictionary words it can be derived from by removing one suffix, one
- * prefix, a cross-product combination of both, or an additional suffix licensed by a
- * continuation class.
+ * prefix, a cross-product combination of both, or an additional affix licensed by a
+ * continuation class. Compound and break-separated forms return the stems of their
+ * recognized parts.
*
* <p>{@link #stem(CharSequence)} returns the first analysis, preferring the word's own
* dictionary entry; {@link #stemAll(CharSequence)} returns every distinct analysis. A
* word with no analysis is returned unchanged, so the stemmer degrades to identity on
- * unknown vocabulary. A form containing uppercase characters is also analyzed in its
- * lowercase variant, so sentence-initial capitalization does not hide an entry.
+ * unknown input. Title-case and all-uppercase forms also use permitted case
+ * variants, subject to the dictionary's case restrictions.
* Entries the dictionary marks as virtual stems ({@code NEEDAFFIX}), compound-only
- * parts ({@code ONLYINCOMPOUND}), or forbidden words ({@code FORBIDDENWORD}) never
- * count as standalone analyses, matching how hunspell reads those flags.</p>
+ * parts ({@code ONLYINCOMPOUND}), or forbidden words ({@code FORBIDDENWORD}) do not
+ * count as standalone analyses.</p>
*
- * <p>Compound part search is capped at {@value #PART_CHECK_BUDGET} part-licensing
- * attempts per input word; beyond that budget further compound analyses are skipped.
+ * <p>Compound part search stops after {@value #PART_CHECK_BUDGET} part-licensing
+ * attempts. The budget applies separately to each capitalization form of the input
+ * and to each word-break search. Once it is used up, further compound analyses are
+ * skipped.
* The {@link Stemmer} interface leaves thread safety to the implementation. This
* implementation reads only the immutable dictionary state, so a single instance is
* safe to share between threads.</p>
@@ -55,14 +60,19 @@ import opennlp.tools.util.StringUtil;
public final class HunspellStemmer implements Stemmer {
/**
- * The most part-licensing attempts one decomposition search may spend. Compounding
- * searches every split of every tail, which on adversarial input with a
- * one-character minimum part length grows without useful bound; the budget stops
- * the search there, missing analyses rather than stalling, in line with the
- * engine's fail-closed posture.
+ * Maximum candidate checks per compound search or recursive word-break search.
*/
private static final int PART_CHECK_BUDGET = 2048;
+ /** Maximum parts in one compound analysis. */
+ private static final int MAX_COMPOUND_PARTS = 64;
+
+ /** Maximum case variants considered for CHECKSHARPS. */
+ private static final int MAX_CASE_VARIANTS = 64;
+
+ /** Maximum distinct morphological readings returned for one input. */
+ private static final int MAX_ANALYSES = 2048;
+
private final HunspellDictionary dictionary;
/**
@@ -101,37 +111,443 @@ public final class HunspellStemmer implements Stemmer {
throw new IllegalArgumentException("word must not be null");
}
final String surface = word.toString();
- if (surface.isEmpty()) {
+ final List<String> analyses = findWord(dictionary.inputForm(surface), false);
+ if (analyses.isEmpty()) {
+ return List.of(surface);
+ }
+ final Set<String> converted = new LinkedHashSet<>();
+ for (String analysis : analyses) {
+ converted.add(dictionary.outputForm(analysis));
+ }
+ return List.copyOf(converted);
+ }
+
+ /**
+ * Returns morphological analyses as space-separated Hunspell fields. Each result
+ * describes a complete accepted reading. Entries without an explicit
+ * {@code st:} field use the dictionary entry as their stem. Compound components
+ * begin with {@code pa:}; entry and affix fields follow in application order.
+ * Results preserve dictionary field text without output conversion.
+ *
+ * @param word The input to analyze. Must not be {@code null}.
+ * @return An immutable list of distinct analyses, or an empty list for unknown input.
+ * At most {@value #MAX_ANALYSES} analyses are returned.
+ * @throws IllegalArgumentException Thrown if {@code word} is {@code null}.
+ */
+ List<String> analyze(CharSequence word) {
+ if (word == null) {
+ throw new IllegalArgumentException("word must not be null");
+ }
+ return findWord(dictionary.inputForm(word.toString()), true);
+ }
+
+ /**
+ * Finds the readings of a complete input. Trailing periods are removed first so
+ * that abbreviations match their entries; when the shortened form has no reading,
+ * one period is restored for entries listed with it.
+ *
+ * @param input The input after conversion.
+ * @param morphological Whether results contain morphological fields.
+ * @return Recognized stems or analyses, or an empty list.
+ */
+ private List<String> findWord(String input, boolean morphological) {
+ int end = input.length();
+ while (end > 0 && input.charAt(end - 1) == '.') {
+ end--;
+ }
+ if (end == 0) {
// a zero-length word has no morphology; without this guard a strip-only rule
// could restore its strip string onto nothing and answer a non-empty stem
- return List.of(surface);
+ return List.of();
}
- final Set<String> analyses = new LinkedHashSet<>();
- for (final String variant : variants(surface)) {
- analyze(variant, analyses);
+ final List<String> analyses = findStems(input.substring(0, end), 0,
+ new int[] {PART_CHECK_BUDGET}, new HashMap<>(), morphological);
+ if (analyses.isEmpty() && end < input.length()) {
+ return findStems(input.substring(0, end + 1), 0,
+ new int[] {PART_CHECK_BUDGET}, new HashMap<>(), morphological);
}
- if (analyses.isEmpty() && dictionary.compoundsDeclared()) {
- for (final String variant : variants(surface)) {
- decompose(variant, surface, analyses);
+ return analyses;
+ }
+
+ /** Accumulates stems or complete morphology readings for one request. */
+ private final class Results {
+ private final boolean morphological;
+ private final Set<String> values = new LinkedHashSet<>();
+ /**
+ * Whether a direct or affixed reading reached a forbidden entry, which blocks the
+ * compound and break readings of the same input.
+ */
+ private boolean forbidden;
+ /**
+ * Whether compound decomposition follows the Hungarian moving rule for the part of
+ * a word before a hyphen.
+ */
+ private boolean hyphenatedFirstPart;
+
+ /**
+ * Selects the output representation.
+ *
+ * @param morphological Whether results contain fields instead of stems.
+ */
+ private Results(boolean morphological) {
+ this.morphological = morphological;
+ }
+
+ /** {@return whether a reading has not been found} */
+ private boolean isEmpty() {
+ return values.isEmpty();
+ }
+
+ /**
+ * Adds distinct results up to the output limit.
+ *
+ * @param additions The accepted results.
+ */
+ private void addAll(List<String> additions) {
+ for (String value : additions) {
+ if (values.size() >= MAX_ANALYSES) {
+ return;
+ }
+ values.add(value);
}
}
- if (analyses.isEmpty()) {
- return List.of(surface);
+
+ /**
+ * Adds a complete compound analysis, preserving homonym alternatives.
+ *
+ * @param parts The selected component entries and affixes.
+ */
+ private void addCompound(List<CompoundPart> parts) {
+ if (!morphological) {
+ for (CompoundPart part : parts) {
+ addAll(part.stems());
+ }
+ return;
+ }
+ List<String> accumulated = List.of("");
+ for (CompoundPart part : parts) {
+ final List<String> next = new ArrayList<>();
+ for (String fields : dictionary.morphologicalAnalyses(part.root(), part.flags(),
+ part == parts.get(parts.size() - 1), part.affixes().toArray(Affix[]::new))) {
+ for (String prior : accumulated) {
+ if (next.size() >= MAX_ANALYSES) {
+ break;
+ }
+ next.add(new StringBuilder(prior).append(prior.isEmpty() ? "" : " ")
+ .append(HunspellDictionary.COMPOUND_PART_FIELD).append(part.surface())
+ .append(fields.isEmpty() ? "" : " ").append(fields).toString());
+ }
+ }
+ accumulated = next;
+ }
+ addAll(accumulated);
+ }
+
+ /**
+ * Combines accepted readings before and after a word break.
+ *
+ * @param leftText The opening text.
+ * @param left The opening readings.
+ * @param rightText The closing text.
+ * @param right The closing readings.
+ */
+ private void addBroken(String leftText, List<String> left, String rightText, List<String> right) {
+ if (!morphological) {
+ addAll(left);
+ addAll(right);
+ return;
+ }
+ for (String first : left) {
+ for (String last : right) {
+ if (values.size() >= MAX_ANALYSES) {
+ return;
+ }
+ values.add(new StringBuilder()
+ .append(first.startsWith(HunspellDictionary.COMPOUND_PART_FIELD) ? ""
+ : HunspellDictionary.COMPOUND_PART_FIELD + leftText + " ").append(first)
+ .append(' ').append(last.startsWith(HunspellDictionary.COMPOUND_PART_FIELD) ? ""
+ : HunspellDictionary.COMPOUND_PART_FIELD + rightText + " ")
+ .append(last).toString());
+ }
+ }
}
- return List.copyOf(analyses);
+ }
+
+ /**
+ * Finds recognized stems without returning identity for unrecognized pieces.
+ *
+ * @param input The normalized input or a break-separated component.
+ * @param depth The current break depth.
+ * @param budget The remaining break attempts.
+ * @param cache Results for completed pieces of this input.
+ * @param morphological Whether results contain morphological fields.
+ * @return Recognized stems, or an empty list.
+ */
+ private List<String> findStems(String input, int depth, int[] budget,
+ Map<String, List<String>> cache, boolean morphological) {
+ if (input.isEmpty() || depth >= MAX_COMPOUND_PARTS || budget[0] <= 0) {
+ return List.of();
+ }
+ final List<String> cached = cache.get(input);
+ if (cached != null) {
+ return cached;
+ }
+ final List<int[]> entries = dictionary.lookup(input);
+ if (entries != null && dictionary.firstForbidden(entries)) {
+ return List.of();
+ }
+ final Results analyses = new Results(morphological);
+ final boolean allCaps = HunspellDictionary.caseType(input) == HunspellDictionary.CaseType.ALLCAP;
+ final List<String> variants = variants(input);
+ for (final String variant : variants) {
+ analyze(variant, new Analysis(input, variant, analyses, allCaps));
+ }
+ // a forbidden direct or affixed reading forbids the spelling as a whole, so no
+ // compound or break reading is attempted
+ if (analyses.isEmpty() && !analyses.forbidden && dictionary.compoundsDeclared()) {
+ for (final String variant : variants) {
+ decompose(variant, input, analyses);
+ }
+ }
+ if (analyses.isEmpty() && !analyses.forbidden) {
+ for (final HunspellDictionary.WordBreak wordBreak : dictionary.wordBreaks()) {
+ if (budget[0] <= 0) {
+ break;
+ }
+ final boolean start = wordBreak.atStart();
+ final boolean end = wordBreak.atEnd();
+ final String separator = wordBreak.separator();
+ for (int at = input.indexOf(separator); at >= 0 && budget[0] > 0;
+ at = input.indexOf(separator, at + separator.length())) {
+ final int after = at + separator.length();
+ if ((start && at != 0) || (end && after != input.length())
+ || (!start && at == 0) || (!end && after == input.length())) {
+ continue;
+ }
+ budget[0]--;
+ if (start && !end) {
+ analyses.addAll(findStems(input.substring(after), depth + 1, budget, cache, morphological));
+ } else if (end && !start) {
+ analyses.addAll(findStems(input.substring(0, at), depth + 1, budget, cache, morphological));
+ } else if (!start) {
+ List<String> left = findStems(input.substring(0, at), depth + 1,
+ budget, cache, morphological);
+ if (left.isEmpty() && "-".equals(separator) && dictionary.hyphenMovingRule()) {
+ left = hyphenatedFirstPart(input.substring(0, at), morphological);
+ }
+ if (!left.isEmpty()) {
+ final List<String> right = findStems(input.substring(after), depth + 1,
+ budget, cache, morphological);
+ if (!right.isEmpty()) {
+ analyses.addBroken(input.substring(0, at), left, input.substring(after), right);
+ }
+ }
+ }
+ }
+ }
+ }
+ final List<String> result = List.copyOf(analyses.values);
+ cache.put(input, result);
+ return result;
+ }
+
+ /**
+ * Finds the readings of the part of a Hungarian word before a hyphen, which is
+ * accepted as a listed word ending in the hyphen or as a compound under the moving
+ * rule.
+ *
+ * @param text The part before the hyphen.
+ * @param morphological Whether results contain morphological fields.
+ * @return Recognized stems or analyses, or an empty list.
+ */
+ private List<String> hyphenatedFirstPart(String text, boolean morphological) {
+ final Results analyses = new Results(morphological);
+ final String hyphenated = text + "-";
+ final boolean allCaps = HunspellDictionary.caseType(text) == HunspellDictionary.CaseType.ALLCAP;
+ for (final String variant : variants(hyphenated)) {
+ analyze(variant, new Analysis(hyphenated, variant, analyses, allCaps));
+ }
+ if (analyses.isEmpty() && !analyses.forbidden && dictionary.compoundsDeclared()) {
+ analyses.hyphenatedFirstPart = true;
+ for (final String variant : variants(text)) {
+ decompose(variant, text, analyses);
+ }
+ }
+ return List.copyOf(analyses.values);
}
/**
* Collects the case variants to analyze: the surface form first, then its lowercase
- * form when the two differ. Ordering matters because the first analysis found wins
- * in {@link #stem(CharSequence)}.
+ * form when the two differ. A capitalized word with a further inner capital, such as a
+ * mixed-case word at the start of a sentence, is also tried with a lowercase initial.
+ * An all-uppercase word containing an apostrophe is also tried with the part after
+ * the apostrophe capitalized, for the elided articles of Catalan, French, and Italian.
+ * Ordering matters because the first analysis found wins in {@link #stem(CharSequence)}.
*
* @param surface The surface form.
* @return The variants in analysis order. Never {@code null} or empty.
*/
private List<String> variants(String surface) {
- final String lowered = StringUtil.toLowerCase(surface);
- return lowered.equals(surface) ? List.of(surface) : List.of(surface, lowered);
+ final Set<String> variants = new LinkedHashSet<>();
+ variants.add(surface);
+ boolean upper = false;
+ boolean lowerAfterFirst = true;
+ boolean allUpper = true;
+ boolean firstUpper = false;
+ int uppers = 0;
+ int letters = 0;
+ for (int i = 0; i < surface.length();) {
+ final int point = surface.codePointAt(i);
+ if (Character.isLowerCase(point)) {
+ allUpper = false;
+ letters++;
+ } else if (Character.isUpperCase(point) || Character.isTitleCase(point)) {
+ if (letters > 0) {
+ lowerAfterFirst = false;
+ }
+ firstUpper |= letters == 0;
+ upper = true;
+ uppers++;
+ letters++;
+ }
+ i += Character.charCount(point);
+ }
+ if (upper && (allUpper || lowerAfterFirst)) {
+ final String lowered = dictionary.lowerCase(surface);
+ final int apostrophe = lowered.indexOf('\'');
+ if (allUpper && apostrophe > 0 && apostrophe < lowered.length() - 1) {
+ final String elided = lowered.substring(0, apostrophe + 1)
+ + initialUpper(lowered.substring(apostrophe + 1));
+ variants.add(elided);
+ variants.add(initialUpper(elided));
+ }
+ variants.add(lowered);
+ if (allUpper) {
+ variants.add(initialUpper(lowered));
+ if (surface.startsWith(HunspellDictionary.DOTTED_CAPITAL_I)) {
+ // capitalizing an all-uppercase word keeps a dotted capital I, so an entry
+ // such as İnci is found outside the Turkic languages too
+ variants.add(HunspellDictionary.DOTTED_CAPITAL_I
+ + lowered.substring(Character.charCount(lowered.codePointAt(0))));
+ }
+ if (dictionary.checkSharps()) {
+ addSharpVariants(lowered, 0, variants);
+ }
+ }
+ } else if (firstUpper && uppers > 1 && !allUpper) {
+ variants.add(dictionary.lowerCaseInitial(surface));
+ }
+ return List.copyOf(variants);
+ }
+
+ /**
+ * Converts the initial code point to uppercase with the dictionary's case mapping.
+ *
+ * @param word The nonempty word.
+ * @return The capitalized form.
+ */
+ private String initialUpper(String word) {
+ return dictionary.upperCaseInitial(word);
+ }
+
+ /**
+ * Collects sharp-s alternatives with a fixed search limit.
+ *
+ * @param word The lowercase candidate.
+ * @param from The next character position to search.
+ * @param variants The resulting forms.
+ */
+ private void addSharpVariants(String word, int from, Set<String> variants) {
+ if (variants.size() >= MAX_CASE_VARIANTS) {
+ return;
+ }
+ final String doubleS = HunspellDictionary.DOUBLE_S;
+ for (int at = word.indexOf(doubleS, from); at >= 0; at = word.indexOf(doubleS, at + 1)) {
+ final String changed = word.substring(0, at) + HunspellDictionary.SHARP_S
+ + word.substring(at + doubleS.length());
+ variants.add(changed);
+ variants.add(initialUpper(changed));
+ addSharpVariants(changed, at + 1, variants);
+ if (variants.size() >= MAX_CASE_VARIANTS) {
+ return;
+ }
+ }
+ }
+
+ /** One case-variant analysis with request-local result storage. */
+ private final class Analysis {
+ private final String surface;
+ private final String variant;
+ private final Results stems;
+ /**
+ * Whether the input is all uppercase, in which case the hidden capitalized forms
+ * of mixed-case entries also match.
+ */
+ private final boolean allCaps;
+
+ /**
+ * Creates an analysis context.
+ *
+ * @param surface The input after conversion.
+ * @param variant The current case variant.
+ * @param stems The destination for recognized stems.
+ * @param allCaps Whether the input is all uppercase.
+ */
+ private Analysis(String surface, String variant, Results stems, boolean allCaps) {
+ this.surface = surface;
+ this.variant = variant;
+ this.stems = stems;
+ this.allCaps = allCaps;
+ }
+
+ /**
+ * Looks up a spelling, including hidden capitalized forms for all-uppercase input.
+ *
+ * @param word The spelling to look up.
+ * @return The flag sets, or {@code null} when absent.
+ */
+ private List<int[]> lookup(String word) {
+ return dictionary.lookup(word, allCaps);
+ }
+
+ /**
+ * Adds morphology-aware stems after case and warning checks.
+ *
+ * @param root The entry spelling.
+ * @param flags The selected flags.
+ * @param affixes The rules in application order.
+ */
+ private void add(String root, int[] flags, Affix... affixes) {
+ if (dictionary.acceptsCase(flags, surface, variant, affixes)) {
+ stems.addAll(stems.morphological
+ ? dictionary.morphologicalAnalyses(root, flags, false, affixes)
+ : dictionary.morphologicalStems(root, flags, affixes));
+ }
+ }
+
+ /**
+ * Records that an affix analysis reached a forbidden entry.
+ *
+ * @param flagSets The stem's flag sets.
+ * @param flag The removed affix's flag.
+ */
+ private void noteForbidden(List<int[]> flagSets, int flag) {
+ if (dictionary.forbidsAffixed(flagSets, flag)) {
+ stems.forbidden = true;
+ }
+ }
+
+ /**
+ * Records that an affix analysis reached a forbidden homonym.
+ *
+ * @param flags One entry's flag set.
+ * @param flag The removed affix's flag.
+ */
+ private void noteForbidden(int[] flags, int flag) {
+ if (dictionary.forbidsAffixed(flags, flag)) {
+ stems.forbidden = true;
+ }
+ }
}
/**
@@ -145,14 +561,17 @@ public final class HunspellStemmer implements Stemmer {
* @param word The case variant to analyze.
* @param analyses The mutable, insertion-ordered set collecting the stems found.
*/
- private void analyze(String word, Set<String> analyses) {
- final List<int[]> entries = dictionary.lookup(word);
+ private void analyze(String word, Analysis analyses) {
+ final List<int[]> entries = analyses.lookup(word);
if (entries != null) {
- if (dictionary.anyForbidden(entries)) {
+ if (dictionary.firstForbidden(entries)) {
+ analyses.stems.forbidden = true;
return;
}
- if (dictionary.validStandalone(entries)) {
- analyses.add(word);
+ for (int[] flags : entries) {
+ if (dictionary.validStandalone(flags)) {
+ analyses.add(word, flags);
+ }
}
}
for (final Affix suffix : dictionary.suffixesEndingWith(
@@ -171,43 +590,271 @@ public final class HunspellStemmer implements Stemmer {
}
/**
- * Decomposes a word into listed compound parts when the affix analysis found
- * nothing: the first part must be admitted to open a compound, every further part
- * to continue or close one, each at least the declared minimum length and counted
- * against the declared maximum. A part stands on its own entry or on an entry plus
- * one affix, the way published dictionaries position their linking forms through
- * zero or dash suffixes. The stems of the parts of every successful splitting are
- * reported left to right, so the head-most material comes last. A word the
- * dictionary lists as forbidden never decomposes; that is how one specific
- * ill-formed compound is blocked while its parts stay productive.
+ * Searches compound components after standalone analysis fails. Entries and
+ * affixes must permit the selected component positions and junctions. Recognized
+ * spaced forms prevent concatenation. The candidate budget applies to compound
+ * decomposition; output follows component order.
*
* @param word The case variant to decompose.
* @param surface The surface form the variant was derived from; character case at
- * junctions is judged against it, so lowercasing a variant cannot
- * sidestep a {@code CHECKCOMPOUNDCASE} declaration.
+ * junctions is checked using this input for {@code CHECKCOMPOUNDCASE}.
* @param analyses The mutable, insertion-ordered set collecting the part stems.
*/
- private void decompose(String word, String surface, Set<String> analyses) {
+ private void decompose(String word, String surface, Results analyses) {
final List<int[]> entries = dictionary.lookup(word);
- if (entries != null && dictionary.anyForbidden(entries)) {
+ if (entries != null && dictionary.firstForbidden(entries)) {
return;
}
- final int codePointCount = word.codePointCount(0, word.length());
- if (codePointCount < 2 * dictionary.compoundMin()) {
+ if (rejectsCompoundText(word)) {
return;
}
- final int[] codePointOffsets = new int[codePointCount + 1];
- int offset = 0;
- for (int i = 0; i < codePointCount; i++) {
- codePointOffsets[i] = offset;
- offset += Character.charCount(word.codePointAt(offset));
- }
- codePointOffsets[codePointCount] = word.length();
// lowercasing may change the length in exceptional mappings, in which case the
// offsets no longer align and the variant itself is the only usable case source
final String caseSource = surface.length() == word.length() ? surface : word;
- search(word, caseSource, codePointOffsets, 0, new ArrayList<>(),
- new ArrayList<>(), analyses, new int[] {PART_CHECK_BUDGET});
+ final int[] budget = {PART_CHECK_BUDGET};
+ if (dictionary.positionalCompoundsDeclared()) {
+ searchSpelling(word, caseSource, analyses, budget, List.of());
+ for (CompoundPattern pattern : dictionary.compoundPatterns()) {
+ if (pattern.replacement() == null || pattern.replacement().isEmpty()) {
+ continue;
+ }
+ for (int at = word.indexOf(pattern.replacement()); at >= 0 && budget[0] > 0;
+ at = word.indexOf(pattern.replacement(), at + 1)) {
+ final int end = at + pattern.replacement().length();
+ final String inserted = pattern.end() + pattern.begin();
+ searchSpelling(word.substring(0, at) + inserted + word.substring(end),
+ caseSource.substring(0, at) + inserted + caseSource.substring(end),
+ analyses, budget, List.of(new Junction(at + pattern.end().length(), pattern)));
+ }
+ }
+ if (dictionary.simplifiedTriple()) {
+ searchTriples(word, caseSource, 0, List.of(), analyses, budget);
+ }
+ }
+ for (HunspellCompoundRule rule : dictionary.compoundRules()) {
+ searchRule(word, caseSource, 0, rule, new ArrayList<>(), new ArrayList<>(), analyses, budget);
+ }
+ }
+
+ /**
+ * Applies the text-level compound checks to the text every compound level splits: a
+ * {@code CHECKCOMPOUNDREP} replacement or a space inserted at any position must not
+ * produce a recognized non-compound form.
+ *
+ * @param text The complete input or the remainder a compound level splits.
+ * @return {@code true} if a check forbids splitting the text.
+ */
+ private boolean rejectsCompoundText(String text) {
+ if (dictionary.rejectsCompoundReplacement(text, this::isNoncompoundForm)) {
+ return true;
+ }
+ // without this bound every split position of a long word would be analyzed again
+ // at every compound level
+ if (!dictionary.mayReadSpacedForm(text.length() + 1)) {
+ return false;
+ }
+ for (int at = Character.charCount(text.codePointAt(0)); at < text.length();
+ at += Character.charCount(text.codePointAt(at))) {
+ if (isNoncompoundForm(text.substring(0, at) + " " + text.substring(at))) {
+ return true;
+ }
+ }
+ return false;
+ }
+
+ /**
+ * A required boundary after a compound spelling substitution.
+ *
+ * @param offset The boundary's UTF-16 offset.
+ * @param pattern The pattern permitting this boundary, or null for a repeated letter.
+ */
+ private record Junction(int offset, CompoundPattern pattern) { }
+
+ /**
+ * Restores simplified triple letters at independently checked boundaries.
+ *
+ * @param word The current text.
+ * @param surface The aligned case source.
+ * @param from The next position eligible for restoration.
+ * @param junctions The required boundaries accumulated so far.
+ * @param analyses The destination for stems.
+ * @param budget The remaining search attempts.
+ */
+ private void searchTriples(String word, String surface, int from, List<Junction> junctions,
+ Results analyses, int[] budget) {
+ if (junctions.size() >= MAX_COMPOUND_PARTS - 1) {
+ return;
+ }
+ for (int at = from; at < word.length() && budget[0] > 0;) {
+ final int point = word.codePointAt(at);
+ final int width = Character.charCount(point);
+ final int next = at + width;
+ if (next < word.length() && word.codePointAt(next) == point
+ && (at == 0 || word.codePointBefore(at) != point)
+ && (next + width == word.length() || word.codePointAt(next + width) != point)) {
+ final String inserted = new String(Character.toChars(point));
+ final String expanded = new StringBuilder(word).insert(next, inserted).toString();
+ final String expandedCase = new StringBuilder(surface).insert(next, inserted).toString();
+ for (int boundary : new int[] {next, next + width}) {
+ if (budget[0]-- <= 0) {
+ return;
+ }
+ final List<Junction> required = new ArrayList<>(junctions);
+ required.add(new Junction(boundary, null));
+ searchSpelling(expanded, expandedCase, analyses, budget, required);
+ searchTriples(expanded, expandedCase, next + 2 * width, required, analyses, budget);
+ }
+ }
+ at = next;
+ }
+ }
+
+ /**
+ * Searches one spelling, optionally requiring a restored compound junction.
+ *
+ * @param word The spelling with any compound substitution expanded.
+ * @param surface The aligned case source.
+ * @param analyses The destination for stems.
+ * @param budget The remaining candidate checks across forms.
+ * @param junctions Required boundaries for spelling substitutions.
+ */
+ private void searchSpelling(String word, String surface, Results analyses, int[] budget,
+ List<Junction> junctions) {
+ final int count = word.codePointCount(0, word.length());
+ if (count < 2 * dictionary.compoundMin()) {
+ return;
+ }
+ for (Junction junction : junctions) {
+ if (junction.offset() <= 0 || junction.offset() >= word.length()) {
+ return;
+ }
+ }
+ final int[] offsets = new int[count + 1];
+ int offset = 0;
+ for (int i = 0; i < count; i++) {
+ offsets[i] = offset;
+ offset += Character.charCount(word.codePointAt(offset));
+ }
+ offsets[count] = word.length();
+ search(word, surface, offsets, 0, new ArrayList<>(), analyses, budget, junctions,
+ new byte[count + 1]);
+ }
+
+ /**
+ * One selected compound-part reading.
+ *
+ * @param surface The part as written.
+ * @param root The entry spelling.
+ * @param flags The selected homonym's flags.
+ * @param affixes The rules in application order.
+ * @param stems The morphology-aware stems.
+ */
+ private record CompoundPart(String surface, String root, int[] flags,
+ List<Affix> affixes, List<String> stems) { }
+
+ /**
+ * Searches compound-rule paths without combining flags from different homonyms.
+ *
+ * @param word The input case variant.
+ * @param surface The input used for case restrictions.
+ * @param from The current UTF-16 offset.
+ * @param rule The compound pattern.
+ * @param parts The selected readings.
+ * @param flags The flags corresponding to the readings.
+ * @param analyses The destination for stems.
+ * @param budget The remaining candidate checks.
+ */
+ private void searchRule(String word, String surface, int from, HunspellCompoundRule rule,
+ List<CompoundPart> parts, List<int[]> flags, Results analyses, int[] budget) {
+ if (parts.size() >= MAX_COMPOUND_PARTS
+ || word.codePointCount(from, word.length()) < dictionary.compoundMin()) {
+ return;
+ }
+ int end = word.offsetByCodePoints(from, dictionary.compoundMin());
+ while (end <= word.length() && budget[0] > 0) {
+ final boolean last = end == word.length();
+ if (!(from == 0 && last)) {
+ budget[0]--;
+ final String part = word.substring(from, end);
+ final String caseSource = surface.substring(from, end);
+ for (CompoundPart candidate : ruleParts(part, caseSource, last)) {
+ if (last && dictionary.checkCompoundDup() && !parts.isEmpty()
+ && parts.get(parts.size() - 1).root().equals(candidate.root())) {
+ continue;
+ }
+ parts.add(candidate);
+ flags.add(candidate.flags());
+ if (rule.matches(flags, last)) {
+ if (last) {
+ analyses.addCompound(parts);
+ } else {
+ searchRule(word, surface, end, rule, parts, flags, analyses, budget);
+ }
+ }
+ parts.remove(parts.size() - 1);
+ flags.remove(flags.size() - 1);
+ }
+ }
+ if (last) {
+ break;
+ }
+ end += Character.charCount(word.codePointAt(end));
+ }
+ }
+
+ /**
+ * Finds selected dictionary entries for a compound-rule part.
+ *
+ * @param part The part's case variant.
+ * @param surface The part's supplied case.
+ * @param last Whether suffix removal is permitted at this position.
+ * @return The permitted readings.
+ */
+ private List<CompoundPart> ruleParts(String part, String surface, boolean last) {
+ final List<CompoundPart> result = new ArrayList<>();
+ addRulePart(part, part, surface, last, result);
+ if (last) {
+ for (Affix suffix : dictionary.suffixesEndingWith(part.codePointBefore(part.length()))) {
+ final String root = removeSuffixAllowingIdentity(part, suffix);
+ if (root != null) {
+ addRulePart(part, root, surface, true, result, suffix);
+ }
+ }
+ for (Affix suffix : dictionary.suffixesWithoutMaterial()) {
+ final String root = removeSuffixAllowingIdentity(part, suffix);
+ if (root != null) {
+ addRulePart(part, root, surface, true, result, suffix);
+ }
+ }
+ }
+ return result;
+ }
+
+ /**
+ * Adds homonyms permitted by their blocking and affix flags.
+ *
+ * @param part The component text.
+ * @param root The restored entry spelling.
+ * @param surface The supplied part spelling.
+ * @param last Whether the part closes the compound.
+ * @param result The candidate destination.
+ * @param affixes The applied suffixes.
+ */
+ private void addRulePart(String part, String root, String surface, boolean last,
+ List<CompoundPart> result, Affix... affixes) {
+ final List<int[]> entries = dictionary.lookup(root);
+ if (entries == null || dictionary.firstForbidden(entries)) {
+ return;
+ }
+ for (int[] entry : entries) {
+ if (dictionary.acceptsRulePart(entry, last, affixes)
+ && dictionary.acceptsCase(entry, surface, part, affixes)
+ && (affixes.length == 0
+ || HunspellDictionary.contains(entry, affixes[0].flag()))) {
+ result.add(new CompoundPart(part, root, entry, List.of(affixes),
+ dictionary.morphologicalStems(root, entry, affixes)));
+ }
+ }
}
/**
@@ -223,80 +870,121 @@ public final class HunspellStemmer implements Stemmer {
* form when its offsets align with the variant.
* @param codePointOffsets UTF-16 offsets for each code point boundary.
* @param fromPoint The code point index where the next part starts.
- * @param surfaces The surface strings of the parts taken so far.
- * @param stems The licensed stems of the parts taken so far, one list per part.
+ * @param parts The selected part readings.
* @param analyses The mutable, insertion-ordered set collecting the part stems.
* @param budget The remaining part-licensing attempts, counted down in place.
+ * @param junctions The required boundaries for substitutions.
+ * @param remainderChecks The cached outcome of {@link #rejectsCompoundText(String)}
+ * per code point index: {@code 0} unknown, {@code 1} allowed,
+ * {@code 2} rejected.
*/
private void search(String word, String caseSource, int[] codePointOffsets,
- int fromPoint, List<String> surfaces, List<List<String>> stems,
- Set<String> analyses, int[] budget) {
+ int fromPoint, List<CompoundPart> parts, Results analyses, int[] budget,
+ List<Junction> junctions, byte[] remainderChecks) {
final int from = codePointOffsets[fromPoint];
- if (from > 0 && violatesBoundaryChecks(word, caseSource, from)) {
+ Junction current = null;
+ for (Junction junction : junctions) {
+ if (junction.offset() == from) {
+ current = junction;
+ }
+ }
+ if (parts.size() >= MAX_COMPOUND_PARTS
+ || (from > 0 && violatesBoundaryChecks(word, caseSource, from,
+ current != null && current.pattern() == null))) {
return;
}
final int min = dictionary.compoundMin();
- final int max = dictionary.compoundWordMax();
final boolean first = from == 0;
final int remaining = codePointOffsets.length - 1 - fromPoint;
- // every split leaving room for a further part; a first-position part must also
- // leave the closing part, so the whole word is never one part
- if (remaining >= 2 * min && (max == 0 || surfaces.size() + 2 <= max)) {
- final int lastEndPoint = codePointOffsets.length - 1 - min;
- for (int endPoint = fromPoint + min; endPoint <= lastEndPoint; endPoint++) {
- if (budget[0] <= 0) {
- return;
- }
- budget[0]--;
- final int end = codePointOffsets[endPoint];
- final String part = word.substring(from, end);
- if (duplicatesNeighbor(part, surfaces)) {
+ if (remaining < min) {
+ return;
+ }
+ // this call splits the remainder into further parts, which are subject to the
+ // same text checks as the complete input
+ if (!first) {
+ if (remainderChecks[fromPoint] == 0) {
+ remainderChecks[fromPoint] = (byte) (rejectsCompoundText(word.substring(from)) ? 2 : 1);
+ }
+ if (remainderChecks[fromPoint] == 2) {
+ return;
+ }
+ }
+ for (int endPoint = fromPoint + min; endPoint < codePointOffsets.length; endPoint++) {
+ if (budget[0] <= 0) {
+ return;
+ }
+ final int end = codePointOffsets[endPoint];
+ final boolean last = end == word.length();
+ boolean spansRequiredBoundary = false;
+ for (Junction junction : junctions) {
+ spansRequiredBoundary |= from < junction.offset() && end > junction.offset();
+ }
+ if ((first && last) || spansRequiredBoundary
+ || (!last && codePointOffsets.length - 1 - endPoint < min)) {
+ continue;
+ }
+ budget[0]--;
+ final String part = word.substring(from, end);
+ final CompoundPosition position = first ? CompoundPosition.BEGIN
+ : last ? CompoundPosition.END : CompoundPosition.MIDDLE;
+ for (CompoundPart candidate : partReadings(part, caseSource.substring(from, end),
+ position, first, last, analyses.hyphenatedFirstPart)) {
+ if (!parts.isEmpty() && rejectsJunction(parts.get(parts.size() - 1), candidate,
+ current == null ? null : current.pattern(), last)) {
continue;
}
- final List<String> partStems = partStems(part,
- first ? CompoundPosition.BEGIN : CompoundPosition.MIDDLE, first, false);
- if (partStems.isEmpty()) {
- continue;
+ parts.add(candidate);
+ if (last) {
+ int units = 0;
+ int syllables = 0;
+ for (CompoundPart selected : parts) {
+ units += dictionary.compoundUnits(selected.flags(), selected.affixes());
+ syllables += dictionary.compoundSyllables(selected.surface(), selected.flags(),
+ selected.affixes(), selected == candidate);
+ }
+ if ((analyses.hyphenatedFirstPart || dictionary.compoundSizeAllowed(syllables, units))
+ && dictionary.compoundCaseAllowed(candidate.flags(), caseSource)) {
+ analyses.addCompound(parts);
+ }
+ } else {
+ search(word, caseSource, codePointOffsets, endPoint, parts, analyses,
+ budget, junctions, remainderChecks);
}
- surfaces.add(part);
- stems.add(partStems);
- search(word, caseSource, codePointOffsets, endPoint, surfaces, stems,
- analyses, budget);
- surfaces.remove(surfaces.size() - 1);
- stems.remove(stems.size() - 1);
+ parts.remove(parts.size() - 1);
}
}
- // the closing part takes the whole remainder; a compound has at least two parts
- if (first || remaining < min
- || (max > 0 && surfaces.size() + 1 > max) || budget[0] <= 0) {
- return;
- }
- budget[0]--;
- final String part = word.substring(from);
- if (duplicatesNeighbor(part, surfaces)) {
- return;
- }
- final List<String> partStems = partStems(part, CompoundPosition.END, false, true);
- if (partStems.isEmpty()) {
- return;
- }
- for (final List<String> earlier : stems) {
- analyses.addAll(earlier);
- }
- analyses.addAll(partStems);
}
/**
- * Applies the {@code CHECKCOMPOUNDDUP} declaration: a part must not repeat the
- * part directly before it.
+ * Applies the junction declarations. {@code CHECKCOMPOUNDDUP} forbids the closing
+ * part from repeating the part before it; only the two parts joined at each level
+ * are compared, so an earlier repetition is not checked. A
+ * junction restored from a pattern replacement must satisfy that pattern's flag
+ * conditions and is exempt from the other patterns; any other junction is forbidden
+ * when some pattern matches it.
*
- * @param part The candidate part.
- * @param surfaces The surface strings of the parts taken so far.
- * @return {@code true} if the declaration forbids this part here.
+ * @param left The preceding part.
+ * @param right The candidate part.
+ * @param allowed The rule allowing this substituted junction, or {@code null}.
+ * @param last Whether the candidate closes the compound.
+ * @return {@code true} if a declaration forbids this part here.
*/
- private boolean duplicatesNeighbor(String part, List<String> surfaces) {
- return dictionary.checkCompoundDup() && !surfaces.isEmpty()
- && part.equals(surfaces.get(surfaces.size() - 1));
+ private boolean rejectsJunction(CompoundPart left, CompoundPart right, CompoundPattern allowed,
+ boolean last) {
+ if (last && dictionary.checkCompoundDup() && left.root().equals(right.root())) {
+ return true;
+ }
+ if (allowed != null) {
+ return !allowed.matches(left.surface(), left.flags(), right.surface(),
+ right.flags(), left.affixes(), right.affixes());
+ }
+ for (CompoundPattern pattern : dictionary.compoundPatterns()) {
+ if (pattern.matches(left.surface(), left.flags(), right.surface(),
+ right.flags(), left.affixes(), right.affixes())) {
+ return true;
+ }
+ }
+ return false;
}
/**
@@ -308,9 +996,11 @@ public final class HunspellStemmer implements Stemmer {
* @param word The case variant under decomposition.
* @param caseSource The character-case source for the uppercase judgment.
* @param from The index the junction sits before; greater than zero.
+ * @param allowTriple Whether this junction restores a simplified repeated letter.
* @return {@code true} if a declaration forbids this junction.
*/
- private boolean violatesBoundaryChecks(String word, String caseSource, int from) {
+ private boolean violatesBoundaryChecks(String word, String caseSource, int from,
+ boolean allowTriple) {
final int before = word.codePointBefore(from);
final int after = word.codePointAt(from);
if (dictionary.checkCompoundCase()
@@ -318,7 +1008,7 @@ public final class HunspellStemmer implements Stemmer {
|| Character.isUpperCase(caseSource.codePointAt(from)))) {
return true;
}
- if (dictionary.checkCompoundTriple() && before == after) {
+ if (!allowTriple && dictionary.checkCompoundTriple() && before == after) {
final int beforeStart = from - Character.charCount(before);
final int afterEnd = from + Character.charCount(after);
if ((beforeStart > 0 && word.codePointBefore(beforeStart) == after)
@@ -330,64 +1020,152 @@ public final class HunspellStemmer implements Stemmer {
}
/**
- * Collects the listed stems that admit one part at its compound position: the part
- * as its own entry, or an entry plus one suffix or one prefix whose removal leaves
- * a listed stem, zero-material rules included, because published dictionaries
- * position their linking forms through zero and dash suffixes. An affix at a
- * compound-internal boundary must carry the permit flag, a suffix facing the next
- * part or a prefix facing the previous one. A part not found as written is also
- * tried with its first letter uppercased, the way nouns listed capitalized appear
- * lowercase inside a compound.
+ * Collects direct and affixed readings for a compound component. Affixes at
+ * internal boundaries require the permit flag. If no reading is found, a
+ * capitalized dictionary entry is also considered.
*
* @param part The part's surface text.
- * @param position The part's place in the compound.
+ * @param surface The supplied case.
+ * @param position The component position.
* @param first Whether the part opens the word.
* @param last Whether the part closes the word.
- * @return The stems admitting the part, in discovery order. Never {@code null}.
+ * @param movingRule Whether the Hungarian moving rule relaxes the opening parts.
+ * @return The permitted readings in discovery order.
*/
- private List<String> partStems(String part, CompoundPosition position,
- boolean first, boolean last) {
- final Set<String> stems = new LinkedHashSet<>();
- collectPartStems(part, position, first, last, stems);
- if (stems.isEmpty() && !part.isEmpty()) {
+ private List<CompoundPart> partReadings(String part, String surface, CompoundPosition position,
+ boolean first, boolean last, boolean movingRule) {
+ final List<CompoundPart> readings = new ArrayList<>();
+ final List<int[]> listed = last || movingRule ? null : dictionary.lookup(part);
+ if (listed != null && dictionary.forbidsCompoundStart(listed)) {
+ // a listed spelling barred from compounding is barred in its affixed readings too
+ return readings;
+ }
+ collectPartReadings(part, surface, position, first, last, movingRule, readings);
+ if (readings.isEmpty() && !part.isEmpty()) {
final int initial = part.codePointAt(0);
final int upper = Character.toUpperCase(initial);
if (upper != initial) {
- collectPartStems(new StringBuilder().appendCodePoint(upper)
+ collectPartReadings(new StringBuilder().appendCodePoint(upper)
.append(part, Character.charCount(initial), part.length()).toString(),
- position, first, last, stems);
+ surface, position, first, last, movingRule, readings);
}
}
- return List.copyOf(stems);
+ return readings;
}
/**
- * Collects the stems admitting one spelling of a part, bare and through one affix.
+ * Collects direct entries and permitted affix combinations for a component.
*
* @param part The part spelling to look up.
+ * @param surface The supplied case.
* @param position The part's place in the compound.
* @param first Whether the part opens the word.
* @param last Whether the part closes the word.
- * @param stems The mutable, insertion-ordered set collecting the stems.
+ * @param movingRule Whether an opening entry may also qualify through the hardwired
+ * Hungarian flags.
+ * @param readings The destination for selected readings.
*/
- private void collectPartStems(String part, CompoundPosition position,
- boolean first, boolean last, Set<String> stems) {
+ private void collectPartReadings(String part, String surface, CompoundPosition position,
+ boolean first, boolean last, boolean movingRule, List<CompoundPart> readings) {
final List<int[]> entries = dictionary.lookup(part);
- if (entries != null && dictionary.mayStand(entries, position)) {
- stems.add(part);
+ if (entries != null) {
+ for (int[] flags : entries) {
+ if ((dictionary.mayStand(flags, position)
+ || (movingRule && !last && dictionary.opensHyphenatedCompound(flags)))
+ && dictionary.acceptsCase(flags, surface, part)) {
+ readings.add(new CompoundPart(part, part, flags, List.of(),
+ dictionary.morphologicalStems(part, flags)));
+ }
+ }
}
for (final Affix suffix : dictionary.suffixesEndingWith(
part.codePointBefore(part.length()))) {
- collectAffixedPartStem(part, suffix, true, position, last, stems);
+ collectAffixedPartReading(part, surface, suffix, position, first, last, readings);
}
for (final Affix suffix : dictionary.suffixesWithoutMaterial()) {
- collectAffixedPartStem(part, suffix, true, position, last, stems);
+ collectAffixedPartReading(part, surface, suffix, position, first, last, readings);
}
for (final Affix prefix : dictionary.prefixesStartingWith(part.codePointAt(0))) {
- collectAffixedPartStem(part, prefix, false, position, first, stems);
+ collectAffixedPartReading(part, surface, prefix, position, first, last, readings);
+ collectCrossPartReadings(part, surface, prefix, position, first, last, readings);
}
for (final Affix prefix : dictionary.prefixesWithoutMaterial()) {
- collectAffixedPartStem(part, prefix, false, position, first, stems);
+ collectAffixedPartReading(part, surface, prefix, position, first, last, readings);
+ collectCrossPartReadings(part, surface, prefix, position, first, last, readings);
+ }
+ }
+
+ /**
+ * Searches suffix combinations after removing a compound component's prefix.
+ *
+ * @param part The component text.
+ * @param surface The supplied case.
+ * @param prefix The prefix rule.
+ * @param position The component position.
+ * @param first Whether this is the opening component.
+ * @param last Whether this is the closing component.
+ * @param readings The destination for accepted readings.
+ */
+ private void collectCrossPartReadings(String part, String surface, Affix prefix,
+ CompoundPosition position, boolean first, boolean last, List<CompoundPart> readings) {
+ if (!prefix.crossProduct() || dictionary.forbidsInCompound(prefix)
+ || (!first && !dictionary.permitsInside(prefix))) {
+ return;
+ }
+ final String intermediate = removePrefixAllowingIdentity(part, prefix);
+ if (intermediate == null) {
+ return;
+ }
+ for (Affix suffix : dictionary.suffixesEndingWith(intermediate.codePointBefore(intermediate.length()))) {
+ collectCrossPart(part, surface, intermediate, prefix, suffix, position, last, readings);
+ }
+ for (Affix suffix : dictionary.suffixesWithoutMaterial()) {
+ collectCrossPart(part, surface, intermediate, prefix, suffix, position, last, readings);
+ }
+ }
+
+ /**
+ * Validates a compound component with a prefix and suffix.
+ *
+ * @param part The component text.
+ * @param surface The supplied case.
+ * @param intermediate The text after prefix removal.
+ * @param prefix The prefix rule.
+ * @param suffix The suffix rule.
+ * @param position The component position.
+ * @param last Whether this is the closing component.
+ * @param readings The destination for accepted readings.
+ */
+ private void collectCrossPart(String part, String surface, String intermediate,
+ Affix prefix, Affix suffix, CompoundPosition position, boolean last,
+ List<CompoundPart> readings) {
+ if (!suffix.crossProduct() || dictionary.forbidsInCompound(suffix)
+ || (!last && !dictionary.permitsInside(suffix))
+ || dictionary.circumfixOnly(prefix) != dictionary.circumfixOnly(suffix)
+ || (dictionary.needsFurtherAffix(prefix) && dictionary.needsFurtherAffix(suffix))) {
+ return;
+ }
+ final String root = removeSuffixAllowingIdentity(intermediate, suffix);
+ if (root == null) {
+ return;
+ }
+ final List<int[]> entries = dictionary.lookup(root);
+ if (entries != null) {
+ for (int[] flags : entries) {
+ if (dictionary.supportsCompoundCrossProduct(flags, position, prefix, suffix)
+ && dictionary.acceptsCase(flags, surface, part, prefix, suffix)) {
+ readings.add(new CompoundPart(part, root, flags, List.of(prefix, suffix),
+ dictionary.morphologicalStems(root, flags, prefix, suffix)));
+ }
+ }
+ }
+ if (last && dictionary.compoundMoreSuffixes() && !dictionary.complexPrefixes()) {
+ for (Affix inner : dictionary.suffixesEndingWith(root.codePointBefore(root.length()))) {
+ collectCrossDoublePart(part, surface, root, inner, suffix, prefix, position, readings);
+ }
+ for (Affix inner : dictionary.suffixesWithoutMaterial()) {
+ collectCrossDoublePart(part, surface, root, inner, suffix, prefix, position, readings);
+ }
}
}
@@ -396,18 +1174,21 @@ public final class HunspellStemmer implements Stemmer {
* it at the position.
*
* @param part The part spelling under analysis.
+ * @param surface The supplied case.
* @param affix The rule to undo.
- * @param suffix Whether the rule is a suffix rule.
* @param position The part's place in the compound.
- * @param atEdge Whether the part sits at the word end the rule faces, the closing part
- * for a suffix rule and the opening part for a prefix rule; an affix
- * facing another part instead needs the permit flag.
- * @param stems The mutable, insertion-ordered set collecting the stems.
+ * @param first Whether this is the opening component.
+ * @param last Whether this is the closing component.
+ * @param readings The destination for selected readings.
*/
- private void collectAffixedPartStem(String part, Affix affix, boolean suffix,
- CompoundPosition position, boolean atEdge, Set<String> stems) {
+ private void collectAffixedPartReading(String part, String surface, Affix affix,
+ CompoundPosition position, boolean first, boolean last, List<CompoundPart> readings) {
+ final boolean suffix = affix.suffix();
+ final boolean atEdge = suffix ? last : first;
+ // a compound-only suffix joins parts and never closes a compound on its own
if (dictionary.circumfixOnly(affix) || dictionary.forbidsInCompound(affix)
- || (!atEdge && !dictionary.permitsInside(affix))) {
+ || (!atEdge && !dictionary.permitsInside(affix))
+ || (suffix && last && dictionary.compoundOnly(affix))) {
return;
}
final String stem = removeAffixInCompound(part, affix, suffix);
@@ -415,12 +1196,174 @@ public final class HunspellStemmer implements Stemmer {
return;
}
final List<int[]> flagSets = dictionary.lookup(stem);
- if (flagSets != null && dictionary.supportsPart(flagSets, affix.flag(), position,
- dictionary.affixAdmits(affix, position))) {
- stems.add(stem);
+ if (flagSets != null && !dictionary.needsFurtherAffix(affix)) {
+ for (int[] flags : flagSets) {
+ if (dictionary.supportsPart(flags, affix.flag(), position,
+ dictionary.affixAdmits(affix, position))
+ && dictionary.acceptsCase(flags, surface, part, affix)) {
+ readings.add(new CompoundPart(part, stem, flags, List.of(affix),
+ dictionary.morphologicalStems(stem, flags, affix)));
+ }
+ }
+ }
+ if (dictionary.compoundMoreSuffixes() && suffix != dictionary.complexPrefixes() && atEdge) {
+ final List<Affix> material = suffix
+ ? dictionary.suffixesEndingWith(stem.codePointBefore(stem.length()))
+ : dictionary.prefixesStartingWith(stem.codePointAt(0));
+ final List<Affix> zero = suffix
+ ? dictionary.suffixesWithoutMaterial() : dictionary.prefixesWithoutMaterial();
+ for (Affix inner : material) {
+ collectDoublePart(part, surface, stem, inner, affix, position, last, readings);
+ }
+ for (Affix inner : zero) {
+ collectDoublePart(part, surface, stem, inner, affix, position, last, readings);
+ }
}
}
+ /**
+ * Finds a compound part with continuation-linked affixes.
+ *
+ * @param part The component text.
+ * @param surface The supplied case.
+ * @param intermediate The form after outer affix removal.
+ * @param inner The inner rule.
+ * @param outer The outer rule.
+ * @param position The compound position.
+ * @param last Whether this is the final part.
+ * @param readings The destination for readings.
+ */
+ private void collectDoublePart(String part, String surface, String intermediate,
+ Affix inner, Affix outer, CompoundPosition position, boolean last,
+ List<CompoundPart> readings) {
+ if (!inner.allowsContinuation(outer.flag()) || dictionary.forbidsInCompound(inner)) {
+ return;
+ }
+ final String root = removeAffixInCompound(intermediate, inner, inner.suffix());
+ if (root == null) {
+ return;
+ }
+ final List<int[]> entries = dictionary.lookup(root);
+ if (entries != null && !dictionary.circumfixOnly(inner)) {
+ for (int[] flags : entries) {
+ if (dictionary.supportsPart(flags, inner.flag(), position,
+ dictionary.affixAdmits(inner, position) || dictionary.affixAdmits(outer, position))
+ && dictionary.acceptsCase(flags, surface, part, inner, outer)) {
+ readings.add(new CompoundPart(part, root, flags, List.of(inner, outer),
+ dictionary.morphologicalStems(root, flags, inner, outer)));
+ }
+ }
+ }
+ if (!inner.suffix() && inner.crossProduct() && outer.crossProduct()) {
+ for (Affix suffix : dictionary.suffixesEndingWith(root.codePointBefore(root.length()))) {
+ collectCompoundSuffixAfterPrefixes(part, surface, root, inner, outer, suffix,
+ position, last, readings);
+ }
+ for (Affix suffix : dictionary.suffixesWithoutMaterial()) {
+ collectCompoundSuffixAfterPrefixes(part, surface, root, inner, outer, suffix,
+ position, last, readings);
+ }
+ }
+ }
+
+ /**
+ * Checks a suffix after continuation-linked compound prefixes.
+ *
+ * @param part The component text.
+ * @param surface The supplied case.
+ * @param text The text after removing the prefixes.
+ * @param inner The inner prefix.
+ * @param outer The outer prefix.
+ * @param suffix The suffix rule.
+ * @param position The component position.
+ * @param last Whether this is the final component.
+ * @param readings The result destination.
+ */
+ private void collectCompoundSuffixAfterPrefixes(String part, String surface, String text,
+ Affix inner, Affix outer, Affix suffix, CompoundPosition position, boolean last,
+ List<CompoundPart> readings) {
+ if (!last && !dictionary.permitsInside(suffix)) {
+ return;
+ }
+ final String root = removeSuffixAllowingIdentity(text, suffix);
+ if (root != null) {
+ addCompoundCrossDouble(part, surface, root, inner, outer, suffix, position, readings);
+ }
+ }
+
+ /**
+ * Removes an inner compound suffix after a prefix and outer suffix.
+ *
+ * @param part The component text.
+ * @param surface The supplied case.
+ * @param text The text after removing the outer suffix and prefix.
+ * @param inner The inner suffix.
+ * @param outer The outer suffix.
+ * @param prefix The prefix rule.
+ * @param position The component position.
+ * @param readings The result destination.
+ */
+ private void collectCrossDoublePart(String part, String surface, String text, Affix inner,
+ Affix outer, Affix prefix, CompoundPosition position, List<CompoundPart> readings) {
+ final String root = removeSuffixAllowingIdentity(text, inner);
+ if (root != null) {
+ addCompoundCrossDouble(part, surface, root, inner, outer, prefix, position, readings);
+ }
+ }
+
+ /**
+ * Validates a continuation sequence combined with an opposite-end affix.
+ *
+ * @param part The component text.
+ * @param surface The supplied case.
+ * @param root The restored entry text.
+ * @param inner The inner rule in the continuation sequence.
+ * @param outer The outer rule in the continuation sequence.
+ * @param cross The opposite-end rule.
+ * @param position The component position.
+ * @param readings The result destination.
+ */
+ private void addCompoundCrossDouble(String part, String surface, String root, Affix inner,
+ Affix outer, Affix cross, CompoundPosition position, List<CompoundPart> readings) {
+ if (!inner.allowsContinuation(outer.flag()) || !inner.crossProduct() || !cross.crossProduct()
+ || dictionary.forbidsInCompound(inner) || dictionary.forbidsInCompound(cross)
+ || dictionary.circumfixOnly(outer)
+ || dictionary.circumfixOnly(inner) != dictionary.circumfixOnly(cross)) {
+ return;
+ }
+ final Affix prefix = inner.suffix() ? cross : inner;
+ final Affix suffix = inner.suffix() ? inner : cross;
+ final Affix[] applied = inner.suffix()
+ ? new Affix[] {cross, inner, outer} : new Affix[] {inner, outer, cross};
+ final List<int[]> entries = dictionary.lookup(root);
+ if (entries != null) {
+ for (int[] flags : entries) {
+ if (dictionary.supportsCompoundCrossProduct(flags, position, prefix, suffix, outer)
+ && dictionary.acceptsCase(flags, surface, part, applied)) {
+ readings.add(new CompoundPart(part, root, flags, List.of(applied),
+ dictionary.morphologicalStems(root, flags, applied)));
+ }
+ }
+ }
+ }
+
+ /**
+ * Checks a direct or affixed form without compound recursion.
+ *
+ * @param word The candidate spelling.
+ * @return Whether a non-compound reading exists.
+ */
+ private boolean isNoncompoundForm(String word) {
+ if (word.isEmpty()) {
+ return false;
+ }
+ final Results stems = new Results(false);
+ for (String variant : variants(word)) {
+ analyze(variant, new Analysis(word, variant, stems, false));
+ }
+ return !stems.isEmpty();
+ }
+
/**
* Undoes one affix rule on a compound part. Unlike the standalone removals, a rule
* that neither adds nor removes material is undone here, to its own spelling with
@@ -443,28 +1386,35 @@ public final class HunspellStemmer implements Stemmer {
* the intermediate stem, adding dictionary-confirmed analyses. A rule that applies
* only inside compounds or requires the matching circumfix member is not undone
* because no prefix accompanies this path. A rule requiring a further affix produces
- * no single-removal analysis. An identity rule also produces no single-removal
- * analysis, but it can complete a two-suffix analysis through continuation classes.
+ * no single-removal analysis. A rule that adds and strips no material is undone like
+ * any other, which recognizes a virtual stem it completes and reports its fields.
*
* @param word The case variant under analysis.
* @param suffix The suffix rule to undo.
* @param analyses The mutable, insertion-ordered set collecting the stems found.
*/
- private void undoSuffix(String word, Affix suffix, Set<String> analyses) {
+ private void undoSuffix(String word, Affix suffix, Analysis analyses) {
if (dictionary.compoundOnly(suffix) || dictionary.circumfixOnly(suffix)) {
return;
}
- final boolean identity = isIdentityRule(suffix);
final String stem = removeSuffixAllowingIdentity(word, suffix);
if (stem == null) {
return;
}
- if (!identity && !dictionary.needsFurtherAffix(suffix)) {
- final List<int[]> flagSets = dictionary.lookup(stem);
- if (flagSets != null && dictionary.supports(flagSets, suffix.flag())) {
- analyses.add(stem);
+ if (!dictionary.needsFurtherAffix(suffix)) {
+ final List<int[]> flagSets = analyses.lookup(stem);
+ if (flagSets != null) {
+ analyses.noteForbidden(flagSets, suffix.flag());
+ for (int[] flags : flagSets) {
+ if (dictionary.supports(flags, suffix.flag())) {
+ analyses.add(stem, flags, suffix);
+ }
+ }
}
}
+ if (dictionary.complexPrefixes()) {
+ return;
+ }
for (final Affix inner : dictionary.suffixesEndingWith(
stem.codePointBefore(stem.length()))) {
undoInnerSuffix(stem, suffix, inner, analyses);
@@ -485,7 +1435,7 @@ public final class HunspellStemmer implements Stemmer {
* @param analyses The mutable, insertion-ordered set collecting the stems found.
*/
private void undoInnerSuffix(String stem, Affix outer, Affix inner,
- Set<String> analyses) {
+ Analysis analyses) {
if (!inner.allowsContinuation(outer.flag()) || dictionary.compoundOnly(inner)
|| dictionary.circumfixOnly(inner)) {
return;
@@ -494,9 +1444,13 @@ public final class HunspellStemmer implements Stemmer {
if (doubleStem == null) {
return;
}
- final List<int[]> innerFlags = dictionary.lookup(doubleStem);
- if (innerFlags != null && dictionary.supports(innerFlags, inner.flag())) {
- analyses.add(doubleStem);
+ final List<int[]> innerFlags = analyses.lookup(doubleStem);
+ if (innerFlags != null) {
+ for (int[] flags : innerFlags) {
+ if (dictionary.supports(flags, inner.flag())) {
+ analyses.add(doubleStem, flags, inner, outer);
+ }
+ }
}
}
@@ -505,27 +1459,38 @@ public final class HunspellStemmer implements Stemmer {
* intermediate stem, adding dictionary-confirmed analyses. A rule that
* applies only inside compounds is not undone at all. A rule marked as needing a
* further affix or the matching circumfix member produces no single-removal analysis.
- * An identity rule also produces no single-removal analysis. A valid cross-product
- * suffix can combine with either kind of rule.
+ * A rule that adds and strips no material is undone like any other. A valid
+ * cross-product suffix can combine with either kind of rule.
*
* @param word The case variant under analysis.
* @param prefix The prefix rule to undo.
* @param analyses The mutable, insertion-ordered set collecting the stems found.
*/
- private void undoPrefix(String word, Affix prefix, Set<String> analyses) {
+ private void undoPrefix(String word, Affix prefix, Analysis analyses) {
if (dictionary.compoundOnly(prefix)) {
return;
}
- final boolean identity = isIdentityRule(prefix);
final String stem = removePrefixAllowingIdentity(word, prefix);
if (stem == null) {
return;
}
- if (!identity && !dictionary.needsFurtherAffix(prefix)
- && !dictionary.circumfixOnly(prefix)) {
- final List<int[]> flagSets = dictionary.lookup(stem);
- if (flagSets != null && dictionary.supports(flagSets, prefix.flag())) {
- analyses.add(stem);
+ if (!dictionary.needsFurtherAffix(prefix) && !dictionary.circumfixOnly(prefix)) {
+ final List<int[]> flagSets = analyses.lookup(stem);
+ if (flagSets != null) {
+ analyses.noteForbidden(flagSets, prefix.flag());
+ for (int[] flags : flagSets) {
+ if (dictionary.supports(flags, prefix.flag())) {
+ analyses.add(stem, flags, prefix);
+ }
+ }
+ }
+ }
+ if (dictionary.complexPrefixes()) {
+ for (Affix inner : dictionary.prefixesStartingWith(stem.codePointAt(0))) {
+ undoInnerPrefix(stem, prefix, inner, analyses);
+ }
+ for (Affix inner : dictionary.prefixesWithoutMaterial()) {
+ undoInnerPrefix(stem, prefix, inner, analyses);
}
}
if (!prefix.crossProduct()) {
@@ -552,7 +1517,7 @@ public final class HunspellStemmer implements Stemmer {
* @param analyses The mutable, insertion-ordered set collecting the stems found.
*/
private void undoCrossProductSuffix(String stem, Affix prefix, Affix suffix,
- Set<String> analyses) {
+ Analysis analyses) {
if (!suffix.crossProduct() || dictionary.compoundOnly(suffix)
|| dictionary.circumfixOnly(prefix) != dictionary.circumfixOnly(suffix)) {
return;
@@ -563,11 +1528,20 @@ public final class HunspellStemmer implements Stemmer {
}
// One member can satisfy the other member's needs-further-affix marker. Both rule
// flags must occur in one homonym's flag set.
- final List<int[]> both = dictionary.lookup(doubleStem);
- if (both != null && dictionary.supportsCrossProduct(both, prefix, suffix)
- && !(dictionary.needsFurtherAffix(prefix)
- && dictionary.needsFurtherAffix(suffix))) {
- analyses.add(doubleStem);
+ final List<int[]> both = analyses.lookup(doubleStem);
+ if (both != null && !(dictionary.needsFurtherAffix(prefix)
+ && dictionary.needsFurtherAffix(suffix))) {
+ for (int[] flags : both) {
+ if (dictionary.licensesCrossProduct(flags, prefix, suffix)) {
+ analyses.noteForbidden(flags, suffix.flag());
+ }
+ if (dictionary.supportsCrossProduct(flags, prefix, suffix)) {
+ analyses.add(doubleStem, flags, prefix, suffix);
+ }
+ }
+ }
+ if (dictionary.complexPrefixes()) {
+ return;
}
for (final Affix inner : dictionary.suffixesEndingWith(
doubleStem.codePointBefore(doubleStem.length()))) {
@@ -590,7 +1564,7 @@ public final class HunspellStemmer implements Stemmer {
* @param analyses The mutable, insertion-ordered set collecting the stems found.
*/
private void undoCrossProductInnerSuffix(String stem, Affix prefix, Affix outer,
- Affix inner, Set<String> analyses) {
+ Affix inner, Analysis analyses) {
if (!inner.crossProduct() || !inner.allowsContinuation(outer.flag())
|| dictionary.compoundOnly(inner)
|| dictionary.circumfixOnly(outer)
@@ -601,9 +1575,76 @@ public final class HunspellStemmer implements Stemmer {
if (root == null) {
return;
}
- final List<int[]> flagSets = dictionary.lookup(root);
- if (flagSets != null && dictionary.supportsCrossProduct(flagSets, prefix, inner)) {
- analyses.add(root);
+ final List<int[]> flagSets = analyses.lookup(root);
+ if (flagSets != null) {
+ for (int[] flags : flagSets) {
+ if (dictionary.supportsCrossProduct(flags, prefix, inner)) {
+ analyses.add(root, flags, prefix, inner, outer);
+ }
+ }
+ }
+ }
+
+ /**
+ * Removes a continuation-linked inner prefix in COMPLEXPREFIXES mode.
+ *
+ * @param stem The form after the outer prefix was removed.
+ * @param outer The outer prefix.
+ * @param inner The inner prefix.
+ * @param analyses The result context.
+ */
+ private void undoInnerPrefix(String stem, Affix outer, Affix inner, Analysis analyses) {
+ if (!inner.allowsContinuation(outer.flag()) || dictionary.compoundOnly(inner)
+ || dictionary.circumfixOnly(outer)) {
+ return;
+ }
+ final String root = removePrefixAllowingIdentity(stem, inner);
+ if (root == null) {
+ return;
+ }
+ final List<int[]> entries = analyses.lookup(root);
+ if (entries != null && !dictionary.circumfixOnly(inner)) {
+ for (int[] flags : entries) {
+ if (dictionary.supports(flags, inner.flag())) {
+ analyses.add(root, flags, inner, outer);
+ }
+ }
+ }
+ if (inner.crossProduct() && outer.crossProduct()) {
+ for (Affix suffix : dictionary.suffixesEndingWith(root.codePointBefore(root.length()))) {
+ undoDoublePrefixSuffix(root, inner, outer, suffix, analyses);
+ }
+ for (Affix suffix : dictionary.suffixesWithoutMaterial()) {
+ undoDoublePrefixSuffix(root, inner, outer, suffix, analyses);
+ }
+ }
+ }
+
+ /**
+ * Removes the suffix following a continuation-linked prefix combination.
+ *
+ * @param word The form after prefix removal.
+ * @param inner The inner prefix.
+ * @param outer The outer prefix.
+ * @param suffix The candidate suffix.
+ * @param analyses The result context.
+ */
+ private void undoDoublePrefixSuffix(String word, Affix inner, Affix outer,
+ Affix suffix, Analysis analyses) {
+ if (!suffix.crossProduct() || dictionary.compoundOnly(suffix)
+ || dictionary.circumfixOnly(inner) != dictionary.circumfixOnly(suffix)) {
+ return;
+ }
+ final String root = removeSuffixAllowingIdentity(word, suffix);
+ if (root != null) {
+ final List<int[]> entries = analyses.lookup(root);
+ if (entries != null) {
+ for (int[] flags : entries) {
+ if (dictionary.supportsCrossProduct(flags, inner, suffix)) {
+ analyses.add(root, flags, inner, outer, suffix);
+ }
+ }
+ }
}
}
@@ -612,10 +1653,11 @@ public final class HunspellStemmer implements Stemmer {
* the strip string the rule removed on application, and checks the rule's condition
* against the restored stem. A strip-only rule, whose affix material is empty, is
* undone by restoring its strip string alone. Rules that neither add nor remove
- * material and candidates that would leave an empty stem are rejected. A word the
- * affix material covers entirely reverses a full-strip application, which hunspell
- * only performs when the affix file declares {@code FULLSTRIP}; without that
- * declaration the rule does not apply.
+ * material are handled by {@link #removeSuffixAllowingIdentity(String, Affix)}, and
+ * candidates that would leave an empty stem are rejected. A word the affix material
+ * covers entirely reverses a full-strip application, which hunspell only performs
+ * when the affix file declares {@code FULLSTRIP}; without that declaration the rule
+ * does not apply.
*
* @param word The surface form.
* @param suffix The rule to undo.
@@ -678,7 +1720,8 @@ public final class HunspellStemmer implements Stemmer {
* restores the strip string the rule removed on application, and checks the rule's
* condition against the restored stem. A strip-only rule, whose affix material is
* empty, is undone by restoring its strip string alone. Rules that neither add nor
- * remove material and candidates that would leave an empty stem are rejected. A
+ * remove material are handled by {@link #removePrefixAllowingIdentity(String, Affix)},
+ * and candidates that would leave an empty stem are rejected. A
* word the affix material covers entirely reverses a full-strip application, which
* hunspell only performs when the affix file declares {@code FULLSTRIP}; without
* that declaration the rule does not apply.
diff --git a/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/stemmer/hunspell/HunspellCompatibilityTest.java b/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/stemmer/hunspell/HunspellCompatibilityTest.java
new file mode 100644
index 000000000..206d094c9
--- /dev/null
+++ b/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/stemmer/hunspell/HunspellCompatibilityTest.java
@@ -0,0 +1,480 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package opennlp.tools.stemmer.hunspell;
+
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Stream;
+
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.MethodSource;
+
+/**
+ * Dictionary fixtures for Hunspell conversion, casing, compounding, and morphology.
+ * The fixtures are written for this test: each declares its own words and flags for
+ * the rule it exercises. The reference outcomes are recorded from Hunspell as described
+ * in {@code dev/README-hunspell-dictionaries.md}.
+ */
+class HunspellCompatibilityTest {
+
+ private static final String PLURAL = "SFX A Y 1\nSFX A 0 s .\n";
+ private static final String COMPOUND = "COMPOUNDFLAG C\nCOMPOUNDMIN 1\n";
+
+ /**
+ * One original dictionary and an expected stemming result.
+ *
+ * @param name The test identifier.
+ * @param affix The affix content.
+ * @param words The dictionary content.
+ * @param input The input form.
+ * @param expected The expected stems, with identity for unknown input.
+ */
+ private record Example(String name, String affix, String words,
+ String input, List<String> expected) {
+ /** {@inheritDoc} */
+ @Override
+ public String toString() {
+ return name;
+ }
+ }
+
+ /**
+ * Supplies the fixtures.
+ *
+ * @return The examples.
+ */
+ private static Stream<Example> examples() {
+ return Stream.of(
+ new Example("input-ligature", "ICONV 1\nICONV fi fi\n" + PLURAL,
+ "1\nfield/A\n", "fields", List.of("field")),
+ new Example("longest-input-match", "ICONV 2\nICONV æ ae\nICONV æx ax\n" + PLURAL,
+ "1\nax/A\n", "æxs", List.of("ax")),
+ new Example("input-end-anchor", "ICONV 1\nICONV z_ s\n" + PLURAL,
+ "1\nquartz/A\n", "quartzz", List.of("quartz")),
+ new Example("output-conversion", "OCONV 1\nOCONV ae ä\n" + PLURAL,
+ "1\nbaer/A\n", "baers", List.of("bär")),
+ new Example("ignored-input-and-dictionary", "IGNORE ’\n" + PLURAL,
+ "1\npe’arl/A\n", "pear’ls", List.of("pearl")),
+ new Example("ignored-affix", "IGNORE ’\nSFX A Y 1\nSFX A 0 s’ .\n",
+ "1\npearl/A\n", "pearls", List.of("pearl")),
+ new Example("keepcase-exact", "KEEPCASE K\n" + PLURAL,
+ "1\ncard/AK\n", "cards", List.of("card")),
+ new Example("keepcase-title", "KEEPCASE K\n" + PLURAL,
+ "1\ncard/AK\n", "Cards", List.of("Cards")),
+ new Example("keepcase-uppercase", "KEEPCASE K\n" + PLURAL,
+ "1\ncard/AK\n", "CARDS", List.of("CARDS")),
+ new Example("mixed-case-is-not-lowercase", PLURAL,
+ "1\ncard/A\n", "cArds", List.of("cArds")),
+ new Example("uppercase-proper-name", PLURAL,
+ "1\nMaren/A\n", "MARENS", List.of("Maren")),
+ new Example("turkish-case", "LANG tr\n" + PLURAL,
+ "1\nılık/A\n", "ILIKS", List.of("ılık")),
+ new Example("complex-prefixes", "COMPLEXPREFIXES\nPFX B Y 1\n"
+ + "PFX B 0 re/C .\nPFX C Y 1\nPFX C 0 un .\n" + PLURAL,
+ "1\ndo/AB\n", "unredos", List.of("do")),
+ new Example("complex-prefix-requires-continuation", "COMPLEXPREFIXES\n"
+ + "PFX B Y 1\nPFX B 0 re .\nPFX C Y 1\nPFX C 0 un .\n",
+ "1\ndo/BC\n", "unredo", List.of("unredo")),
+ new Example("complex-prefix-single-suffix", "COMPLEXPREFIXES\n"
+ + "SFX B Y 1\nSFX B 0 er/C .\nSFX C Y 1\nSFX C 0 s .\n",
+ "1\nwalk/B\n", "walkers", List.of("walkers")),
+ new Example("explicit-stem", PLURAL, "1\nfeet/A st:foot is:plural\n",
+ "feet", List.of("foot")),
+ new Example("affixed-explicit-stem", PLURAL,
+ "1\nfeet/A st:foot is:plural\n", "feets", List.of("foot")),
+ new Example("morphology-alias", "AM 1\nAM st:goose is:plural\n" + PLURAL,
+ "1\ngeese/A\t1\n", "geese", List.of("goose")),
+ new Example("derivational-suffix", "SFX A Y 1\nSFX A 0 ness/B . ds:noun\n"
+ + "SFX B Y 1\nSFX B 0 es . is:plural\n",
+ "1\nkind/A po:adj\n", "kindnesses", List.of("kindness")),
+ new Example("surface-prefix", PLURAL, "1\nroot/A sp:pre st:base\n",
+ "roots", List.of("prebase")),
+ new Example("sharp-s-uppercase", "CHECKSHARPS\n" + PLURAL,
+ "1\nsoße/A\n", "SOSSES", List.of("soße")),
+ new Example("sharp-s-keepcase", "CHECKSHARPS\nKEEPCASE K\n" + PLURAL,
+ "1\nsoße/AK\n", "SOSSES", List.of("soße")),
+ new Example("forbidden-warning", "WARN W\nFORBIDWARN\n" + PLURAL,
+ "1\ncard/AW\n", "cards", List.of("cards")),
+ new Example("compound-rule", "COMPOUNDMIN 1\nCOMPOUNDRULE 1\nCOMPOUNDRULE RS\n"
+ + PLURAL, "2\nriver/R\nboat/AS\n", "riverboats", List.of("river", "boat")),
+ new Example("compound-rule-order", "COMPOUNDMIN 1\nCOMPOUNDRULE 1\nCOMPOUNDRULE RS\n",
+ "2\nriver/R\nboat/S\n", "boatriver", List.of("boatriver")),
+ new Example("compound-rule-star", "COMPOUNDMIN 1\nCOMPOUNDRULE 1\nCOMPOUNDRULE R*S\n",
+ "3\nriver/R\nstone/R\nboat/S\n", "riverstoneboat", List.of("river", "stone", "boat")),
+ new Example("compound-rule-optional", "COMPOUNDMIN 1\nCOMPOUNDRULE 1\nCOMPOUNDRULE R?TS\n",
+ "3\nriver/R\nstone/T\nboat/S\n", "stoneboat", List.of("stone", "boat")),
+ new Example("compound-rule-long", "FLAG long\nCOMPOUNDMIN 1\nCOMPOUNDRULE 1\n"
+ + "COMPOUNDRULE (Ra)(Sa)\n", "2\nriver/Ra\nboat/Sa\n",
+ "riverboat", List.of("river", "boat")),
+ new Example("compound-rule-numeric", "FLAG num\nCOMPOUNDMIN 1\nCOMPOUNDRULE 1\n"
+ + "COMPOUNDRULE (12)(34)\n", "2\nriver/12\nboat/34\n",
+ "riverboat", List.of("river", "boat")),
+ new Example("compound-rule-homonyms", "COMPOUNDMIN 1\nCOMPOUNDRULE 1\n"
+ + "COMPOUNDRULE RTS\n", "3\nriver/R\nriver/T\nboat/S\n",
+ "riverboat", List.of("riverboat")),
+ new Example("compound-force-uppercase", COMPOUND + "FORCEUCASE U\n",
+ "2\nriver/C\nboat/CU\n", "Riverboat", List.of("river", "boat")),
+ new Example("compound-force-uppercase-reject", COMPOUND + "FORCEUCASE U\n",
+ "2\nriver/C\nboat/CU\n", "riverboat", List.of("riverboat")),
+ new Example("compound-root-count", COMPOUND + "COMPOUNDROOT R\nCOMPOUNDWORDMAX 3\n",
+ "3\nrain/CR\ncoat/C\nrack/C\n", "raincoatrack", List.of("raincoatrack")),
+ new Example("compound-root-count-accept", COMPOUND + "COMPOUNDROOT R\nCOMPOUNDWORDMAX 4\n",
+ "3\nrain/CR\ncoat/C\nrack/C\n", "raincoatrack", List.of("rain", "coat", "rack")),
+ new Example("compound-replacement-check", COMPOUND + "CHECKCOMPOUNDREP\nREP 1\nREP coat boat\n",
+ "3\nrain/C\ncoat/C\nrainboat\n", "raincoat", List.of("raincoat")),
+ new Example("compound-pattern", COMPOUND + "CHECKCOMPOUNDPATTERN 1\n"
+ + "CHECKCOMPOUNDPATTERN er b\n", "2\nriver/C\nboat/C\n",
+ "riverboat", List.of("riverboat")),
+ new Example("compound-pattern-flags", COMPOUND + "CHECKCOMPOUNDPATTERN 1\n"
+ + "CHECKCOMPOUNDPATTERN er/X b/Y\n", "2\nriver/C\nboat/CY\n",
+ "riverboat", List.of("river", "boat")),
+ new Example("compound-pattern-replacement", COMPOUND + "CHECKCOMPOUNDPATTERN 1\n"
+ + "CHECKCOMPOUNDPATTERN er b X\n", "2\nriver/C\nboat/C\n",
+ "rivXoat", List.of("river", "boat")),
+ new Example("compound-simplified-triple", COMPOUND + "CHECKCOMPOUNDTRIPLE\nSIMPLIFIEDTRIPLE\n",
+ "2\nmill/C\nloom/C\n", "milloom", List.of("mill", "loom")),
+ new Example("compound-more-suffixes", COMPOUND + "COMPOUNDMORESUFFIXES\n"
+ + "SFX A Y 1\nSFX A 0 er/B .\nSFX B Y 1\nSFX B 0 s .\n",
+ "2\nriver/C\nboat/CA\n", "riverboaters", List.of("river", "boat")),
+ new Example("compound-syllable-limit", COMPOUND + "LANG hu\nCOMPOUNDWORDMAX 2\n"
+ + "COMPOUNDSYLLABLE 4 aeiouy\n", "3\nray/C\nme/C\nfa/C\n",
+ "raymefa", List.of("ray", "me", "fa")),
+ new Example("compound-syllable-limit-reject", COMPOUND + "LANG hu\nCOMPOUNDWORDMAX 2\n"
+ + "COMPOUNDSYLLABLE 2 aeiouy\n", "3\nray/C\nme/C\nfa/C\n",
+ "raymefa", List.of("raymefa")),
+ new Example("break-default", PLURAL, "2\nriver/A\nboat/A\n",
+ "rivers-boats", List.of("river", "boat")),
+ new Example("break-recursive", PLURAL, "2\nriver/A\nboat/A\n",
+ "rivers-boats-rivers", List.of("river", "boat")),
+ new Example("break-start", PLURAL, "1\nriver/A\n", "-rivers", List.of("river")),
+ new Example("break-end", PLURAL, "1\nriver/A\n", "rivers-", List.of("river")),
+ new Example("break-custom", "BREAK 1\nBREAK ::\n" + PLURAL,
+ "2\nriver/A\nboat/A\n", "rivers::boats", List.of("river", "boat")),
+ new Example("break-disabled", "BREAK 0\n" + PLURAL,
+ "2\nriver/A\nboat/A\n", "rivers-boats", List.of("rivers-boats")),
+ new Example("break-unknown-part", PLURAL, "1\nriver/A\n",
+ "rivers-absent", List.of("rivers-absent")),
+ new Example("break-internal-only", "BREAK 1\nBREAK -\n" + PLURAL,
+ "1\nriver/A\n", "-rivers", List.of("-rivers")),
+ new Example("replacement-trailing-fields", COMPOUND + "CHECKCOMPOUNDREP\n"
+ + "REP 1\nREP coat boat trailing_metadata\n",
+ "3\nrain/C\ncoat/C\nrainboat\n", "raincoat", List.of("raincoat")),
+ new Example("replacement-morphology", COMPOUND + "CHECKCOMPOUNDREP\n",
+ "3\nrain/C\ncoat/C\nrainboat ph:raincoat\n", "raincoat", List.of("raincoat")),
+ new Example("replacement-morphology-arrow", COMPOUND + "CHECKCOMPOUNDREP\n",
+ "3\nrain/C\ncoat/C\nrainboat ph:coat->boat\n", "raincoat", List.of("raincoat")),
+ new Example("replacement-morphology-star-unlisted", COMPOUND + "CHECKCOMPOUNDREP\n" + PLURAL,
+ "3\nrain/C\ncoat/C\nrainboats/A ph:raincoats*\n", "raincoat", List.of("rain", "coat")),
+ new Example("replacement-morphology-star", COMPOUND + "CHECKCOMPOUNDREP\n" + PLURAL,
+ "4\nrain/C\ncoat/C\nrainboats/A ph:raincoats*\nrainboat\n",
+ "raincoat", List.of("raincoat")),
+ new Example("derivation-surface-prefix", "PFX U Y 1\nPFX U 0 mis . dp:pfx_mis sp:mis\n"
+ + "SFX A Y 1\nSFX A 0 ment/U . ds:der_ment\n", "1\nmanage/A po:verb\n",
+ "mismanagement", List.of("mismanagement")),
+ new Example("derivation-inflectional-prefix", "PFX P Y 1\nPFX P 0 mis . ip:mis\n"
+ + "SFX R Y 1\nSFX R 0 ment/P . ds:DER\n", "1\nmanage/R po:verb\n",
+ "mismanagement", List.of("management")),
+ new Example("compound-pattern-substitution-only", COMPOUND + "CHECKCOMPOUNDPATTERN 2\n"
+ + "CHECKCOMPOUNDPATTERN a t k\nCHECKCOMPOUNDPATTERN ea ta y\n",
+ "2\nsea/C\ntable/C\n", "sekable", List.of("sea", "table")),
+ new Example("compound-pattern-substitution-second", COMPOUND + "CHECKCOMPOUNDPATTERN 2\n"
+ + "CHECKCOMPOUNDPATTERN a t k\nCHECKCOMPOUNDPATTERN ea ta y\n",
+ "2\nsea/C\ntable/C\n", "syble", List.of("sea", "table")),
+ new Example("compound-duplicate-last-parts", COMPOUND + "CHECKCOMPOUNDDUP\n",
+ "2\nfoo/C\nbar/C\n", "foofoobar", List.of("foo", "bar")),
+ new Example("compound-duplicate-reject", COMPOUND + "CHECKCOMPOUNDDUP\n",
+ "2\nfoo/C\nbar/C\n", "foobarbar", List.of("foobarbar")),
+ new Example("compound-forbid-entry", COMPOUND_FORBID, COMPOUND_FORBID_WORDS,
+ "firewardhouse", List.of("firewardhouse")),
+ new Example("compound-forbid-entry-other-suffix", COMPOUND_FORBID, COMPOUND_FORBID_WORDS,
+ "firewoodhouse", List.of("fire", "house")),
+ new Example("compound-only-suffix-at-end", ONLY_IN_COMPOUND, ONLY_IN_COMPOUND_WORDS,
+ "lampglassen", List.of("lampglassen")),
+ new Example("compound-only-suffix-inside", ONLY_IN_COMPOUND, ONLY_IN_COMPOUND_WORDS,
+ "glassenlamp", List.of("glass", "lamp")),
+ new Example("compound-replacement-inner", COMPOUND + "CHECKCOMPOUNDREP\n"
+ + "REP 1\nREP wallpaper wall_paper\n",
+ "3\nwall/C\npaper/C\nwall paper\n", "paperwallpaper",
+ List.of("paperwallpaper")),
+ new Example("compound-replacement-inner-unaffected", COMPOUND + "CHECKCOMPOUNDREP\n"
+ + "REP 1\nREP wallpaper wall_paper\n",
+ "3\nwall/C\npaper/C\nwall paper\n", "paperwall",
+ List.of("paper", "wall")),
+ new Example("mixed-case-initial-capital", "PFX b Y 1\nPFX b e ra e\n",
+ "1\neMarket/b\n", "RaMarket", List.of("eMarket")),
+ new Example("mixed-case-initial-capital-entry", "PFX b Y 1\nPFX b e ra e\n",
+ "1\neMarket/b\n", "EMarket", List.of("eMarket")),
+ new Example("forbidden-affixed-blocks-compound", FORBIDDEN_AFFIXED, FORBIDDEN_AFFIXED_WORDS,
+ "sunlightrooms", List.of("sunlightrooms")),
+ new Example("forbidden-affixed-other-order", FORBIDDEN_AFFIXED, FORBIDDEN_AFFIXED_WORDS,
+ "roomlightsuns", List.of("room", "light", "sun")),
+ new Example("turkic-capitalized-entry", "LANG tr\n", "1\nİnci\n",
+ "İNCİ", List.of("İnci")),
+ new Example("break-number-sign", "BREAK 1\nBREAK #\n" + PLURAL,
+ "2\nriver/A\nboat/A\n", "rivers#boats", List.of("river", "boat")),
+ new Example("flag-number-sign", "NEEDAFFIX #\n" + PLURAL,
+ "2\nfoo/#A\nbar/A\n", "foos", List.of("foo")),
+ new Example("flag-number-sign-virtual-stem", "NEEDAFFIX #\n" + PLURAL,
+ "2\nfoo/#A\nbar/A\n", "foo", List.of("foo")),
+ new Example("hidden-capital-mixed-case", PLURAL, "1\neBook/A\n", "EBOOKS", List.of("Ebook")),
+ new Example("hidden-capital-initial-capital", PLURAL, "1\neBook/A\n", "Ebooks", List.of("Ebooks")),
+ new Example("hidden-capital-all-caps-entry", "SFX S N 1\nSFX S 0 's .\n",
+ "1\nACME/S\n", "ACME'S", List.of("Acme")),
+ new Example("hidden-capital-listed-form-wins", PLURAL, "2\neBook/A\nEbook\n",
+ "EBOOKS", List.of("EBOOKS")),
+ new Example("hidden-capital-unflagged-all-caps", PLURAL, "1\nACME\n", "Acme", List.of("Acme")),
+ new Example("hidden-capital-not-in-compound", COMPOUND + PLURAL, "2\neBook/AC\ncover/C\n",
+ "EBOOKCOVER", List.of("EBOOKCOVER")),
+ new Example("hungarian-hyphen-moving-rule", HUNGARIAN_HYPHEN, HUNGARIAN_WORDS,
+ "folyómeder-őr", List.of("folyó", "meder", "őr")),
+ new Example("hungarian-hyphen-rule-needs-hyphen", HUNGARIAN_HYPHEN, HUNGARIAN_WORDS,
+ "folyómeder", List.of("folyómeder")),
+ new Example("hungarian-hyphen-rule-needs-language", HUNGARIAN_HYPHEN.replace("LANG hu", "LANG de"),
+ HUNGARIAN_WORDS, "folyómeder-őr", List.of("folyómeder-őr")),
+ new Example("hungarian-hyphen-rule-needs-flag", HUNGARIAN_HYPHEN,
+ "3\nfoly/S\nmeder/Y\nőr/Y\n", "folymeder-őr", List.of("folymeder-őr")),
+ new Example("hungarian-hyphen-rule-first-part-only", HUNGARIAN_HYPHEN, HUNGARIAN_WORDS,
+ "őr-folyómeder", List.of("őr-folyómeder")),
+ new Example("apostrophe-all-caps", "PFX P Y 1\nPFX P 0 d' .\n", "1\nOrient/P\n",
+ "D'ORIENT", List.of("Orient")),
+ new Example("apostrophe-capitalized", "PFX P Y 1\nPFX P 0 d' .\n", "1\nOrient/P\n",
+ "D'Orient", List.of("Orient")),
+ new Example("trailing-period", PLURAL, "2\ntext/A\netc.\n", "texts.", List.of("text")),
+ new Example("trailing-periods", PLURAL, "2\ntext/A\netc.\n", "texts...", List.of("text")),
+ new Example("trailing-period-entry", PLURAL, "2\ntext/A\netc.\n", "etc.", List.of("etc.")),
+ new Example("trailing-period-not-added", PLURAL, "2\ntext/A\netc.\n", "etc", List.of("etc")),
+ new Example("numeric-flag-maximum", "FLAG num\nSFX 65535 Y 1\nSFX 65535 0 s .\n",
+ "1\ndog/65535\n", "dogs", List.of("dog")),
+ // the manual's example of part stems here against the concatenated Hunspell stem
+ new Example("compound-part-stems", COMPOUND + PLURAL, "2\nriver/C\nboat/CA\n",
+ "riverboats", List.of("river", "boat")));
+ }
+
+ private static final String HUNGARIAN_HYPHEN = "LANG hu\nCOMPOUNDFLAG Y\nCOMPOUNDMIN 2\n"
+ + "COMPOUNDFORBIDFLAG !\nBREAK 1\nBREAK -\nSFX S Y 1\nSFX S 0 ó .\n";
+ private static final String HUNGARIAN_WORDS = "4\nfoly/S\nmeder/Y\nfolyó/F!\nőr/Y\n";
+ private static final String COMPOUND_FORBID = "COMPOUNDFLAG K\nCOMPOUNDPERMITFLAG L\n"
+ + "COMPOUNDFORBIDFLAG M\nSFX T Y 2\nSFX T 0 wood/LK .\nSFX T 0 ward/LK .\n";
+ private static final String COMPOUND_FORBID_WORDS = "3\nfire/T\nhouse/K\nfireward/M\n";
+ private static final String ONLY_IN_COMPOUND = COMPOUND + "ONLYINCOMPOUND Q\n"
+ + "COMPOUNDPERMITFLAG P\nSFX D Y 1\nSFX D 0 en/QP .\n";
+ private static final String ONLY_IN_COMPOUND_WORDS = "2\nlamp/C\nglass/CD\n";
+ private static final String FORBIDDEN_AFFIXED = "FORBIDDENWORD F\nCOMPOUNDFLAG C\n"
+ + "COMPOUNDMIN 1\nSFX S Y 1\nSFX S 0 s .\n";
+ private static final String FORBIDDEN_AFFIXED_WORDS = "4\nsun/CS\nlight/C\nroom/CS\nsunlightroom/FS\n";
+
+ /**
+ * Checks the Java implementation on one fixture.
+ *
+ * @param example The dictionary and assertion.
+ * @throws IOException Thrown if loading fails.
+ */
+ @ParameterizedTest(name = "{0}")
+ @MethodSource("examples")
+ void testStemming(Example example) throws IOException {
+ final HunspellDictionary dictionary = HunspellDictionary.load(
+ new ByteArrayInputStream(("SET UTF-8\n" + example.affix())
+ .getBytes(StandardCharsets.UTF_8)),
+ new ByteArrayInputStream(example.words().getBytes(StandardCharsets.UTF_8)));
+ final List<String> actual = new HunspellStemmer(dictionary).stemAll(example.input())
+ .stream().map(CharSequence::toString).toList();
+ Assertions.assertEquals(example.expected(), actual);
+ }
+
+ /**
+ * The stems each fixture was recorded with, from Hunspell as described in
+ * {@code dev/README-hunspell-dictionaries.md}. An empty reference result is recorded
+ * as the input itself; a fixture missing from the list fails.
+ *
+ * @param example The fixture.
+ * @return The recorded reference stems.
+ */
+ private static List<String> referenceStems(Example example) {
+ return switch (example.name()) {
+ case "keepcase-title", "keepcase-uppercase", "forbidden-warning" -> List.of("card");
+ case "complex-prefixes", "sharp-s-uppercase", "sharp-s-keepcase" -> List.of(example.input());
+ case "compound-rule", "compound-force-uppercase", "compound-force-uppercase-reject",
+ "compound-pattern-flags" -> List.of("river");
+ case "compound-root-count-accept" -> List.of("raincoat");
+ case "compound-replacement-check", "replacement-trailing-fields",
+ "replacement-morphology", "replacement-morphology-arrow", "replacement-morphology-star",
+ "replacement-morphology-star-unlisted" ->
+ List.of("rain");
+ case "compound-pattern-replacement", "compound-simplified-triple",
+ "compound-pattern-substitution-only", "compound-pattern-substitution-second",
+ "compound-only-suffix-inside" -> List.of(example.input());
+ case "compound-syllable-limit" -> List.of("rayme");
+ case "compound-duplicate-last-parts" -> List.of("foofoo");
+ case "compound-forbid-entry-other-suffix" -> List.of("firewood");
+ case "compound-replacement-inner" -> List.of("paperwall");
+ case "compound-replacement-inner-unaffected" -> List.of("paper");
+ case "forbidden-affixed-other-order" -> List.of("roomlightsun");
+ case "forbidden-affixed-blocks-compound" -> List.of("sunlightroom");
+ case "hidden-capital-initial-capital" -> List.of("Ebook");
+ case "mixed-case-initial-capital", "mixed-case-initial-capital-entry" ->
+ List.of(example.input());
+ case "break-default", "break-recursive", "break-start", "break-end", "break-custom",
+ "break-number-sign", "hungarian-hyphen-moving-rule", "apostrophe-all-caps",
+ "apostrophe-capitalized" -> List.of(example.input());
+ case "affixed-explicit-stem", "explicit-stem" -> List.of("foot");
+ case "break-disabled" -> List.of("rivers-boats");
+ case "break-internal-only" -> List.of("-rivers");
+ case "break-unknown-part" -> List.of("rivers-absent");
+ case "complex-prefix-requires-continuation" -> List.of("unredo");
+ case "complex-prefix-single-suffix" -> List.of("walkers");
+ case "compound-duplicate-reject" -> List.of("foobarbar");
+ case "compound-forbid-entry" -> List.of("firewardhouse");
+ case "compound-more-suffixes", "compound-part-stems", "compound-pattern",
+ "compound-rule-homonyms", "compound-rule-long", "compound-rule-numeric" ->
+ List.of("riverboat");
+ case "compound-only-suffix-at-end" -> List.of("lampglassen");
+ case "compound-root-count" -> List.of("raincoatrack");
+ case "compound-rule-optional" -> List.of("stoneboat");
+ case "compound-rule-order" -> List.of("boatriver");
+ case "compound-rule-star" -> List.of("riverstoneboat");
+ case "compound-syllable-limit-reject" -> List.of("raymefa");
+ case "derivational-suffix" -> List.of("kindness");
+ case "derivation-inflectional-prefix" -> List.of("management");
+ case "derivation-surface-prefix" -> List.of("mismanagement");
+ case "flag-number-sign", "flag-number-sign-virtual-stem" -> List.of("foo");
+ case "hidden-capital-all-caps-entry", "hidden-capital-unflagged-all-caps" -> List.of("Acme");
+ case "hidden-capital-listed-form-wins" -> List.of("EBOOKS");
+ case "hidden-capital-mixed-case" -> List.of("Ebook");
+ case "hidden-capital-not-in-compound" -> List.of("EBOOKCOVER");
+ case "hungarian-hyphen-rule-first-part-only" -> List.of("őr-folyómeder");
+ case "hungarian-hyphen-rule-needs-flag" -> List.of("folymeder-őr");
+ case "hungarian-hyphen-rule-needs-hyphen" -> List.of("folyómeder");
+ case "hungarian-hyphen-rule-needs-language" -> List.of("folyómeder-őr");
+ case "ignored-affix", "ignored-input-and-dictionary" -> List.of("pearl");
+ case "input-end-anchor" -> List.of("quartz");
+ case "input-ligature" -> List.of("field");
+ case "keepcase-exact" -> List.of("card");
+ case "longest-input-match" -> List.of("ax");
+ case "mixed-case-is-not-lowercase" -> List.of("cArds");
+ case "morphology-alias" -> List.of("goose");
+ case "numeric-flag-maximum" -> List.of("dog");
+ case "output-conversion" -> List.of("bär");
+ case "surface-prefix" -> List.of("prebase");
+ case "trailing-period-entry" -> List.of("etc.");
+ case "trailing-period-not-added" -> List.of("etc");
+ case "trailing-periods", "trailing-period" -> List.of("text");
+ case "turkic-capitalized-entry" -> List.of("İnci");
+ case "turkish-case" -> List.of("ılık");
+ case "uppercase-proper-name" -> List.of("Maren");
+ default -> throw new IllegalStateException("no recorded stems for " + example.name());
+ };
+ }
+
+ /**
+ * Whether the Hunspell spell checker accepted a fixture input when the fixtures
+ * were recorded.
+ *
+ * @param example The fixture.
+ * @return The recorded recognition outcome.
+ */
+ private static boolean referenceAccepts(Example example) {
+ return switch (example.name()) {
+ case "keepcase-title", "keepcase-uppercase", "forbidden-warning",
+ "mixed-case-is-not-lowercase", "complex-prefix-requires-continuation",
+ "complex-prefix-single-suffix", "compound-rule-order", "compound-rule-homonyms",
+ "compound-force-uppercase-reject", "compound-root-count", "compound-replacement-check",
+ "compound-pattern", "compound-syllable-limit-reject", "break-disabled",
+ "break-unknown-part", "break-internal-only", "replacement-trailing-fields",
+ "replacement-morphology", "replacement-morphology-arrow", "replacement-morphology-star",
+ "compound-duplicate-reject", "compound-forbid-entry", "compound-only-suffix-at-end",
+ "compound-replacement-inner", "forbidden-affixed-blocks-compound",
+ "flag-number-sign-virtual-stem", "hidden-capital-initial-capital",
+ "hidden-capital-listed-form-wins", "hidden-capital-unflagged-all-caps",
+ "hidden-capital-not-in-compound", "hungarian-hyphen-rule-needs-hyphen",
+ "hungarian-hyphen-rule-needs-language", "hungarian-hyphen-rule-needs-flag",
+ "hungarian-hyphen-rule-first-part-only", "trailing-period-not-added",
+ "turkic-capitalized-entry" -> false;
+ default -> true;
+ };
+ }
+
+ /**
+ * The fixtures whose recognition deliberately differs from the recorded reference
+ * spell-checker outcome, each with the manual's reason.
+ */
+ private static final Map<String, String> RECOGNITION_DEVIATIONS = Map.of(
+ "turkic-capitalized-entry", "the reference checker rejects what its analyzer stems");
+
+ /**
+ * The fixtures whose single stem differs from the recorded reference stem, each with
+ * the manual's reason. Fixtures with several part stems, and fixtures the reference
+ * checker rejects while its analyzer still stems them, are covered structurally.
+ */
+ private static final Map<String, String> STEM_DEVIATIONS = Map.ofEntries(
+ Map.entry("complex-prefixes", "Hunspell's analyzer reverses field text under COMPLEXPREFIXES"),
+ Map.entry("sharp-s-uppercase", "Hunspell's analyzer does not expand SS to a sharp s"),
+ Map.entry("sharp-s-keepcase", "Hunspell's analyzer does not expand SS to a sharp s"),
+ Map.entry("compound-pattern-replacement", "Hunspell's analyzer does not restore replaced junctions"),
+ Map.entry("compound-simplified-triple", "Hunspell's analyzer does not restore simplified triples"),
+ Map.entry("mixed-case-initial-capital", "Hunspell's analyzer keeps the initial capital"),
+ Map.entry("mixed-case-initial-capital-entry", "Hunspell's analyzer keeps the initial capital"),
+ Map.entry("apostrophe-all-caps", "Hunspell's analyzer does not undo an elided-article prefix"),
+ Map.entry("apostrophe-capitalized", "Hunspell's analyzer does not undo an elided-article prefix"),
+ Map.entry("hidden-capital-initial-capital", "Hunspell's analyzer ignores the capitalized input"),
+ Map.entry("break-start", "Hunspell's analyzer does not split at BREAK separators"),
+ Map.entry("break-end", "Hunspell's analyzer does not split at BREAK separators"));
+
+ /**
+ * Tests recognition against the recorded reference outcome. A fixture listed in
+ * {@link #RECOGNITION_DEVIATIONS} must differ, so a stale entry fails too.
+ *
+ * @param example The fixture.
+ * @throws IOException Thrown if loading fails.
+ */
+ @ParameterizedTest(name = "recognition {0}")
+ @MethodSource("examples")
+ void testRecognitionAgainstReference(Example example) throws IOException {
+ final HunspellDictionary dictionary = HunspellDictionary.load(
+ new ByteArrayInputStream(("SET UTF-8\n" + example.affix()).getBytes(StandardCharsets.UTF_8)),
+ new ByteArrayInputStream(example.words().getBytes(StandardCharsets.UTF_8)));
+ final boolean recognized = !new HunspellStemmer(dictionary).analyze(example.input()).isEmpty();
+ final String deviation = RECOGNITION_DEVIATIONS.get(example.name());
+ Assertions.assertEquals(deviation == null, recognized == referenceAccepts(example),
+ deviation == null ? "recognition differs from the reference" : deviation);
+ }
+
+ /**
+ * Tests stems against the recorded reference stems. Part stems of compounds and
+ * break forms differ by design from the concatenated reference stem, the reference
+ * analyzer stems some forms its checker rejects, and the remaining differences are
+ * listed in {@link #STEM_DEVIATIONS}; such a fixture must differ, so a stale entry
+ * fails too.
+ *
+ * @param example The fixture.
+ */
+ @ParameterizedTest(name = "stems {0}")
+ @MethodSource("examples")
+ void testStemsAgainstReference(Example example) {
+ final boolean same = example.expected().equals(referenceStems(example));
+ if (example.expected().size() > 1 || !referenceAccepts(example)) {
+ return;
+ }
+ final String deviation = STEM_DEVIATIONS.get(example.name());
+ Assertions.assertEquals(deviation == null, same,
+ deviation == null ? "stems differ from the reference: " + referenceStems(example) : deviation);
+ }
+}
diff --git a/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/stemmer/hunspell/HunspellCompletionTest.java b/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/stemmer/hunspell/HunspellCompletionTest.java
new file mode 100644
index 000000000..504a431fe
--- /dev/null
+++ b/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/stemmer/hunspell/HunspellCompletionTest.java
@@ -0,0 +1,410 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package opennlp.tools.stemmer.hunspell;
+
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeSet;
+import java.util.concurrent.Callable;
+import java.util.concurrent.Executors;
+import java.util.stream.Stream;
+
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
+
+import opennlp.tools.util.StringUtil;
+
+/**
+ * Compound and morphology fixtures written for this test, with the recognition and
+ * analyses recorded from Hunspell as described in {@code dev/README-hunspell-dictionaries.md}.
+ */
+class HunspellCompletionTest {
+
+ private static final String COMPOUND = "COMPOUNDFLAG C\nCOMPOUNDMIN 1\n";
+ private static final String HUNGARIAN = COMPOUND + "LANG hu\nCOMPOUNDWORDMAX 2\n";
+ private static final String THREE_WORDS = "3\nray/C\nme/C\nfa/C";
+ private static final String NEEDS_AFFIX_ZERO_RULES = "NEEDAFFIX N\n"
+ + "SFX P Y 1\nSFX P 0 0 . is:bare\nSFX Q Y 1\nSFX Q 0 0 . is:plain\n"
+ + "SFX R Y 2\nSFX R 0 0/NPQ . dp:none\nSFX R 0 ix/NPQ . dp:ix\n";
+
+ /**
+ * An original dictionary with expected recognition and stems.
+ *
+ * @param name The case identifier.
+ * @param affix The affix definitions.
+ * @param words The dictionary entries.
+ * @param input The text to analyze.
+ * @param stems The expected stems.
+ * @param accepted Whether Hunspell recognized the input.
+ */
+ private record Example(String name, String affix, String words, String input,
+ List<String> stems, boolean accepted) {
+ /** {@inheritDoc} */
+ @Override
+ public String toString() {
+ return name;
+ }
+ }
+
+ /** {@return the compound and morphology cases} */
+ private static Stream<Example> examples() {
+ return Stream.of(
+ new Example("legacy-lemma", "LEMMA_PRESENT L\nSFX A Y 1\nSFX A 0 s .\n",
+ "1\nfeet/AL st:foot\n", "feets", List.of("foot"), true),
+ new Example("syllable-c-reject", HUNGARIAN + "COMPOUNDSYLLABLE 5 aeiouy\n"
+ + "SYLLABLENUM klmc\nSFX c Y 1\nSFX c 0 s .\n",
+ THREE_WORDS + "c\n", "raymefas", List.of("raymefas"), false),
+ new Example("syllable-c-accept", HUNGARIAN + "COMPOUNDSYLLABLE 6 aeiouy\n"
+ + "SYLLABLENUM klmc\nSFX c Y 1\nSFX c 0 s .\n",
+ THREE_WORDS + "c\n", "raymefas", List.of("ray", "me", "fa"), true),
+ new Example("syllable-j-reject", HUNGARIAN + "COMPOUNDSYLLABLE 4 aeiouy\n"
+ + "SYLLABLENUM klmc\nSFX J Y 1\nSFX J 0 s .\n",
+ THREE_WORDS + "J\n", "raymefas", List.of("raymefas"), false),
+ new Example("syllable-i-with-j", HUNGARIAN + "COMPOUNDSYLLABLE 4 aeiouy\n"
+ + "SYLLABLENUM klmc\nSFX I Y 1\nSFX I 0 s .\n",
+ THREE_WORDS + "IJ\n", "raymefas", List.of("raymefas"), false),
+ new Example("syllable-i-without-j", HUNGARIAN + "COMPOUNDSYLLABLE 4 aeiouy\n"
+ + "SYLLABLENUM klmc\nSFX I Y 1\nSFX I 0 s .\n",
+ THREE_WORDS + "I\n", "raymefas", List.of("ray", "me", "fa"), true),
+ new Example("syllable-terminal-inflection", HUNGARIAN + "COMPOUNDSYLLABLE 4 aeiouy\n"
+ + "SFX A Y 1\nSFX A 0 a .\n", THREE_WORDS + "A\n",
+ "raymefaa", List.of("ray", "me", "fa"), true),
+ new Example("syllable-unaffixed-i", HUNGARIAN + "COMPOUNDSYLLABLE 3 aeiouy\n",
+ THREE_WORDS + "I\n", "raymefa", List.of("ray", "me", "fa"), true),
+ new Example("syllable-prefix-word-count", HUNGARIAN + "COMPOUNDSYLLABLE 2 aeiou\n"
+ + "PFX A Y 1\nPFX A 0 reco .\n", "2\nme/CA\nfa/C\n",
+ "recomefa", List.of("recomefa"), false),
+ new Example("multiple-triple-junctions", COMPOUND + "CHECKCOMPOUNDTRIPLE\nSIMPLIFIEDTRIPLE\n",
+ "2\nmill/C\nloom/C\n", "milloommilloom", List.of("mill", "loom"), true),
+ new Example("multiple-pattern-junctions", COMPOUND + "CHECKCOMPOUNDPATTERN 1\n"
+ + "CHECKCOMPOUNDPATTERN er b X\n", "2\nriver/C\nboat/C\n",
+ "rivXoatrivXoat", List.of("rivXoatrivXoat"), false),
+ new Example("compound-cross-product", COMPOUND + "COMPOUNDPERMITFLAG P\n"
+ + "PFX A Y 1\nPFX A 0 re .\nSFX B Y 1\nSFX B 0 s/P .\n",
+ "2\nriver/CAB\nboat/C\n", "reriversboat", List.of("river", "boat"), true),
+ new Example("compound-cross-product-no-permit", COMPOUND + "COMPOUNDPERMITFLAG P\n"
+ + "PFX A Y 1\nPFX A 0 re .\nSFX B Y 1\nSFX B 0 s .\n",
+ "2\nriver/CAB\nboat/C\n", "reriversboat", List.of("reriversboat"), false),
+ new Example("optional-affix-condition", "SFX A Y 1\nSFX A 0 s\n",
+ "1\ncard/A\n", "cards", List.of("card"), true),
+ new Example("optional-condition-with-morphology", "SFX A Y 1\nSFX A 0 s is:plural\n",
+ "1\ncard/A\n", "cards", List.of("cards"), false),
+ new Example("obsolete-compound-first", "COMPOUNDFIRST V\n",
+ "2\nriver/V\nboat/V\n", "riverboat", List.of("riverboat"), false),
+ new Example("obsolete-compound-last", "COMPOUNDLAST V\n",
+ "2\nriver/V\nboat/V\n", "riverboat", List.of("riverboat"), false),
+ new Example("obsolete-only-root", "ONLYROOT V\nSFX A Y 1\nSFX A 0 s .\n",
+ "1\ncard/VA\n", "cards", List.of("card"), true),
+ new Example("obsolete-hungarian-linking-vowel", "LANG hu\nHU_KOTOHANGZO V\n"
+ + "SFX A Y 1\nSFX A 0 s .\n", "1\ncard/VA\n", "cards", List.of("card"), true),
+ new Example("generation-option", "GENERATE 1\nSFX A Y 1\nSFX A 0 s .\n",
+ "1\ncard/A\n", "cards", List.of("card"), true),
+ new Example("compound-word-with-space", COMPOUND,
+ "3\nriver/C\nboat/C\nriver boat\n", "riverboat", List.of("riverboat"), false),
+ new Example("compound-affixed-word-with-space", COMPOUND + "SFX A Y 1\nSFX A 0 s .\n",
+ "3\nriver/C\nboat/CA\nriver boat/A\n", "riverboats", List.of("riverboats"), false),
+ new Example("compound-affixed-duplicate", COMPOUND + "CHECKCOMPOUNDDUP\n"
+ + "COMPOUNDPERMITFLAG P\nSFX A Y 1\nSFX A 0 s/P .\n",
+ "1\nriver/CA\n", "riversriver", List.of("riversriver"), false),
+ // the Hunspell spell checker rejects these forms while its analyzer stems them
+ new Example("turkish-capitalized-name", "LANG tr_TR\n", "1\nİpek\n",
+ "İPEK", List.of("İpek"), false),
+ new Example("azerbaijani-capitalized-name", "LANG az_AZ\n", "1\nİpek\n",
+ "İPEK", List.of("İpek"), false),
+ new Example("compound-pattern-suffix-flag", COMPOUND + "COMPOUNDPERMITFLAG P\n"
+ + "CHECKCOMPOUNDPATTERN 1\nCHECKCOMPOUNDPATTERN s/X b\n"
+ + "SFX A Y 1\nSFX A 0 s/PX .\n", "2\nriver/CA\nboat/C\n",
+ "riversboat", List.of("riversboat"), false),
+ new Example("compound-pattern-prefix-flag", COMPOUND + "COMPOUNDPERMITFLAG P\n"
+ + "CHECKCOMPOUNDPATTERN 1\nCHECKCOMPOUNDPATTERN r r/X\n"
+ + "PFX A Y 1\nPFX A 0 re/PX .\n", "2\nriver/C\nboat/CA\n",
+ "riverreboat", List.of("riverreboat"), false),
+ new Example("compound-cross-double-suffix", COMPOUND + "COMPOUNDPERMITFLAG P\n"
+ + "COMPOUNDMORESUFFIXES\nPFX R Y 1\nPFX R 0 re .\n"
+ + "SFX A Y 1\nSFX A 0 er/BP .\nSFX B Y 1\nSFX B 0 s/P .\n",
+ "2\nboat/CAR\nriver/C\n", "reboatersriver", List.of("reboatersriver"), false),
+ new Example("compound-cross-double-suffix-final", COMPOUND + "COMPOUNDPERMITFLAG P\n"
+ + "COMPOUNDMORESUFFIXES\nPFX R Y 1\nPFX R 0 re/P .\n"
+ + "SFX A Y 1\nSFX A 0 er/B .\nSFX B Y 1\nSFX B 0 s .\n",
+ "2\nboat/CAR\nriver/C\n", "riverreboaters", List.of("river", "boat"), true),
+ new Example("compound-complex-prefix", COMPOUND + "COMPLEXPREFIXES\n"
+ + "COMPOUNDMORESUFFIXES\nPFX A Y 1\nPFX A 0 re/B .\n"
+ + "PFX B Y 1\nPFX B 0 un .\n", "2\nriver/CA\nboat/C\n",
+ "unreriverboat", List.of("river", "boat"), true),
+ new Example("compound-complex-prefix-suffix", COMPOUND + "COMPLEXPREFIXES\n"
+ + "COMPOUNDMORESUFFIXES\nCOMPOUNDPERMITFLAG P\n"
+ + "PFX A Y 1\nPFX A 0 re/B .\nPFX B Y 1\nPFX B 0 un .\n"
+ + "SFX S Y 1\nSFX S 0 s/P .\n", "2\nriver/CAS\nboat/C\n",
+ "unreriversboat", List.of("river", "boat"), true),
+ // deviations listed in the manual: Unicode case mapping, the suggester-based
+ // rejection of multi-part compounds, and numeric tokens
+ new Example("dotted-capital-i", "", "1\ninvite\n", "İnvite", List.of("invite"), false),
+ new Example("dotted-capital-i-all-caps", "", "1\nİnci\n", "İNCİ", List.of("İnci"), true),
+ // KEEPCASE with CHECKSHARPS admits the SS spelling of an all-uppercase form only
+ new Example("keepcase-sharp-s-double-s", "CHECKSHARPS\nKEEPCASE k\n", "1\nfleiß/k\n",
+ "FLEISS", List.of("fleiß"), true),
+ new Example("keepcase-capital-sharp-s", "CHECKSHARPS\nKEEPCASE k\n", "1\nfleiß/k\n",
+ "FLEIẞ", List.of("FLEIẞ"), false),
+ new Example("keepcase-sharp-s-capitalized", "CHECKSHARPS\nKEEPCASE k\n", "1\nfleiß/k\n",
+ "Fleiß", List.of("fleiß"), true),
+ new Example("multi-part-compound-near-listed-word", "TRY abcdefghijklmnopqrstuvwxyz\n"
+ + "COMPOUNDFLAG x\n", "5\nsun/x\nset/x\nlamp/x\nbunset\nbunsetlamp\n",
+ "sunsetlamp", List.of("sun", "set", "lamp"), false),
+ new Example("numeric-token", "", "1\nfoo\n", "1.5", List.of("1.5"), true));
+ }
+
+ /**
+ * Tests Java stems under strict loading.
+ *
+ * @param example The fixture.
+ * @throws IOException Thrown if loading fails.
+ */
+ @ParameterizedTest(name = "{0}")
+ @MethodSource("examples")
+ void testStems(Example example) throws IOException {
+ final HunspellDictionary dictionary = HunspellDictionary.load(
+ new ByteArrayInputStream(("SET UTF-8\n" + example.affix()).getBytes(StandardCharsets.UTF_8)),
+ new ByteArrayInputStream(example.words().getBytes(StandardCharsets.UTF_8)));
+ Assertions.assertEquals(example.stems(), new HunspellStemmer(dictionary).stemAll(example.input()));
+ }
+
+ /**
+ * The fixtures whose recognition deliberately differs from the recorded reference
+ * spell-checker outcome, each with the manual's reason.
+ */
+ private static final Map<String, String> RECOGNITION_DEVIATIONS = Map.of(
+ "turkish-capitalized-name", "the reference checker rejects what its analyzer stems",
+ "azerbaijani-capitalized-name", "the reference checker rejects what its analyzer stems",
+ "dotted-capital-i", "a dotted capital I is lowercased outside the Turkic languages",
+ "multi-part-compound-near-listed-word", "the reference rejects it through its suggester",
+ "numeric-token", "numbers are accepted natively before any lookup");
+
+ /**
+ * Tests recognition against the outcome recorded from the Hunspell spell checker.
+ * A fixture listed in {@link #RECOGNITION_DEVIATIONS} must differ, so a stale entry
+ * fails too.
+ *
+ * @param example The fixture.
+ * @throws IOException Thrown if loading fails.
+ */
+ @ParameterizedTest(name = "recognition {0}")
+ @MethodSource("examples")
+ void testRecognitionAgainstReference(Example example) throws IOException {
+ final HunspellDictionary dictionary = HunspellDictionary.load(
+ new ByteArrayInputStream(("SET UTF-8\n" + example.affix()).getBytes(StandardCharsets.UTF_8)),
+ new ByteArrayInputStream(example.words().getBytes(StandardCharsets.UTF_8)));
+ final boolean recognized = !new HunspellStemmer(dictionary).analyze(example.input()).isEmpty();
+ final String deviation = RECOGNITION_DEVIATIONS.get(example.name());
+ Assertions.assertEquals(deviation == null, recognized == example.accepted(),
+ deviation == null ? "recognition differs from the reference" : deviation);
+ }
+
+ /**
+ * Checks the package-private analysis operation, which preserves entry and affix
+ * fields and answers an immutable list.
+ *
+ * @throws Exception Thrown if loading or analysis fails.
+ */
+ @Test
+ void testMorphologicalAnalysis() throws Exception {
+ final HunspellDictionary dictionary = HunspellDictionary.load(
+ new ByteArrayInputStream(("SFX A Y 1\nSFX A 0 s . is:plural\n")
+ .getBytes(StandardCharsets.UTF_8)),
+ new ByteArrayInputStream("2\ncard/A po:noun\ncard/A po:verb\n".getBytes(StandardCharsets.UTF_8)));
+ final HunspellStemmer stemmer = new HunspellStemmer(dictionary);
+ Assertions.assertEquals(List.of("st:card po:noun is:plural", "st:card po:verb is:plural"),
+ stemmer.analyze("cards"));
+ Assertions.assertEquals(List.of(), stemmer.analyze("unlisted"));
+ Assertions.assertThrows(IllegalArgumentException.class, () -> stemmer.analyze(null));
+ Assertions.assertEquals(List.of(), stemmer.analyze(""));
+ Assertions.assertThrows(UnsupportedOperationException.class, () -> stemmer.analyze("cards").clear());
+ }
+
+ /**
+ * Checks the documented maximum number of CHECKSHARPS case variants.
+ *
+ * @throws IOException Thrown if fixture loading fails.
+ */
+ @Test
+ void testSharpVariantLimit() throws IOException {
+ final StringBuilder words = new StringBuilder("129\nSSSSSSSSSSSS\n");
+ for (int mask = 0; mask < 64; mask++) {
+ final StringBuilder entry = new StringBuilder();
+ for (int bit = 0; bit < 6; bit++) {
+ entry.append((mask & (1 << bit)) == 0 ? "ss" : "ß");
+ }
+ words.append(entry).append('\n');
+ words.append(StringUtil.toUpperCase(entry.substring(0, 1))).append(entry.substring(1)).append('\n');
+ }
+ final HunspellStemmer stemmer = new HunspellStemmer(HunspellDictionary.load(
+ new ByteArrayInputStream("CHECKSHARPS\n".getBytes(StandardCharsets.UTF_8)),
+ new ByteArrayInputStream(words.toString().getBytes(StandardCharsets.UTF_8))));
+ Assertions.assertTrue(stemmer.stemAll("SSSSSSSSSSSS").size() <= 64);
+ }
+
+ /**
+ * Checks that the flag-set lists a lookup answers are immutable and shared across
+ * concurrent stemming.
+ *
+ * @throws Exception Thrown if fixture loading or a worker fails.
+ */
+ @Test
+ void testSharedMorphologyAndImmutableFlags() throws Exception {
+ final HunspellDictionary dictionary = HunspellDictionary.load(
+ new ByteArrayInputStream("SFX A Y 1\nSFX A 0 s . is:plural\n".getBytes(StandardCharsets.UTF_8)),
+ new ByteArrayInputStream("2\ncard/A po:noun\ncard/A po:verb\n".getBytes(StandardCharsets.UTF_8)));
+ Assertions.assertThrows(UnsupportedOperationException.class, () -> dictionary.lookup("card").clear());
+ Assertions.assertThrows(UnsupportedOperationException.class,
+ () -> dictionary.lookup("card", true).add(new int[0]));
+ final HunspellStemmer stemmer = new HunspellStemmer(dictionary);
+ final List<String> expected = List.of("st:card po:noun is:plural", "st:card po:verb is:plural");
+ try (var executor = Executors.newFixedThreadPool(4)) {
+ final List<Callable<List<String>>> tasks = new ArrayList<>();
+ for (int i = 0; i < 64; i++) {
+ tasks.add(() -> {
+ Assertions.assertEquals(List.of("card"), stemmer.stemAll("cards"));
+ Assertions.assertEquals(List.of(), stemmer.analyze("unlisted"));
+ return stemmer.analyze("cards");
+ });
+ }
+ for (var result : executor.invokeAll(tasks)) {
+ Assertions.assertEquals(expected, result.get());
+ }
+ }
+ }
+
+ /**
+ * Supplies morphology recorded from Hunspell.
+ *
+ * @return Affix content, dictionary content, input, and expected analysis.
+ */
+ private static Stream<String[]> morphology() {
+ return Stream.of(
+ new String[] {"SFX A Y 1\nSFX A 0 s .\n", "1\ncard/A\n", "cards", "st:card fl:A"},
+ new String[] {"SFX A Y 1\nSFX A 0 s . is:plural\n", "1\ncard/A po:noun\n",
+ "cards", "st:card po:noun is:plural"},
+ new String[] {"AM 2\nAM st:foot ts:present\nAM is:plural\nSFX A Y 1\nSFX A 0 s . 2\n",
+ "1\nfeet/A\t1\n", "feets", "st:foot ts:present is:plural"},
+ new String[] {"PFX B Y 1\nPFX B 0 re . dp:again\nSFX A Y 1\nSFX A 0 s . is:plural\n",
+ "1\ngo/AB po:verb\n", "regos", "dp:again st:go po:verb is:plural"},
+ new String[] {COMPOUND + "SFX A Y 1\nSFX A 0 s . is:plural\n",
+ "2\nriver/C po:noun\nboat/CA po:noun\n", "riverboats",
+ "pa:river st:river po:noun pa:boats st:boat po:noun is:plural"},
+ new String[] {"SFX A Y 1\nSFX A 0 s . is:plural\n", "1\ncard/A custom\n",
+ "cards", "st:card is:plural"},
+ new String[] {"NEEDAFFIX X\nSFX A Y 1\nSFX A 0 0 .\n", "1\nfoo/XA\n", "foo", "st:foo fl:A"},
+ new String[] {"PFX C Y 1\nPFX C 0 pre .\n", "1\nfoo/C\n", "prefoo", "pre st:foo fl:C"},
+ new String[] {"PFX C Y 1\nPFX C 0 pre .\n", "1\nfoo/C po:noun\n", "prefoo", "pre st:foo po:noun"},
+ new String[] {"PFX B Y 1\nPFX B 0 re .\nSFX A Y 1\nSFX A 0 s . is:plural\n",
+ "1\ngo/AB po:verb\n", "regos", "fl:B st:go po:verb is:plural"},
+ new String[] {"PFX B Y 1\nPFX B 0 re . dp:again\nSFX A Y 1\nSFX A 0 s .\n",
+ "1\ngo/AB\n", "regos", "dp:again st:go fl:A"},
+ new String[] {"SFX A Y 1\nSFX A 0 er/B .\nSFX B Y 1\nSFX B 0 s .\n",
+ "1\nwalk/A po:verb\n", "walkers", "st:walk po:verb fl:A fl:B"},
+ new String[] {COMPOUND, "2\nfoo/C id:1\nbar/C\n", "foobar", "pa:foo st:foo id:1 pa:bar"},
+ new String[] {COMPOUND, "3\nfoo/C\nbar/C id:2\nbaz/C\n", "foobarbaz",
+ "pa:foo st:foo pa:bar st:bar id:2 pa:baz"},
+ new String[] {COMPOUND + "SFX A Y 1\nSFX A 0 s .\n", "2\nfoo/C id:1\nbar/CA\n", "foobars",
+ "pa:foo st:foo id:1 pa:bars st:bar fl:A"},
+ new String[] {"SFX A Y 1\nSFX A 0 s .\n", "1\neBook/A po:noun\n", "EBOOKS", "st:Ebook po:noun fl:A"});
+ }
+
+ /**
+ * Analyses that include a rule adding and removing no material, in reference order.
+ *
+ * @return Affix content, word list, input, and every expected analysis.
+ */
+ private static Stream<Arguments> zeroAffixAnalyses() {
+ return Stream.of(
+ Arguments.of("SFX A Y 1\nSFX A 0 0 . is:zero\n", "1\nbar/A\n", "bar",
+ List.of("st:bar", "st:bar is:zero")),
+ Arguments.of("PFX A Y 1\nPFX A 0 0 . dp:zero\n", "1\nbar/A\n", "bar",
+ List.of("st:bar", "dp:zero st:bar fl:A")),
+ Arguments.of(NEEDS_AFFIX_ZERO_RULES, "1\nlumen/NPQR po:noun\n", "lumen",
+ List.of("st:lumen po:noun is:bare", "st:lumen po:noun is:plain",
+ "st:lumen po:noun dp:none is:bare", "st:lumen po:noun dp:none is:plain")),
+ Arguments.of(NEEDS_AFFIX_ZERO_RULES, "1\nlumen/NPQR po:noun\n", "lumenix",
+ List.of("st:lumen po:noun dp:ix is:bare", "st:lumen po:noun dp:ix is:plain")));
+ }
+
+ /**
+ * Tests that a rule without material is undone on its own and inside a continuation.
+ *
+ * @param affix The affix content.
+ * @param words The word list.
+ * @param input The analyzed word.
+ * @param expected The distinct analyses, as recorded from Hunspell's analyzer.
+ * @throws IOException Thrown if loading fails.
+ */
+ @ParameterizedTest
+ @MethodSource("zeroAffixAnalyses")
+ void testZeroAffixAnalyses(String affix, String words, String input, List<String> expected)
+ throws IOException {
+ final HunspellStemmer stemmer = new HunspellStemmer(HunspellDictionary.load(
+ new ByteArrayInputStream(affix.getBytes(StandardCharsets.UTF_8)),
+ new ByteArrayInputStream(words.getBytes(StandardCharsets.UTF_8))));
+ Assertions.assertEquals(new TreeSet<>(expected), new TreeSet<>(stemmer.analyze(input)));
+ }
+
+ /**
+ * Tests that only the first listed homonym decides whether a spelling is forbidden.
+ * The Hunspell spell checker accepts {@code reed} with the valid homonym listed first
+ * and rejects it with the forbidden homonym listed first.
+ *
+ * @throws IOException Thrown if loading fails.
+ */
+ @Test
+ void testForbiddenFirstHomonym() throws IOException {
+ final String affix = "FORBIDDENWORD F\nCOMPOUNDFLAG K\nCOMPOUNDMIN 1\n";
+ final HunspellStemmer allowed = new HunspellStemmer(HunspellDictionary.load(
+ new ByteArrayInputStream(affix.getBytes(StandardCharsets.UTF_8)),
+ new ByteArrayInputStream("2\nreed/T\nreed/KF\n".getBytes(StandardCharsets.UTF_8))));
+ Assertions.assertEquals(List.of("st:reed"), allowed.analyze("reed"));
+ Assertions.assertEquals(List.of(), allowed.analyze("reedreed"));
+ final HunspellStemmer forbidden = new HunspellStemmer(HunspellDictionary.load(
+ new ByteArrayInputStream(affix.getBytes(StandardCharsets.UTF_8)),
+ new ByteArrayInputStream("2\nreed/KF\nreed/T\n".getBytes(StandardCharsets.UTF_8))));
+ Assertions.assertEquals(List.of(), forbidden.analyze("reed"));
+ }
+
+ /**
+ * Tests analyses against the field text recorded from Hunspell's analyzer, with
+ * separator whitespace normalized to single spaces.
+ *
+ * @param affix The affix content.
+ * @param words The word list.
+ * @param input The analyzed word.
+ * @param expected The recorded analysis.
+ * @throws IOException Thrown if loading fails.
+ */
+ @ParameterizedTest
+ @MethodSource("morphology")
+ void testMorphologyFields(String affix, String words, String input, String expected) throws IOException {
+ final HunspellStemmer stemmer = new HunspellStemmer(HunspellDictionary.load(
+ new ByteArrayInputStream(affix.getBytes(StandardCharsets.UTF_8)),
+ new ByteArrayInputStream(words.getBytes(StandardCharsets.UTF_8))));
+ Assertions.assertEquals(List.of(expected), stemmer.analyze(input));
+ }
+}
diff --git a/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/stemmer/hunspell/HunspellConversionTest.java b/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/stemmer/hunspell/HunspellConversionTest.java
new file mode 100644
index 000000000..461931839
--- /dev/null
+++ b/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/stemmer/hunspell/HunspellConversionTest.java
@@ -0,0 +1,65 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+
+package opennlp.tools.stemmer.hunspell;
+
+import java.io.IOException;
+
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.CsvSource;
+import org.junit.jupiter.params.provider.ValueSource;
+
+/**
+ * Tests the parsing of the conversion tables, apart from the stemmer.
+ */
+class HunspellConversionTest {
+
+ /**
+ * Verifies that an underscore inside a pattern is a space in each table that
+ * {@link HunspellConversion} parses, after the anchors are removed: the REP entry
+ * {@code c_oat coat} offers {@code coat} for the text {@code c oat}.
+ *
+ * @param pattern The REP pattern as written in the affix file.
+ * @throws IOException Thrown if the table fails to parse.
+ */
+ @ParameterizedTest
+ @ValueSource(strings = {"c_oat", "^c_oat", "c_oat$", "^c_oat$"})
+ void testReplacementPatternUnderscoreIsSpace(String pattern) throws IOException {
+ final HunspellConversion table = HunspellConversion.parse(
+ new String[][] {{"REP", "1"}, {"REP", pattern, "coat"}}, HunspellDictionary.REPLACEMENT_TAG);
+ Assertions.assertTrue(table.anyReplacement("c oat", "coat"::equals));
+ Assertions.assertFalse(table.anyReplacement("c_oat", "coat"::equals));
+ }
+
+ /**
+ * Rejects a conversion pattern that is anchors only, since it has no text to match.
+ *
+ * @param directive The table.
+ * @param pattern The pattern as written.
+ */
+ @ParameterizedTest
+ @CsvSource({"ICONV, _", "ICONV, __", "OCONV, _", "REP, ^", "REP, ^$"})
+ void testAnchorsOnlyPatternIsRejected(String directive, String pattern) {
+ final IOException error = Assertions.assertThrows(IOException.class,
+ () -> HunspellConversion.parse(
+ new String[][] {{directive, "1"}, {directive, pattern, "x"}}, directive));
+ Assertions.assertTrue(error.getMessage().contains("empty " + directive + " pattern"),
+ error.getMessage());
+ }
+}
diff --git a/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/stemmer/hunspell/HunspellDictionaryLoadTest.java b/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/stemmer/hunspell/HunspellDictionaryLoadTest.java
new file mode 100644
index 000000000..2a1685531
--- /dev/null
+++ b/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/stemmer/hunspell/HunspellDictionaryLoadTest.java
@@ -0,0 +1,538 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package opennlp.tools.stemmer.hunspell;
+
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.Arrays;
+import java.util.List;
+import java.util.stream.Stream;
+
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.EnumSource;
+import org.junit.jupiter.params.provider.MethodSource;
+import org.junit.jupiter.params.provider.ValueSource;
+
+import opennlp.tools.stemmer.hunspell.HunspellDictionary.LoadMode;
+import opennlp.tools.stemmer.hunspell.HunspellDictionary.UnsupportedDirective;
+
+/** Tests the loading policy with project-authored affix and dictionary content. */
+class HunspellDictionaryLoadTest {
+
+ private static final String WORDS = "1\ndog/A\n";
+ private static final String RULES = "SFX A Y 1\nSFX A 0 s .\n";
+
+ @TempDir
+ private Path directory;
+
+ /**
+ * Rejects directives with behavior not implemented by the stemmer.
+ *
+ * @param line An unsupported directive with arguments.
+ */
+ @ParameterizedTest
+ @ValueSource(strings = {
+ "UNSUPPORTED_CONVERSION 1", "UNSUPPORTED_CASE k", "UNSUPPORTED_SYLLABLES ABC",
+ "UNSUPPORTED_LEMMA L", "UNRECOGNIZED value"
+ })
+ void testUnsupportedDirectiveFailsByDefault(String line) {
+ final IOException error = Assertions.assertThrows(IOException.class,
+ () -> HunspellDictionary.load(stream("SET UTF-8\n" + line + "\n" + RULES),
+ stream(WORDS)));
+ final int separator = line.indexOf(' ');
+ final String directive = separator < 0 ? line : line.substring(0, separator);
+ Assertions.assertTrue(error.getMessage().contains(directive));
+ Assertions.assertTrue(error.getMessage().contains("affix stream"));
+ Assertions.assertTrue(error.getMessage().contains("line 2"));
+ }
+
+ /**
+ * Counts logical lines with each supported line separator.
+ *
+ * @param separator A supported line separator.
+ */
+ @ParameterizedTest
+ @ValueSource(strings = {"\n", "\r\n", "\r"})
+ void testUnsupportedDirectiveLineNumber(String separator) {
+ final String affix = "# comment" + separator + separator + " UNSUPPORTED_CASE K";
+ final IOException error = Assertions.assertThrows(IOException.class,
+ () -> HunspellDictionary.load(stream(affix), stream(WORDS)));
+ Assertions.assertTrue(error.getMessage().contains("UNSUPPORTED_CASE"));
+ Assertions.assertTrue(error.getMessage().contains("line 3"));
+ }
+
+ /**
+ * Reads the affix file and the word list with each supported line separator, so a
+ * file with carriage return line endings loads like one with line feeds.
+ *
+ * @param separator A supported line separator.
+ * @throws IOException Thrown if loading fails.
+ */
+ @ParameterizedTest
+ @ValueSource(strings = {"\n", "\r\n", "\r"})
+ void testLineSeparatorsInAffixAndWordList(String separator) throws IOException {
+ final String affix = String.join(separator, "SFX A Y 1", "SFX A 0 s .", "");
+ final String words = String.join(separator, "1", "dog/A", "");
+ final HunspellDictionary dictionary = HunspellDictionary.load(stream(affix), stream(words));
+ Assertions.assertEquals("dog", new HunspellStemmer(dictionary).stem("dogs").toString());
+ }
+
+ /**
+ * Reports the line of a malformed supported directive with each line separator.
+ *
+ * @param separator A supported line separator.
+ */
+ @ParameterizedTest
+ @ValueSource(strings = {"\n", "\r\n", "\r"})
+ void testMalformedDirectiveLineNumber(String separator) {
+ final String affix = String.join(separator, "# comment", "", "IGNORE", "");
+ final IOException error = Assertions.assertThrows(IOException.class,
+ () -> HunspellDictionary.load(stream(affix), stream(WORDS)));
+ Assertions.assertTrue(error.getMessage().contains("IGNORE"), error.getMessage());
+ Assertions.assertTrue(error.getMessage().contains("line 3"), error.getMessage());
+ }
+
+ /** Rejects an unsupported directive immediately after a UTF-8 byte-order mark. */
+ @Test
+ void testByteOrderMarkDoesNotHideUnsupportedDirective() {
+ final IOException error = Assertions.assertThrows(IOException.class,
+ () -> HunspellDictionary.load(stream("\uFEFFUNSUPPORTED_CONVERSION 1\n"), stream(WORDS)));
+ Assertions.assertTrue(error.getMessage().contains("UNSUPPORTED_CONVERSION"));
+ Assertions.assertTrue(error.getMessage().contains("line 1"));
+ }
+
+ /**
+ * Identifies the source file when path-based loading rejects a directive.
+ *
+ * @throws IOException Thrown if writing a fixture fails.
+ */
+ @Test
+ void testPathErrorIdentifiesAffixFile() throws IOException {
+ final Path affix = directory.resolve("sample.aff");
+ final Path words = directory.resolve("sample.dic");
+ Files.writeString(affix, "SET UTF-8\nUNSUPPORTED_CASE K\n");
+ Files.writeString(words, WORDS);
+ final IOException error = Assertions.assertThrows(IOException.class,
+ () -> HunspellDictionary.load(affix, words));
+ Assertions.assertTrue(error.getMessage().contains(affix.toString()));
+ Assertions.assertTrue(error.getMessage().contains("line 2"));
+ }
+
+ /**
+ * Loads settings outside the stemmer's operations without a diagnostic.
+ *
+ * @param setting A metadata or suggestion setting.
+ * @throws IOException Thrown if loading fails.
+ */
+ @ParameterizedTest
+ @ValueSource(strings = {
+ "NAME Example", "HOME https://example.org", "VERSION 1", "KEY abc|def",
+ "TRY abc", "REP 1\nREP ph f", "MAP 1\nMAP aá", "PHONE 1\nPHONE ph f",
+ "NOSUGGEST N", "MAXCPDSUGS 0", "MAXNGRAMSUGS 0", "MAXDIFF 5",
+ "ONLYMAXDIFF", "NOSPLITSUGS", "SUGSWITHDOTS", "WARN W",
+ "SUBSTANDARD S", "WORDCHARS -", "NONGRAMSUGGEST N", "CHECKNUM"
+ })
+ void testSettingsOutsideStemmingDoNotPreventStrictLoading(String setting)
+ throws IOException {
+ final HunspellDictionary dictionary = HunspellDictionary.load(
+ stream(setting + "\n" + RULES), stream(WORDS));
+ Assertions.assertEquals("dog", new HunspellStemmer(dictionary).stem("dogs").toString());
+ Assertions.assertTrue(dictionary.getUnsupportedDirectives().isEmpty());
+ }
+
+ /**
+ * Accepts under strict loading every directive name Hunspell's affix and dictionary
+ * parsers read, as a stemming setting or as one the stemmer ignores. The value is a
+ * placeholder, so a directive may still fail on its value, but never as unsupported.
+ *
+ * @param directive A directive name Hunspell reads.
+ */
+ @ParameterizedTest
+ @ValueSource(strings = {
+ "AF", "AM", "BREAK", "CHECKCOMPOUNDCASE", "CHECKCOMPOUNDDUP", "CHECKCOMPOUNDPATTERN",
+ "CHECKCOMPOUNDREP", "CHECKCOMPOUNDTRIPLE", "CHECKNUM", "CHECKSHARPS", "CIRCUMFIX",
+ "COMPLEXPREFIXES", "COMPOUNDBEGIN", "COMPOUNDEND", "COMPOUNDFLAG", "COMPOUNDFORBIDFLAG",
+ "COMPOUNDMIDDLE", "COMPOUNDMIN", "COMPOUNDMORESUFFIXES", "COMPOUNDPERMITFLAG",
+ "COMPOUNDROOT", "COMPOUNDRULE", "COMPOUNDSYLLABLE", "COMPOUNDWORDMAX", "FLAG",
+ "FORBIDDENWORD", "FORBIDWARN", "FORCEUCASE", "FULLSTRIP", "ICONV", "IGNORE", "KEEPCASE",
+ "KEY", "LANG", "LEMMA_PRESENT", "MAP", "MAXCPDSUGS", "MAXDIFF", "MAXNGRAMSUGS",
+ "NEEDAFFIX", "NONGRAMSUGGEST", "NOSPLITSUGS", "NOSUGGEST", "OCONV", "ONLYINCOMPOUND",
+ "ONLYMAXDIFF", "PHONE", "PSEUDOROOT", "REP", "SET", "SIMPLIFIEDTRIPLE", "SUBSTANDARD",
+ "SUGSWITHDOTS", "SYLLABLENUM", "TRY", "VERSION", "WARN", "WORDCHARS"
+ })
+ void testHunspellDirectivesAreNotUnsupported(String directive) {
+ try {
+ HunspellDictionary.load(stream(directive + " X\n" + RULES), stream(WORDS));
+ } catch (IOException e) {
+ Assertions.assertFalse(e.getMessage().contains("unsupported affix directive"),
+ e.getMessage());
+ }
+ }
+
+ /**
+ * Keeps a number sign that is a directive value, which the Hunspell format allows
+ * for flags, separators, and affix material.
+ *
+ * @param affix Affix content in which {@code #} is a value.
+ * @param input The stemmed word.
+ * @param expected The stem.
+ * @throws IOException Thrown if loading fails.
+ */
+ @ParameterizedTest
+ @MethodSource("numberSignValues")
+ void testNumberSignValuesAreKept(String affix, String words, String input, String expected)
+ throws IOException {
+ final HunspellDictionary dictionary = HunspellDictionary.load(stream(affix), stream(words));
+ Assertions.assertEquals(expected, new HunspellStemmer(dictionary).stem(input).toString());
+ }
+
+ /**
+ * Directive values consisting of a number sign.
+ *
+ * @return Affix content, word list, input, and expected stem.
+ */
+ private static Stream<Arguments> numberSignValues() {
+ return Stream.of(
+ Arguments.of("BREAK 1\nBREAK #\n" + RULES, WORDS, "dogs#dogs", "dog"),
+ Arguments.of("NEEDAFFIX #\n" + RULES, "1\ndog/#A\n", "dogs", "dog"),
+ Arguments.of("FLAG long\nAF 1\nAF #A\nSFX #A Y 1\nSFX #A 0 s .\n", "1\ndog/1\n", "dogs", "dog"),
+ Arguments.of("SFX A Y 1\nSFX A 0 # .\n", WORDS, "dog#", "dog"),
+ Arguments.of("SFX A Y 1\nSFX A # s [#]\n", "1\ndog#/A\n", "dogs", "dog#"));
+ }
+
+ /**
+ * Ignores trailing comments after the fields a directive consumes, as the reference
+ * implementation ignores those fields.
+ *
+ * @param affix Affix content with a trailing comment.
+ * @throws IOException Thrown if loading fails.
+ */
+ @ParameterizedTest
+ @ValueSource(strings = {
+ "COMPOUNDMIN 3 # comment\nSFX A Y 1 # comment\nSFX A 0 s . # comment",
+ "NEEDAFFIX X # comment\nSFX A Y 1\nSFX A 0 s .",
+ "SET UTF-8 # comment\nFLAG UTF-8 # comment\nSFX A Y 1\nSFX A 0 s .",
+ "AF 1\nAF A # comment\nSFX A Y 1\nSFX A 0 s ."
+ })
+ void testTrailingCommentsAreIgnored(String affix) throws IOException {
+ final String words = affix.startsWith("AF") ? "1\ndog/1\n" : WORDS;
+ final HunspellDictionary dictionary = HunspellDictionary.load(stream(affix), stream(words));
+ Assertions.assertEquals("dog", new HunspellStemmer(dictionary).stem("dogs").toString());
+ }
+
+ /**
+ * Reports the first location for each skipped directive in source order.
+ *
+ * @param separator A supported line separator.
+ * @throws IOException Thrown if partial loading fails.
+ */
+ @ParameterizedTest
+ @ValueSource(strings = {"\n", "\r\n", "\r"})
+ void testPartialLoadingReportsFirstOccurrences(String separator) throws IOException {
+ final String affix = String.join(separator, "UNSUPPORTED_CONVERSION 1", "UNSUPPORTED_CONVERSION a b",
+ "UNSUPPORTED_CASE K", "UNRECOGNIZED 1", "UNRECOGNIZED x", RULES);
+ final HunspellDictionary dictionary = HunspellDictionary.load(
+ stream(affix), stream(WORDS), LoadMode.ALLOW_PARTIAL);
+ final List<UnsupportedDirective> diagnostics = dictionary.getUnsupportedDirectives();
+ Assertions.assertEquals(List.of(
+ new UnsupportedDirective("UNSUPPORTED_CONVERSION", "affix stream", 1),
+ new UnsupportedDirective("UNSUPPORTED_CASE", "affix stream", 3),
+ new UnsupportedDirective("UNRECOGNIZED", "affix stream", 4)), diagnostics);
+ Assertions.assertThrows(UnsupportedOperationException.class, diagnostics::clear);
+ Assertions.assertEquals("dog", new HunspellStemmer(dictionary).stem("dogs").toString());
+ }
+
+ /**
+ * Includes the affix path in partial-loading diagnostics.
+ *
+ * @throws IOException Thrown if writing or loading fixtures fails.
+ */
+ @Test
+ void testPartialLoadingReportsFilePath() throws IOException {
+ final Path affix = directory.resolve("partial.aff");
+ final Path words = directory.resolve("partial.dic");
+ Files.writeString(affix, "SET UTF-8\nUNSUPPORTED_CASE K\n" + RULES);
+ Files.writeString(words, WORDS);
+ final HunspellDictionary dictionary = HunspellDictionary.load(
+ affix, words, LoadMode.ALLOW_PARTIAL);
+ Assertions.assertEquals(List.of(new UnsupportedDirective(
+ "UNSUPPORTED_CASE", affix.toString(), 2)), dictionary.getUnsupportedDirectives());
+ Assertions.assertEquals("dog", new HunspellStemmer(dictionary).stem("dogs").toString());
+ }
+
+ /**
+ * Rejects supported malformed content under either loading policy.
+ *
+ * @param malformed Malformed affix content.
+ */
+ @ParameterizedTest
+ @ValueSource(strings = {"AF -1\n", "FLAG num\nSFX 65536 Y 0\n",
+ "COMPOUNDMIN -1\n", "SFX A Y 2\nSFX A 0 s .\n", "FLAG short\n",
+ "ICONV 1\n", "ICONV -1\n", "ICONV 1\nICONV a b\nICONV c d\n",
+ "OCONV 1\nOCONV _ x\n", "AM 1\n", "AM -1\n",
+ "COMPOUNDRULE 1\n", "COMPOUNDRULE 1\nCOMPOUNDRULE *A\n",
+ "COMPOUNDRULE 1\nCOMPOUNDRULE (\n", "CHECKCOMPOUNDPATTERN 1\n",
+ "CHECKCOMPOUNDPATTERN -1\n", "BREAK 1\n", "BREAK -1\n", "BREAK 1\nBREAK ^\n",
+ "IGNORE\n", "LANG\n", "COMPOUNDSYLLABLE -1 ae\n", "KEEPCASE\n",
+ "SYLLABLENUM\n", "LEMMA_PRESENT\n", "LEMMA_PRESENT AB\n",
+ "AM 1 extra\nAM po:noun\n", "CHECKCOMPOUNDPATTERN 0 extra\n"})
+ void testPartialLoadingDoesNotIgnoreMalformedRules(String malformed) {
+ Assertions.assertThrows(IOException.class, () -> HunspellDictionary.load(
+ stream(malformed), stream("1\ndog\n"), LoadMode.STRICT));
+ Assertions.assertThrows(IOException.class, () -> HunspellDictionary.load(
+ stream("UNSUPPORTED_CASE K\n" + malformed), stream("1\ndog\n"), LoadMode.ALLOW_PARTIAL));
+ }
+
+ /**
+ * Rejects invalid AM references in entries and affix fields under either policy.
+ *
+ * @param reference The invalid reference.
+ */
+ @ParameterizedTest
+ @ValueSource(strings = {"-1", "0", "2", "invalid", "1 1"})
+ void testInvalidMorphologyAliases(String reference) {
+ final String aliases = "AM 1\nAM po:noun\n";
+ for (LoadMode mode : LoadMode.values()) {
+ Assertions.assertThrows(IOException.class, () -> HunspellDictionary.load(
+ stream(aliases), stream("1\ndog\t" + reference + "\n"), mode));
+ Assertions.assertThrows(IOException.class, () -> HunspellDictionary.load(
+ stream(aliases + "SFX A Y 1\nSFX A 0 s . " + reference + "\n"), stream(WORDS), mode));
+ }
+ }
+
+ /**
+ * Rejects malformed text in partial mode.
+ *
+ * @param file The file containing malformed UTF-8.
+ */
+ @ParameterizedTest
+ @ValueSource(strings = {"affix", "dictionary"})
+ void testPartialLoadingRejectsMalformedText(String file) {
+ final byte[] malformed = {(byte) 0xc3};
+ final ByteArrayInputStream affix = "affix".equals(file)
+ ? new ByteArrayInputStream(concat("SET UTF-8\nUNSUPPORTED_CASE K\nSFX A Y 1\nSFX A 0 ",
+ malformed)) : stream("UNSUPPORTED_CASE K\n");
+ final ByteArrayInputStream words = "dictionary".equals(file)
+ ? new ByteArrayInputStream(concat("1\n", malformed)) : stream(WORDS);
+ final IOException error = Assertions.assertThrows(IOException.class,
+ () -> HunspellDictionary.load(affix, words, LoadMode.ALLOW_PARTIAL));
+ Assertions.assertEquals(file + " stream is not valid UTF-8", error.getMessage());
+ }
+
+ /**
+ * Accepts legacy bytes in recognized metadata without decoding them as rules.
+ *
+ * @throws IOException Thrown if loading fails.
+ */
+ @Test
+ void testLegacyMetadataBytesAreIgnored() throws IOException {
+ final byte[] affix = concat("SET UTF-8\nNAME ", new byte[] {(byte) 0xc3});
+ final HunspellDictionary dictionary = HunspellDictionary.load(
+ new ByteArrayInputStream(affix), stream(WORDS));
+ Assertions.assertNotNull(dictionary.lookup("dog"));
+ Assertions.assertTrue(dictionary.getUnsupportedDirectives().isEmpty());
+ }
+
+ /**
+ * Preserves raw flag bytes in compound-boundary conditions without changing word text.
+ *
+ * @param matchingFlag Whether the left entry has the boundary flag.
+ * @throws IOException Thrown if loading fails.
+ */
+ @ParameterizedTest
+ @ValueSource(booleans = {true, false})
+ void testCompoundPatternByteFlag(boolean matchingFlag) throws IOException {
+ final byte[] prefix = concat("SET UTF-8\nCOMPOUNDFLAG C\nCOMPOUNDMIN 1\n"
+ + "CHECKCOMPOUNDPATTERN 1\nCHECKCOMPOUNDPATTERN er/", new byte[] {(byte) 0xc3});
+ final byte[] affix = Arrays.copyOf(prefix, prefix.length + 3);
+ System.arraycopy(" b\n".getBytes(StandardCharsets.UTF_8), 0, affix, prefix.length, 3);
+ final byte[] wordPrefix = concat("2\nriver/C", matchingFlag
+ ? new byte[] {(byte) 0xc3} : new byte[0]);
+ final byte[] ending = "\nboat/C\n".getBytes(StandardCharsets.UTF_8);
+ final byte[] words = Arrays.copyOf(wordPrefix, wordPrefix.length + ending.length);
+ System.arraycopy(ending, 0, words, wordPrefix.length, ending.length);
+ final HunspellStemmer stemmer = new HunspellStemmer(HunspellDictionary.load(
+ new ByteArrayInputStream(affix), new ByteArrayInputStream(words)));
+ Assertions.assertEquals(matchingFlag ? List.of("riverboat") : List.of("river", "boat"),
+ stemmer.stemAll("riverboat"));
+ }
+
+ /**
+ * Loads supported affix rules after a UTF-8 byte-order mark.
+ *
+ * @throws IOException Thrown if loading fails.
+ */
+ @Test
+ void testByteOrderMarkDoesNotHideSupportedDirective() throws IOException {
+ final HunspellDictionary dictionary = HunspellDictionary.load(
+ stream("\uFEFF" + RULES), stream(WORDS));
+ Assertions.assertEquals("dog", new HunspellStemmer(dictionary).stem("dogs").toString());
+ }
+
+ /**
+ * Reads the flag declaration on the first line after a UTF-8 byte-order mark, so
+ * the declared flag mode, not the raw-byte default, applies to the rules.
+ *
+ * @param affix The affix content following the byte-order mark.
+ * @param words The word list.
+ * @throws IOException Thrown if loading fails.
+ */
+ @ParameterizedTest
+ @MethodSource("flagDeclarationsAfterByteOrderMark")
+ void testByteOrderMarkDoesNotHideFlagDeclaration(byte[] affix, byte[] words)
+ throws IOException {
+ final byte[] marked = new byte[affix.length + 3];
+ marked[0] = (byte) 0xef;
+ marked[1] = (byte) 0xbb;
+ marked[2] = (byte) 0xbf;
+ System.arraycopy(affix, 0, marked, 3, affix.length);
+ final HunspellDictionary dictionary = HunspellDictionary.load(
+ new ByteArrayInputStream(marked), new ByteArrayInputStream(words));
+ Assertions.assertEquals("card", new HunspellStemmer(dictionary).stem("cards").toString());
+ }
+
+ /** {@return affix content with a flag declaration or a raw byte flag on the first line} */
+ private static Stream<Arguments> flagDeclarationsAfterByteOrderMark() {
+ return Stream.of(
+ Arguments.of("FLAG UTF-8\nSFX \u00e4 Y 1\nSFX \u00e4 0 s .\n".getBytes(StandardCharsets.UTF_8),
+ "1\ncard/\u00e4\n".getBytes(StandardCharsets.UTF_8)),
+ Arguments.of("FLAG long\nSFX Qz Y 1\nSFX Qz 0 s .\n".getBytes(StandardCharsets.UTF_8),
+ "1\ncard/Qz\n".getBytes(StandardCharsets.UTF_8)),
+ Arguments.of("FLAG num\nSFX 312 Y 1\nSFX 312 0 s .\n".getBytes(StandardCharsets.UTF_8),
+ "1\ncard/312\n".getBytes(StandardCharsets.UTF_8)),
+ Arguments.of(new byte[] {'S', 'F', 'X', ' ', (byte) 0xe4, ' ', 'Y', ' ', '1', '\n',
+ 'S', 'F', 'X', ' ', (byte) 0xe4, ' ', '0', ' ', 's', ' ', '.', '\n'},
+ new byte[] {'1', '\n', 'c', 'a', 'r', 'd', '/', (byte) 0xe4, '\n'}));
+ }
+
+ /**
+ * Rejects null arguments before reading either stream or opening a file.
+ *
+ * @param mode The loading policy.
+ */
+ @ParameterizedTest
+ @EnumSource(LoadMode.class)
+ void testNullArgumentsAreRejected(LoadMode mode) {
+ Assertions.assertThrows(IllegalArgumentException.class,
+ () -> HunspellDictionary.load((Path) null, directory, mode));
+ Assertions.assertThrows(IllegalArgumentException.class,
+ () -> HunspellDictionary.load(directory, null, mode));
+ Assertions.assertThrows(IllegalArgumentException.class,
+ () -> HunspellDictionary.load(null, stream(WORDS), mode));
+ Assertions.assertThrows(IllegalArgumentException.class,
+ () -> HunspellDictionary.load(stream(RULES), null, mode));
+ }
+
+ /** Rejects a null loading policy through both public entry points. */
+ @Test
+ void testNullModeIsRejected() {
+ Assertions.assertThrows(IllegalArgumentException.class,
+ () -> HunspellDictionary.load(directory, directory, null));
+ final ByteArrayInputStream affix = stream(RULES);
+ Assertions.assertThrows(IllegalArgumentException.class,
+ () -> HunspellDictionary.load(affix, stream(WORDS), null));
+ Assertions.assertEquals(RULES.getBytes(StandardCharsets.UTF_8).length, affix.available());
+ }
+
+ /**
+ * Preserves ownership of input streams on success and failure.
+ *
+ * @param mode The loading policy.
+ * @throws IOException Thrown if valid content fails to load.
+ */
+ @ParameterizedTest
+ @EnumSource(LoadMode.class)
+ void testStreamsAreNotClosed(LoadMode mode) throws IOException {
+ final TrackedStream affix = new TrackedStream(RULES);
+ final TrackedStream words = new TrackedStream(WORDS);
+ HunspellDictionary.load(affix, words, mode);
+ Assertions.assertFalse(affix.closed);
+ Assertions.assertFalse(words.closed);
+ final TrackedStream invalid = new TrackedStream("AF -1\n");
+ Assertions.assertThrows(IOException.class,
+ () -> HunspellDictionary.load(invalid, words, mode));
+ Assertions.assertFalse(invalid.closed);
+ Assertions.assertFalse(words.closed);
+ }
+
+ /** Verifies validation of a diagnostic's public fields. */
+ @Test
+ void testDiagnosticArgumentsAreValidated() {
+ Assertions.assertThrows(IllegalArgumentException.class,
+ () -> new UnsupportedDirective(null, "source", 1));
+ Assertions.assertThrows(IllegalArgumentException.class,
+ () -> new UnsupportedDirective(" ", "source", 1));
+ Assertions.assertThrows(IllegalArgumentException.class,
+ () -> new UnsupportedDirective("UNSUPPORTED_CONVERSION", null, 1));
+ Assertions.assertThrows(IllegalArgumentException.class,
+ () -> new UnsupportedDirective("UNSUPPORTED_CONVERSION", " ", 1));
+ Assertions.assertThrows(IllegalArgumentException.class,
+ () -> new UnsupportedDirective("UNSUPPORTED_CONVERSION", "source", 0));
+ }
+
+ /** Detects close calls while allowing further input operations. */
+ private static final class TrackedStream extends ByteArrayInputStream {
+ private boolean closed;
+
+ /**
+ * Creates an encoded fixture stream.
+ *
+ * @param content The fixture text.
+ */
+ private TrackedStream(String content) {
+ super(content.getBytes(StandardCharsets.UTF_8));
+ }
+
+ /** {@inheritDoc} */
+ @Override
+ public void close() {
+ closed = true;
+ }
+ }
+
+ /**
+ * Appends raw bytes to a UTF-8 fixture prefix.
+ *
+ * @param prefix The fixture prefix.
+ * @param bytes The raw suffix.
+ * @return The combined content.
+ */
+ private byte[] concat(String prefix, byte[] bytes) {
+ final byte[] encoded = prefix.getBytes(StandardCharsets.UTF_8);
+ final byte[] result = Arrays.copyOf(encoded, encoded.length + bytes.length);
+ System.arraycopy(bytes, 0, result, encoded.length, bytes.length);
+ return result;
+ }
+
+ /**
+ * Creates a UTF-8 stream for a fixture.
+ *
+ * @param content The fixture text.
+ * @return The encoded stream.
+ */
+ private ByteArrayInputStream stream(String content) {
+ return new ByteArrayInputStream(content.getBytes(StandardCharsets.UTF_8));
+ }
+}
diff --git a/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/stemmer/hunspell/HunspellManualExampleTest.java b/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/stemmer/hunspell/HunspellManualExampleTest.java
index 21daec7e0..a0ad935fa 100644
--- a/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/stemmer/hunspell/HunspellManualExampleTest.java
+++ b/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/stemmer/hunspell/HunspellManualExampleTest.java
@@ -20,10 +20,13 @@ package opennlp.tools.stemmer.hunspell;
import java.io.ByteArrayInputStream;
import java.io.IOException;
import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.Path;
import java.util.List;
import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
import opennlp.tools.stemmer.Stemmer;
@@ -70,4 +73,29 @@ public class HunspellManualExampleTest {
// unknown vocabulary passes through unchanged
Assertions.assertEquals("table", stemmer.stem("table").toString());
}
+
+ /**
+ * Checks partial loading and the diagnostic format used in the manual.
+ *
+ * @param directory The temporary fixture directory.
+ * @throws IOException Thrown if fixture creation or loading fails.
+ */
+ @Test
+ void testPartialLoadingDiagnostics(@TempDir Path directory) throws IOException {
+ final Path affix = directory.resolve("dictionary.aff");
+ final Path words = directory.resolve("dictionary.dic");
+ Files.writeString(affix, "UNSUPPORTED value\n" + AFFIX);
+ Files.writeString(words, WORDS);
+ final HunspellDictionary partial = HunspellDictionary.load(
+ affix, words, HunspellDictionary.LoadMode.ALLOW_PARTIAL);
+
+ Assertions.assertEquals("work",
+ new HunspellStemmerFactory(partial).newStemmer().stem("workers").toString());
+ Assertions.assertEquals(1, partial.getUnsupportedDirectives().size());
+ for (HunspellDictionary.UnsupportedDirective diagnostic : partial.getUnsupportedDirectives()) {
+ final String message = diagnostic.directive() + " at "
+ + diagnostic.source() + ":" + diagnostic.lineNumber();
+ Assertions.assertEquals("UNSUPPORTED at " + affix + ":1", message);
+ }
+ }
}
diff --git a/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/stemmer/hunspell/HunspellRealDictionaryTest.java b/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/stemmer/hunspell/HunspellRealDictionaryTest.java
deleted file mode 100644
index 9221658dd..000000000
--- a/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/stemmer/hunspell/HunspellRealDictionaryTest.java
+++ /dev/null
@@ -1,126 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package opennlp.tools.stemmer.hunspell;
-
-import java.io.IOException;
-import java.nio.file.Files;
-import java.nio.file.Path;
-
-import org.junit.jupiter.api.Assertions;
-import org.junit.jupiter.api.Assumptions;
-import org.junit.jupiter.api.Test;
-
-/**
- * Gated checks against published dictionaries, which are never bundled: the tests run
- * only when {@code -Dopennlp.hunspell.dict.dir} names a directory holding
- * {@code <name>.aff}/{@code <name>.dic} pairs, and each test additionally skips when
- * its dictionary pair is absent. The download helper in {@code dev/} fetches the pairs
- * together with their license files; see {@code dev/README-hunspell-dictionaries.md}.
- *
- * <p>The assertions are limited to morphology stable across dictionary revisions:
- * everyday inflections, and for German the decomposability of ordinary compounds.</p>
- */
-public class HunspellRealDictionaryTest {
-
- private static final String DICT_DIR_PROPERTY = "opennlp.hunspell.dict.dir";
-
- /**
- * Loads one dictionary pair from the gated directory, skipping the test when the
- * gate or the pair is absent.
- *
- * @param name The dictionary base name, such as {@code en_US}.
- * @return A stemmer over the loaded pair. Never {@code null}.
- * @throws IOException Thrown if a present pair fails to load, which is a failure,
- * not a skip.
- */
- private static HunspellStemmer loadOrSkip(String name) throws IOException {
- final String dir = System.getProperty(DICT_DIR_PROPERTY);
- Assumptions.assumeTrue(dir != null && !dir.isBlank(),
- "no " + DICT_DIR_PROPERTY + " given");
- final Path affix = Path.of(dir, name + HunspellDictionary.AFFIX_FILE_SUFFIX);
- final Path words = Path.of(dir, name + HunspellDictionary.DICTIONARY_FILE_SUFFIX);
- Assumptions.assumeTrue(Files.isReadable(affix) && Files.isReadable(words),
- name + " pair not present under " + dir);
- return new HunspellStemmer(HunspellDictionary.load(affix, words));
- }
-
- /**
- * Checks everyday English inflections against {@code en_US}, plus the identity
- * fallback on vocabulary no dictionary lists.
- *
- * @throws IOException Thrown if a present dictionary pair fails to load.
- */
- @Test
- void testEnglishInflections() throws IOException {
- final HunspellStemmer stemmer = loadOrSkip("en_US");
- Assertions.assertEquals("worker", stemmer.stem("workers").toString());
- Assertions.assertEquals("cat", stemmer.stem("cats").toString());
- Assertions.assertEquals("unhappy", stemmer.stem("unhappiest").toString());
- Assertions.assertEquals("quick", stemmer.stem("quickly").toString());
- Assertions.assertEquals("look", stemmer.stem("looked").toString());
- // unknown vocabulary degrades to identity
- Assertions.assertEquals("zyzzyvax", stemmer.stem("zyzzyvax").toString());
- }
-
- /**
- * Checks everyday German inflections against {@code de_DE_frami}: a plural, an
- * umlauted plural, and a superlative.
- *
- * @throws IOException Thrown if a present dictionary pair fails to load.
- */
- @Test
- void testGermanInflections() throws IOException {
- final HunspellStemmer stemmer = loadOrSkip("de_DE_frami");
- Assertions.assertEquals("Kind", stemmer.stem("Kinder").toString());
- // Haeuser, written with a-umlaut, stems to Haus
- Assertions.assertEquals("Haus", stemmer.stem("H\u00E4user").toString());
- Assertions.assertEquals("schnell", stemmer.stem("schnellsten").toString());
- }
-
- /**
- * Checks that ordinary German compounds decompose against {@code de_DE_frami}. Only
- * the part count is asserted: the exact part spellings follow the dictionary's own
- * entries and may shift between its revisions.
- *
- * @throws IOException Thrown if a present dictionary pair fails to load.
- */
- @Test
- void testGermanCompoundsDecompose() throws IOException {
- final HunspellStemmer stemmer = loadOrSkip("de_DE_frami");
- // Haustuer, written with u-umlaut, is Haus + Tuer
- Assertions.assertTrue(stemmer.stemAll("Haust\u00FCr").size() >= 2);
- Assertions.assertTrue(stemmer.stemAll("Kinderzimmer").size() >= 2);
- Assertions.assertTrue(stemmer.stemAll("Abbildungsverzeichnis").size() >= 2);
- }
-
- /**
- * Checks everyday Hungarian inflections against {@code hu_HU}: a plural and two
- * case-suffixed forms.
- *
- * @throws IOException Thrown if a present dictionary pair fails to load.
- */
- @Test
- void testHungarianInflections() throws IOException {
- final HunspellStemmer stemmer = loadOrSkip("hu_HU");
- // kutyak, written with a-acute, is the plural of kutya
- Assertions.assertEquals("kutya", stemmer.stem("kuty\u00E1k").toString());
- Assertions.assertEquals("asztal", stemmer.stem("asztalon").toString());
- // konyveket, written with o-umlaut, is an inflected form of konyv
- Assertions.assertEquals("k\u00F6nyv", stemmer.stem("k\u00F6nyveket").toString());
- }
-}
diff --git a/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/stemmer/hunspell/HunspellStemmerTest.java b/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/stemmer/hunspell/HunspellStemmerTest.java
index 312d78548..77512fa7a 100644
--- a/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/stemmer/hunspell/HunspellStemmerTest.java
+++ b/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/stemmer/hunspell/HunspellStemmerTest.java
@@ -22,6 +22,7 @@ import java.io.IOException;
import java.io.InputStream;
import java.nio.charset.Charset;
import java.nio.charset.StandardCharsets;
+import java.time.Duration;
import java.util.Arrays;
import java.util.List;
@@ -1182,8 +1183,9 @@ public class HunspellStemmerTest {
}
/**
- * Verifies CHECKCOMPOUNDDUP: a part must not repeat its left neighbor, while the
- * same dictionary without the declaration accepts the repetition.
+ * Verifies CHECKCOMPOUNDDUP: the closing part must not repeat the part before it,
+ * while an earlier repetition passes, as in Hunspell, and the
+ * same dictionary without the declaration accepts both.
*
* @throws IOException Thrown if a fixture fails to load.
*/
@@ -1192,10 +1194,11 @@ public class HunspellStemmerTest {
final String words = "2\ndog/Z\nhouse/Z\n";
final HunspellStemmer checked = new HunspellStemmer(load(
"COMPOUNDFLAG Z\nCOMPOUNDMIN 3\nCHECKCOMPOUNDDUP\n", words));
- Assertions.assertEquals(List.of("dogdoghouse"), checked.stemAll("dogdoghouse"));
+ Assertions.assertEquals(List.of("doghousehouse"), checked.stemAll("doghousehouse"));
+ Assertions.assertEquals(List.of("dog", "house"), checked.stemAll("dogdoghouse"));
final HunspellStemmer unchecked = new HunspellStemmer(load(
"COMPOUNDFLAG Z\nCOMPOUNDMIN 3\n", words));
- Assertions.assertEquals(List.of("dog", "house"), unchecked.stemAll("dogdoghouse"));
+ Assertions.assertEquals(List.of("dog", "house"), unchecked.stemAll("doghousehouse"));
}
/**
@@ -1249,38 +1252,27 @@ public class HunspellStemmerTest {
}
/**
- * Verifies that directives outside the affix-stemming subset do not prevent use of
- * the rules this implementation supports.
+ * Checks supported rules when partial loading skips unsupported directives.
*
* @param line The affix file line.
*/
@ParameterizedTest
@ValueSource(strings = {
- "ICONV 1",
- "OCONV 1",
- "COMPLEXPREFIXES",
- "COMPOUNDRULE 1",
- "COMPOUNDMORESUFFIXES",
- "COMPOUNDROOT R",
- "CHECKCOMPOUNDREP",
- "SIMPLIFIEDTRIPLE",
- "CHECKCOMPOUNDPATTERN 1",
- "FORCEUCASE U",
- "COMPOUNDSYLLABLE 6 aeiou",
- "SYLLABLENUM ABC",
- "LANG tr",
- "CHECKSHARPS",
- "BREAK 1",
- "FORBIDWARN",
- "IGNORE x",
- "KEEPCASE k"
+ "UNSUPPORTED_SYLLABLES ABC",
+ "UNSUPPORTED_LEMMA L",
+ "UNSUPPORTED value"
})
void testUnsupportedDirectiveDoesNotBlockSupportedRules(String line)
throws IOException {
- final HunspellStemmer stemmer = new HunspellStemmer(load(
- line + "\nSFX A Y 1\nSFX A 0 s .\n", "1\ndog/A\n"));
+ final HunspellDictionary dictionary = HunspellDictionary.load(
+ new ByteArrayInputStream((line + "\nSFX A Y 1\nSFX A 0 s .\n")
+ .getBytes(StandardCharsets.UTF_8)),
+ new ByteArrayInputStream("1\ndog/A\n".getBytes(StandardCharsets.UTF_8)),
+ HunspellDictionary.LoadMode.ALLOW_PARTIAL);
+ final HunspellStemmer stemmer = new HunspellStemmer(dictionary);
Assertions.assertEquals("dog", stemmer.stem("dogs").toString());
+ Assertions.assertEquals(1, dictionary.getUnsupportedDirectives().size());
}
/**
@@ -1493,13 +1485,19 @@ public class HunspellStemmerTest {
Assertions.assertEquals("foo", stemmer.stem("unfoosbar").toString());
}
- /** Verifies that an unrecognized directive does not block supported affix rules. */
+ /** Checks supported rules when partial loading skips an unknown directive. */
@Test
void testUnknownAffixDirectiveIsSkipped() throws IOException {
- final HunspellStemmer stemmer = new HunspellStemmer(load(
- "UNRECOGNIZED value\nSFX A Y 1\nSFX A 0 s .\n", "1\ndog/A\n"));
+ final HunspellDictionary dictionary = HunspellDictionary.load(
+ new ByteArrayInputStream("UNRECOGNIZED value\nSFX A Y 1\nSFX A 0 s .\n"
+ .getBytes(StandardCharsets.UTF_8)),
+ new ByteArrayInputStream("1\ndog/A\n".getBytes(StandardCharsets.UTF_8)),
+ HunspellDictionary.LoadMode.ALLOW_PARTIAL);
+ final HunspellStemmer stemmer = new HunspellStemmer(dictionary);
Assertions.assertEquals("dog", stemmer.stem("dogs").toString());
+ Assertions.assertEquals("UNRECOGNIZED",
+ dictionary.getUnsupportedDirectives().get(0).directive());
}
/** Verifies validation of the {@code AF} count line. */
@@ -1535,12 +1533,12 @@ public class HunspellStemmerTest {
* @param flag The invalid numeric flag.
*/
@ParameterizedTest
- @ValueSource(strings = {"-1", "0", "65001"})
+ @ValueSource(strings = {"-1", "0", "65536"})
void testNumericFlagOutsideRangeIsRejected(String flag) {
final IOException e = Assertions.assertThrows(IOException.class,
() -> load("FLAG num\n", "1\nword/" + flag + "\n"));
- Assertions.assertEquals("numeric flag outside 1..65000 at line 2: " + flag,
+ Assertions.assertEquals("numeric flag outside 1..65535 at line 2: " + flag,
e.getMessage());
}
@@ -1690,25 +1688,30 @@ public class HunspellStemmerTest {
@Test
void testForbiddenSurfaceOverridesAffixAnalysis() throws IOException {
final HunspellStemmer stemmer = new HunspellStemmer(load(
- "FORBIDDENWORD X\nSFX A Y 1\nSFX A 0 s .\n",
- "2\nfoo/A\nfoos/X\n"));
+ "FORBIDDENWORD F\nSFX A Y 1\nSFX A 0 s .\n",
+ "2\nreed/A\nreeds/F\n"));
- Assertions.assertEquals(List.of("foos"), stemmer.stemAll("foos"));
+ Assertions.assertEquals(List.of("reeds"), stemmer.stemAll("reeds"));
}
/**
- * Verifies that a forbidden homonym blocks affix analysis even when another entry
- * for the same surface is valid as a standalone entry.
+ * Verifies that the first listed homonym decides whether a surface form is
+ * forbidden, as in Hunspell: a forbidden first homonym blocks the
+ * standalone and affix analyses, while a valid first homonym keeps both.
*
* @throws IOException Thrown if the fixture fails to load.
*/
@Test
- void testForbiddenHomonymOverridesStandaloneEntry() throws IOException {
- final HunspellStemmer stemmer = new HunspellStemmer(load(
- "FORBIDDENWORD X\nSFX A Y 1\nSFX A 0 s .\n",
- "3\nfoo/A\nfoos\nfoos/X\n"));
-
- Assertions.assertEquals(List.of("foos"), stemmer.stemAll("foos"));
+ void testForbiddenFirstHomonymOverridesStandaloneEntry() throws IOException {
+ final HunspellStemmer forbiddenFirst = new HunspellStemmer(load(
+ "FORBIDDENWORD F\nSFX A Y 1\nSFX A 0 s .\n",
+ "3\nreed/A\nreeds/F\nreeds\n"));
+ Assertions.assertEquals(List.of("reeds"), forbiddenFirst.stemAll("reeds"));
+ Assertions.assertEquals(List.of(), forbiddenFirst.analyze("reeds"));
+ final HunspellStemmer validFirst = new HunspellStemmer(load(
+ "FORBIDDENWORD F\nSFX A Y 1\nSFX A 0 s .\n",
+ "3\nreed/A\nreeds\nreeds/F\n"));
+ Assertions.assertEquals(List.of("reeds", "reed"), validFirst.stemAll("reeds"));
}
/**
@@ -1790,16 +1793,16 @@ public class HunspellStemmerTest {
final HunspellStemmer stemmer = new HunspellStemmer(load(
String.join("\n",
"AF 2",
- "AF AB",
- "AF A",
- "SFX A Y 1",
- "SFX A 0 x .",
- "SFX B Y 1",
- "SFX B 0 y/2 .",
+ "AF PQ",
+ "AF P",
+ "SFX P Y 1",
+ "SFX P 0 s .",
+ "SFX Q Y 1",
+ "SFX Q 0 en/2 .",
""),
- "1\nfoo/1\n"));
+ "1\nquick/1\n"));
- Assertions.assertEquals(List.of("foo"), stemmer.stemAll("fooyx"));
+ Assertions.assertEquals(List.of("quick"), stemmer.stemAll("quickens"));
}
/**
@@ -1913,4 +1916,75 @@ public class HunspellStemmerTest {
Assertions.assertEquals(List.of(surface), stemmer.stemAll(surface));
}
+ /**
+ * Verifies that a FORBIDDENWORD flag among an affix rule's continuation classes
+ * marks the generated form as forbidden, as in Hunspell: the
+ * derived surface has no analysis even without a listed forbidden entry, in suffix
+ * position, in prefix position, and through a cross product involving the marked
+ * rule. A derivation over rules without the flag still reaches the listed stem.
+ *
+ * @throws IOException Thrown if a fixture fails to load.
+ */
+ @Test
+ void testForbiddenWordOnContinuationBlocksTheDerivedForm() throws IOException {
+ final HunspellStemmer markedSuffix = new HunspellStemmer(load(
+ "FORBIDDENWORD F\nPFX P Y 1\nPFX P 0 un .\n"
+ + "SFX S Y 1\nSFX S 0 s/F .\n",
+ "1\ndog/PS\n"));
+ Assertions.assertEquals(List.of("dogs"), markedSuffix.stemAll("dogs"));
+ Assertions.assertEquals(List.of("undogs"), markedSuffix.stemAll("undogs"));
+ Assertions.assertEquals(List.of("dog"), markedSuffix.stemAll("undog"));
+
+ final HunspellStemmer markedPrefix = new HunspellStemmer(load(
+ "FORBIDDENWORD F\nPFX P Y 1\nPFX P 0 un/F .\n",
+ "1\ndog/P\n"));
+ Assertions.assertEquals(List.of("undog"), markedPrefix.stemAll("undog"));
+ }
+
+ /**
+ * Verifies that a long word under a compound-declaring dictionary is answered
+ * promptly. The word-pair check inserts a space at every position of each text a
+ * compound level splits, so without a bound on the spaced forms a dictionary can
+ * read, the work grows with the cube of the word length.
+ *
+ * @param words The word list, with and without an entry containing a space.
+ */
+ @ParameterizedTest
+ @ValueSource(strings = {"1\nlo/CA\n", "2\nlo/CA\nlo la/A\n"})
+ void testLongWordWithCompoundsIsBounded(String words) {
+ final String word = "lo".repeat(10_000) + "q";
+ final List<CharSequence> stems = Assertions.assertTimeoutPreemptively(
+ Duration.ofSeconds(10), () -> new HunspellStemmer(load(
+ "COMPOUNDFLAG C\nCOMPOUNDMIN 1\nSFX A Y 1\nSFX A 0 s .\n", words))
+ .stemAll(word));
+ Assertions.assertEquals(List.of(word), stems);
+ }
+
+ /**
+ * Verifies that an underscore inside an {@code ICONV} or {@code OCONV} pattern stands
+ * for a space, as Hunspell reads it, after the leading and trailing anchors are
+ * removed. The input conversion turns {@code b ok} into the listed {@code book}, also
+ * with a mathematical bold b in the pattern; the output conversion turns the listed
+ * {@code a b} into {@code ab}.
+ *
+ * @param affix The conversion table.
+ * @param words The word list.
+ * @param input The stemmed text.
+ * @param expected The stem after both conversions.
+ * @throws IOException Thrown if the fixture fails to load.
+ */
+ @ParameterizedTest
+ @CsvSource(delimiter = '|', value = {
+ "ICONV 1\\nICONV b_ok book|1\\nbook|b ok|book",
+ "ICONV 1\\nICONV _b_ok_ book|1\\nbook|b ok|book",
+ "OCONV 1\\nOCONV a_b ab|1\\na b|a b|ab",
+ // a supplementary code point before the underscore, so the scan stays on code points
+ "ICONV 1\\nICONV \uD835\uDC1B_ok book|1\\nbook|\uD835\uDC1B ok|book"
+ })
+ void testConversionPatternUnderscoreIsSpace(String affix, String words, String input,
+ String expected) throws IOException {
+ final HunspellStemmer stemmer = new HunspellStemmer(load(
+ affix.replace("\\n", "\n") + "\n", words.replace("\\n", "\n") + "\n"));
+ Assertions.assertEquals(expected, stemmer.stem(input).toString());
+ }
}
diff --git a/opennlp-docs/src/docbkx/stemmer.xml b/opennlp-docs/src/docbkx/stemmer.xml
index d2367684b..1f8f41fa4 100644
--- a/opennlp-docs/src/docbkx/stemmer.xml
+++ b/opennlp-docs/src/docbkx/stemmer.xml
@@ -73,11 +73,12 @@ new CachingStemmer(factory).stem("running"); // "run"]]>
<section xml:id="tools.stemmer.hunspell">
<title>Hunspell dictionaries</title>
<para>
- <code>opennlp.tools.stemmer.hunspell</code> implements the documented
- Hunspell dictionary format: a user-supplied
+ <code>opennlp.tools.stemmer.hunspell</code> loads a user-supplied
<code>.aff</code> affix file and its <code>.dic</code> word list. OpenNLP
bundles no dictionary data; dictionaries are downloaded separately, and
- each states its own license. The dictionary is immutable and safe to share;
+ each provides license information. Retain the dictionary's copyright notices and
+ license text with downloaded files. OpenNLP's Apache License does not
+ relicense those files. The dictionary is immutable and safe to share;
<code>HunspellStemmerFactory</code> creates a fresh stemmer per call.
<code>HunspellManualExampleTest</code> asserts the behavior shown here.
<programlisting language="java"><![CDATA[
@@ -99,18 +100,132 @@ stemmer.stem("table"); // "table" (unknown vocabulary is unchanged)]]>
<code>-Dopennlp.download.remote=true</code>, and fetches through the
digest-verified <code>ResourceInstaller</code> path. A file that already exists
in the target is not replaced. Remove old files before refreshing a dictionary.
- Directives outside the supported affix-stemming subset are skipped, so
- published dictionaries can still use their supported rules. Conversion,
- suggestion, and advanced compound behavior from skipped directives is not
- applied to the returned stems.
- This includes <code>ICONV</code>, <code>OCONV</code>,
- <code>COMPLEXPREFIXES</code>, <code>COMPOUNDRULE</code>,
- <code>IGNORE</code>, and <code>KEEPCASE</code>. Results can differ from
- Hunspell for words that need these rules.
+ Loading uses <code>HunspellDictionary.LoadMode.STRICT</code> by default.
+ Unsupported affix directives cause an <code>IOException</code> with the
+ directive name and source line. File-based loading also includes the path.
+ Unknown directive names are rejected. Metadata and suggestion settings
+ that do not affect stemming or analysis are ignored.
+ Applications can select <code>ALLOW_PARTIAL</code> to use the supported
+ rules and inspect skipped directives:
+ <programlisting language="java"><![CDATA[
+HunspellDictionary partial = HunspellDictionary.load(
+ Path.of("dictionary.aff"), Path.of("dictionary.dic"),
+ HunspellDictionary.LoadMode.ALLOW_PARTIAL);
+for (HunspellDictionary.UnsupportedDirective diagnostic : partial.getUnsupportedDirectives()) {
+ System.err.println(diagnostic.directive() + " at "
+ + diagnostic.source() + ":" + diagnostic.lineNumber());
+}]]>
+ </programlisting>
+ The diagnostic list is immutable and contains the first source location
+ for each unsupported directive, in file order. Partial loading does not
+ apply behavior from skipped directives. Strict affix loading does not
+ establish complete Hunspell compatibility.
+ <code>ICONV</code> and <code>OCONV</code> convert input and output;
+ <code>IGNORE</code> removes configured characters. <code>KEEPCASE</code>,
+ <code>CHECKSHARPS</code>, and language-specific lowercase mappings govern
+ case variants. <code>COMPLEXPREFIXES</code> permits 2 prefixes and 1
+ suffix instead of 1 prefix and 2 suffixes. Compound rules, boundary
+ checks, simplified junctions, and recursive <code>BREAK</code> separators
+ return recognized part stems in order, without duplicates.
+ Morphological <code>st:</code> fields supply explicit stems,
+ <code>ds:</code> suffixes make the derived form the stem, and <code>sp:</code>
+ fields prepend surface material. <code>AM</code> expands morphology aliases.
+ Rules that add and strip no material are undone like any other rule. The
+ first listed homonym decides whether a spelling is forbidden, and a
+ forbidden direct or affixed reading also blocks compound and
+ <code>BREAK</code> readings.
+ <code>SYLLABLENUM</code> applies Hungarian compound syllable adjustments.
+ The deprecated <code>LEMMA_PRESENT</code> directive is accepted without
+ changing results. OpenNLP does not generate inflected forms or spelling
+ suggestions, and the morphological fields of an analysis are not part of the
+ public API; the stemmer returns stems.
+ </para>
+ <para>
+ Native Hunspell's spelling, stemming, and analysis operations can return
+ different results from one another. OpenNLP has one engine, so it follows
+ the spell checker for recognition and the analyzer for stems and fields
+ wherever the two agree, and resolves each disagreement the same way for
+ every caller. The results deviate from the native library in the
+ following cases. <code>HunspellCompatibilityEval</code> in
+ <code>opennlp-eval-tests</code> records each expected difference together
+ with the Hunspell output.
+ </para>
+ <itemizedlist>
+ <listitem>
+ <para>
+ Compound stems are the stems of the recognized parts, in order. The
+ native library concatenates the parts and stems the last one, so
+ <code>riverboats</code> gives <code>river</code> and
+ <code>boat</code> here and <code>riverboat</code> natively. The native
+ form is the input with its last part replaced by that part's stem.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ Purely numeric tokens such as <code>1.5</code> or <code>42-42</code>
+ are accepted by the native spell checker before any dictionary lookup.
+ Here they are unknown input and are returned unchanged. Recognize
+ numbers in the tokenizer or before stemming.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ A word starting with a dotted capital I is lowercased with Unicode
+ mapping outside the Turkic languages, so <code>İnvite</code> is
+ recognized when <code>invite</code> is listed. The native checker
+ never tries the lowercase form of such a word without a Turkic
+ <code>LANG</code>. Declare <code>LANG tr</code>, <code>az</code>, or
+ <code>crh</code> for Turkic data, where both implementations keep the
+ dotted and dotless letters apart.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ A compound of three or more parts that is one edit away from a listed
+ word, such as <code>sunsetlamp</code> next to <code>bunsetlamp</code>, is
+ rejected natively by running the spelling suggester, which OpenNLP
+ does not provide. Such compounds are recognized here. An application
+ that needs the rejection can validate multi-part compounds with a
+ spelling checker, or limit <code>COMPOUNDWORDMAX</code> in the affix
+ file where the language allows.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ Where the native spell checker and analyzer disagree, recognition
+ follows the spell checker and output follows the analyzer. This
+ affects all-uppercase Turkish words such as <code>İPEK</code>, which
+ the native checker rejects while its analyzer stems them; mixed-case
+ words with a capitalized initial such as <code>RaMarket</code> and
+ <code>BREAK</code> forms, which the native analyzer does not handle;
+ and forbidden affixed compounds, which the native analyzer stems. In
+ each case OpenNLP recognizes the word when the native checker does and
+ returns the analyzer's stems.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ An affix file without <code>SET</code> is read as UTF-8 and rejected
+ when it is not valid UTF-8; the native default is ISO-8859-1. Add
+ <code>SET ISO8859-1</code> to such a file.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ Malformed declarations are rejected where the native parser guesses:
+ a multi-character flag under the default flag mode, an <code>AF</code>
+ or <code>AM</code> table whose entry count does not match its header,
+ and a numeric flag outside 1 through 65535. Correct the declaration;
+ <code>ALLOW_PARTIAL</code> skips unsupported directives, not malformed
+ ones.
+ </para>
+ </listitem>
+ </itemizedlist>
+ <para>
<code>FLAG</code> and <code>AF</code> declarations apply to the complete
affix file, including rules listed before those declarations. Parsing
rejects malformed text in parsed rules, invalid counts, numeric flags
- outside the range 1 through 65000, and <code>COMPOUNDMIN</code> values
+ outside the range 1 through 65535, and <code>COMPOUNDMIN</code> values
that cannot be doubled safely. Comments and unused metadata may retain a
legacy encoding. Default and <code>long</code> flag modes preserve raw
one-byte flag values in UTF-8 files. Compound length and boundary checks
@@ -119,6 +234,14 @@ stemmer.stem("table"); // "table" (unknown vocabulary is unchanged)]]>
declares <code>FULLSTRIP</code>, as in Hunspell itself.
Each affix or dictionary stream is rejected when it exceeds
<code>HunspellDictionary.MAX_STREAM_BYTES</code> (64 MiB).
+ Search permits at most 64 compound components, 2048 compound candidate
+ checks per case variant, and 2048 word-break attempts per input. Sharp-s
+ expansion permits at most 64 case variants. Output is limited to 2048
+ distinct results. Compound-rule patterns permit 4096 flag elements.
+ Simplified triple letters can be restored at multiple junctions;
+ <code>CHECKCOMPOUNDPATTERN</code> replacement applies at one junction.
+ These limits can exclude valid analyses; candidates must pass validation
+ before inclusion in the result.
</para>
</section>
</chapter>
diff --git a/opennlp-eval-tests/src/test/java/opennlp/tools/eval/HunspellCompatibilityEval.java b/opennlp-eval-tests/src/test/java/opennlp/tools/eval/HunspellCompatibilityEval.java
new file mode 100644
index 000000000..80ce4a665
--- /dev/null
+++ b/opennlp-eval-tests/src/test/java/opennlp/tools/eval/HunspellCompatibilityEval.java
@@ -0,0 +1,401 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package opennlp.tools.eval;
+
+import java.io.File;
+import java.math.BigInteger;
+import java.nio.file.Path;
+import java.util.ArrayList;
+import java.util.EnumMap;
+import java.util.LinkedHashMap;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.Callable;
+import java.util.concurrent.Executors;
+import java.util.concurrent.TimeUnit;
+
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.TestReporter;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.EnumSource;
+
+import opennlp.tools.stemmer.hunspell.HunspellDictionary;
+import opennlp.tools.stemmer.hunspell.HunspellStemmer;
+
+/**
+ * Evaluates the Hunspell stemmer with the LibreOffice English, German, and Hungarian
+ * dictionaries under the {@code hunspell} directory of {@code OPENNLP_DATA_DIR}: strict
+ * loading, expected inflections and compounds, concurrent use, and agreement with the
+ * stems and the recognition recorded from Hunspell as described in
+ * {@code dev/README-hunspell-dictionaries.md}.
+ */
+public class HunspellCompatibilityEval extends AbstractEvalTest {
+
+ private static final String DATA_DIRECTORY = "hunspell";
+ private static final int THREADS = 4;
+ private static final int REPETITIONS = 10;
+ private static final int TIMEOUT_SECONDS = 60;
+ private static final String UNKNOWN = "zyzzyvax";
+
+ private static final Map<Dictionary, HunspellStemmer> STEMMERS = new EnumMap<>(Dictionary.class);
+
+ /** The external dictionaries, their MD5 digests, and the evaluated inputs. */
+ private enum Dictionary {
+ ENGLISH("en_US", "bbb118ea006c22ebe9ef7dbfe0dbfc2a", "7e671db5244b0496f9888e9f0176c360",
+ List.of("workers", "cats", "unhappiest", "quickly", "looked", "reading",
+ "dogs", "books", "walked", "walking", "talked", "talking", "played",
+ "playing", "helped", "helping", "houses", "children", "feet", "better",
+ "Workers", "WORKERS", "cAtS", "worker's", "well-known", "unhappy", "undone", UNKNOWN)),
+ GERMAN("de_DE_frami", "9fd6eb96145bdccb2dc0213e1df5a46e", "07c5fd0780eca6ba448cba8c7be170cd",
+ List.of("gegangen", "Kinder", "Häuser", "schnellsten", "Freunden", "Vorschläge",
+ "Haustür", "Kinderzimmer", "Abbildungsverzeichnis", "Haus", "Baum", "Buch", "schnell", UNKNOWN)),
+ HUNGARIAN("hu_HU", "c13482742489921ce2d8d19ff5122986", "66bab5478e829ba5e33afcde24a7259c",
+ List.of("kutyák", "asztalon", "könyveket", "házak", "emberek", "kutyáknak", UNKNOWN));
+
+ private final String id;
+ private final BigInteger affixChecksum;
+ private final BigInteger dictionaryChecksum;
+ private final List<String> inputs;
+
+ /**
+ * Describes one external dictionary.
+ *
+ * @param id The file name without suffix.
+ * @param affixChecksum The MD5 digest of the affix file, in hexadecimal.
+ * @param dictionaryChecksum The MD5 digest of the word list, in hexadecimal.
+ * @param inputs The evaluated inputs.
+ */
+ Dictionary(String id, String affixChecksum, String dictionaryChecksum, List<String> inputs) {
+ this.id = id;
+ this.affixChecksum = new BigInteger(affixChecksum, 16);
+ this.dictionaryChecksum = new BigInteger(dictionaryChecksum, 16);
+ this.inputs = inputs;
+ }
+ }
+
+ /**
+ * The outcome recorded from Hunspell for one input.
+ *
+ * @param accepted Whether the Hunspell spell checker accepted the input.
+ * @param stems The distinct stems the Hunspell stemmer returned.
+ */
+ private record Recorded(boolean accepted, Set<String> stems) { }
+
+ /** The classification of one comparison. */
+ private enum Outcome {
+ EXACT, EXPECTED_DIFFERENCE, IDENTITY_FALLBACK, UNEXPECTED
+ }
+
+ /**
+ * Verifies the dictionary digests and loads each dictionary once, strictly.
+ *
+ * @throws Exception If a file is missing, changed, or malformed.
+ */
+ @BeforeAll
+ static void verifyAndLoadDictionaries() throws Exception {
+ for (Dictionary dictionary : Dictionary.values()) {
+ final Path affix = file(dictionary, HunspellDictionary.AFFIX_FILE_SUFFIX);
+ final Path words = file(dictionary, HunspellDictionary.DICTIONARY_FILE_SUFFIX);
+ verifyFileChecksum(affix, dictionary.affixChecksum);
+ verifyFileChecksum(words, dictionary.dictionaryChecksum);
+ final HunspellDictionary loaded = HunspellDictionary.load(affix, words);
+ Assertions.assertTrue(loaded.getUnsupportedDirectives().isEmpty());
+ STEMMERS.put(dictionary, new HunspellStemmer(loaded));
+ }
+ }
+
+ /** Checks the comparison classification on synthetic results. */
+ @Test
+ void comparisonValidation() {
+ final Set<String> complete = Set.of("card");
+ final Set<String> empty = Set.of();
+ final Set<String> identity = Set.of(UNKNOWN);
+ final Recorded accepted = new Recorded(true, complete);
+ final Recorded acceptedEmpty = new Recorded(true, Set.of());
+ final Recorded rejected = new Recorded(false, Set.of());
+ Assertions.assertAll(
+ () -> Assertions.assertEquals(Outcome.EXACT, classify("card", accepted, complete, null)),
+ () -> Assertions.assertEquals(Outcome.EXPECTED_DIFFERENCE,
+ classify("card", acceptedEmpty, complete, complete)),
+ () -> Assertions.assertEquals(Outcome.IDENTITY_FALLBACK, classify(UNKNOWN, rejected, identity, null)),
+ () -> Assertions.assertEquals(Outcome.UNEXPECTED, classify("card", accepted, empty, null)),
+ () -> Assertions.assertEquals(Outcome.UNEXPECTED, classify("card", acceptedEmpty, complete, null)),
+ () -> Assertions.assertEquals(Outcome.UNEXPECTED, classify("card", rejected, Set.of("cards"), null)),
+ () -> Assertions.assertEquals(Outcome.UNEXPECTED, classify("card", accepted, complete, complete)),
+ () -> Assertions.assertEquals(Outcome.UNEXPECTED, classify("card", acceptedEmpty, empty, complete)),
+ () -> Assertions.assertEquals(Outcome.UNEXPECTED, classify(UNKNOWN, accepted, identity, null)));
+ }
+
+ /**
+ * Confirms that partial loading of a dictionary that loads strictly skips nothing.
+ *
+ * @param dictionary The external dictionary.
+ * @throws Exception If a file is missing or malformed.
+ */
+ @ParameterizedTest
+ @EnumSource(Dictionary.class)
+ void partialLoadingSkipsNothing(Dictionary dictionary) throws Exception {
+ final HunspellDictionary partial = HunspellDictionary.load(
+ file(dictionary, HunspellDictionary.AFFIX_FILE_SUFFIX),
+ file(dictionary, HunspellDictionary.DICTIONARY_FILE_SUFFIX),
+ HunspellDictionary.LoadMode.ALLOW_PARTIAL);
+ Assertions.assertTrue(partial.getUnsupportedDirectives().isEmpty());
+ }
+
+ /**
+ * Checks expected stems and compound decompositions.
+ *
+ * @param dictionary The external dictionary.
+ */
+ @ParameterizedTest
+ @EnumSource(Dictionary.class)
+ void expectedInflections(Dictionary dictionary) {
+ final HunspellStemmer stemmer = STEMMERS.get(dictionary);
+ final Map<String, String> expected = switch (dictionary) {
+ case ENGLISH -> Map.of("workers", "worker", "cats", "cat", "unhappiest", "unhappy",
+ "quickly", "quick", "looked", "look");
+ case GERMAN -> Map.of("Kinder", "Kind", "Häuser", "Haus", "schnellsten", "schnell");
+ case HUNGARIAN -> Map.of("kutyák", "kutya", "asztalon", "asztal", "könyveket", "könyv");
+ };
+ expected.forEach((word, stem) -> Assertions.assertEquals(stem,
+ stemmer.stem(word).toString(), word));
+ Assertions.assertEquals(List.of(UNKNOWN), stemmer.stemAll(UNKNOWN));
+ if (dictionary == Dictionary.GERMAN) {
+ for (String word : List.of("Haustür", "Kinderzimmer", "Abbildungsverzeichnis")) {
+ Assertions.assertTrue(stemmer.stemAll(word).size() >= 2, word);
+ }
+ }
+ }
+
+ /**
+ * Compares stems and recognition with the recorded Hunspell results and fails on any
+ * result that is neither exact, an expected difference, nor an identity fallback for
+ * an input Hunspell rejects.
+ *
+ * @param dictionary The external dictionary.
+ * @param reporter The test reporter for the counts.
+ */
+ @ParameterizedTest
+ @EnumSource(Dictionary.class)
+ void referenceCompatibility(Dictionary dictionary, TestReporter reporter) {
+ final HunspellStemmer stemmer = STEMMERS.get(dictionary);
+ final Map<String, Recorded> recorded = recorded(dictionary);
+ int exact = 0;
+ int differences = 0;
+ int fallback = 0;
+ final List<String> failures = new ArrayList<>();
+ for (String word : dictionary.inputs) {
+ final Recorded reference = recorded.get(word);
+ Assertions.assertNotNull(reference, "no recorded reference result for " + word);
+ final Set<String> stems = stems(stemmer, word);
+ switch (classify(word, reference, stems, expectedDifference(dictionary, word))) {
+ case EXACT -> exact++;
+ case EXPECTED_DIFFERENCE -> differences++;
+ case IDENTITY_FALLBACK -> fallback++;
+ case UNEXPECTED -> failures.add(word + ": reference=" + reference + ", OpenNLP=" + stems);
+ }
+ }
+ reporter.publishEntry(dictionary.id, "inputs=" + dictionary.inputs.size() + ", exact=" + exact
+ + ", expectedDifferences=" + differences + ", identityFallback=" + fallback
+ + ", unexpected=" + failures.size());
+ Assertions.assertTrue(failures.isEmpty(), () -> String.join("\n", failures));
+ }
+
+ /**
+ * Repeats the evaluated inputs from several threads on one shared stemmer and
+ * compares each result with the single-threaded result.
+ *
+ * @param dictionary The external dictionary.
+ * @throws Exception If a task fails.
+ */
+ @ParameterizedTest
+ @EnumSource(Dictionary.class)
+ void concurrentStemming(Dictionary dictionary) throws Exception {
+ final HunspellStemmer stemmer = STEMMERS.get(dictionary);
+ final Map<String, Set<String>> expected = new LinkedHashMap<>();
+ dictionary.inputs.forEach(word -> expected.put(word, stems(stemmer, word)));
+ final List<Callable<Void>> tasks = new ArrayList<>();
+ for (int thread = 0; thread < THREADS; thread++) {
+ final int offset = thread;
+ tasks.add(() -> {
+ for (int repeat = 0; repeat < REPETITIONS; repeat++) {
+ for (int index = 0; index < dictionary.inputs.size(); index++) {
+ final String word = dictionary.inputs.get((index + offset + repeat) % dictionary.inputs.size());
+ Assertions.assertEquals(expected.get(word), stems(stemmer, word), word);
+ }
+ }
+ return null;
+ });
+ }
+ try (var executor = Executors.newFixedThreadPool(THREADS)) {
+ for (var future : executor.invokeAll(tasks, TIMEOUT_SECONDS, TimeUnit.SECONDS)) {
+ Assertions.assertFalse(future.isCancelled(), "concurrent stemming timed out");
+ future.get();
+ }
+ }
+ }
+
+ /**
+ * Locates a dictionary file under the evaluation data directory.
+ *
+ * @param dictionary The external dictionary.
+ * @param suffix The file suffix.
+ * @return The file path.
+ * @throws Exception If the data directory is not configured or does not exist.
+ */
+ private static Path file(Dictionary dictionary, String suffix) throws Exception {
+ return new File(getOpennlpDataDir(), DATA_DIRECTORY + File.separator + dictionary.id + suffix).toPath();
+ }
+
+ /**
+ * Collects the distinct stems of one input.
+ *
+ * @param stemmer The stemmer.
+ * @param word The input.
+ * @return The stems in result order.
+ */
+ private static Set<String> stems(HunspellStemmer stemmer, String word) {
+ return new LinkedHashSet<>(stemmer.stemAll(word).stream().map(CharSequence::toString).toList());
+ }
+
+ /**
+ * Classifies one comparison. An expected difference must match the recorded
+ * OpenNLP output exactly and must still differ from the reference, so a stale entry
+ * is reported. For an input Hunspell rejects, the unchanged input is the identity
+ * fallback of unknown vocabulary; the public API does not tell it apart from a
+ * listed word that is its own stem.
+ *
+ * @param word The input.
+ * @param reference The recorded Hunspell outcome.
+ * @param stems The OpenNLP stems.
+ * @param expected The recorded OpenNLP stems for a known difference, or {@code null}.
+ * @return The classification.
+ */
+ private static Outcome classify(String word, Recorded reference, Set<String> stems, Set<String> expected) {
+ if (expected != null) {
+ return reference.accepted() && !reference.stems().equals(stems) && expected.equals(stems)
+ ? Outcome.EXPECTED_DIFFERENCE : Outcome.UNEXPECTED;
+ }
+ if (reference.accepted() && reference.stems().equals(stems)) {
+ return Outcome.EXACT;
+ }
+ if (!reference.accepted() && reference.stems().isEmpty() && stems.equals(Set.of(word))) {
+ return Outcome.IDENTITY_FALLBACK;
+ }
+ return Outcome.UNEXPECTED;
+ }
+
+ /**
+ * Specifies the OpenNLP stems for inputs whose recorded Hunspell stems differ:
+ * compound and break parts against concatenated or missing Hunspell stems, and
+ * standalone readings of capitalized German nouns against Hunspell's additional
+ * lowercase compound-only readings.
+ *
+ * @param dictionary The external dictionary.
+ * @param word The input.
+ * @return The expected OpenNLP stems, or {@code null} for an exact comparison.
+ */
+ private static Set<String> expectedDifference(Dictionary dictionary, String word) {
+ if (dictionary == Dictionary.ENGLISH && word.equals("well-known")) {
+ return Set.of("well", "known");
+ }
+ if (dictionary != Dictionary.GERMAN) {
+ return null;
+ }
+ return switch (word) {
+ case "Kinder" -> Set.of("Kind");
+ case "Häuser" -> Set.of("Haus");
+ case "Freunden" -> Set.of("freunden", "Freund");
+ case "Vorschläge" -> Set.of("Vor", "schlag");
+ case "Haustür" -> Set.of("Haus", "tür");
+ case "Kinderzimmer" -> Set.of("Kinder", "zimmer");
+ case "Abbildungsverzeichnis" -> Set.of("Abbildungs", "verzeichnis");
+ case "Haus" -> Set.of("Haus");
+ case "Baum" -> Set.of("Baum");
+ case "Buch" -> Set.of("Buch");
+ default -> null;
+ };
+ }
+
+ /**
+ * The results recorded from Hunspell for the evaluated inputs, keyed by input.
+ *
+ * @param dictionary The external dictionary.
+ * @return The recorded outcomes.
+ */
+ private static Map<String, Recorded> recorded(Dictionary dictionary) {
+ return switch (dictionary) {
+ case ENGLISH -> Map.ofEntries(
+ Map.entry("workers", new Recorded(true, Set.of("worker"))),
+ Map.entry("cats", new Recorded(true, Set.of("cat"))),
+ Map.entry("unhappiest", new Recorded(true, Set.of("unhappy"))),
+ Map.entry("quickly", new Recorded(true, Set.of("quick"))),
+ Map.entry("looked", new Recorded(true, Set.of("look"))),
+ Map.entry("reading", new Recorded(true, Set.of("reading", "read"))),
+ Map.entry("dogs", new Recorded(true, Set.of("dog"))),
+ Map.entry("books", new Recorded(true, Set.of("book"))),
+ Map.entry("walked", new Recorded(true, Set.of("walk"))),
+ Map.entry("walking", new Recorded(true, Set.of("walking", "walk"))),
+ Map.entry("talked", new Recorded(true, Set.of("talk"))),
+ Map.entry("talking", new Recorded(true, Set.of("talk"))),
+ Map.entry("played", new Recorded(true, Set.of("play"))),
+ Map.entry("playing", new Recorded(true, Set.of("play"))),
+ Map.entry("helped", new Recorded(true, Set.of("help"))),
+ Map.entry("helping", new Recorded(true, Set.of("helping", "help"))),
+ Map.entry("houses", new Recorded(true, Set.of("house"))),
+ Map.entry("children", new Recorded(true, Set.of("children"))),
+ Map.entry("feet", new Recorded(true, Set.of("feet"))),
+ Map.entry("better", new Recorded(true, Set.of("better"))),
+ Map.entry("Workers", new Recorded(true, Set.of("worker"))),
+ Map.entry("WORKERS", new Recorded(true, Set.of("worker"))),
+ Map.entry("cAtS", new Recorded(false, Set.of())),
+ Map.entry("worker's", new Recorded(true, Set.of("worker"))),
+ Map.entry("well-known", new Recorded(true, Set.of())),
+ Map.entry("unhappy", new Recorded(true, Set.of("unhappy", "happy"))),
+ Map.entry("undone", new Recorded(true, Set.of("done"))),
+ Map.entry(UNKNOWN, new Recorded(false, Set.of())));
+ case GERMAN -> Map.ofEntries(
+ Map.entry("gegangen", new Recorded(true, Set.of("gegangen"))),
+ Map.entry("Kinder", new Recorded(true, Set.of("kinder", "kind", "Kind"))),
+ Map.entry("Häuser", new Recorded(true, Set.of("häuser", "haus", "Haus"))),
+ Map.entry("schnellsten", new Recorded(true, Set.of("schnell"))),
+ Map.entry("Freunden", new Recorded(true, Set.of("freunden", "freund", "Freund"))),
+ Map.entry("Vorschläge", new Recorded(true, Set.of())),
+ Map.entry("Haustür", new Recorded(true, Set.of())),
+ Map.entry("Kinderzimmer", new Recorded(true, Set.of())),
+ Map.entry("Abbildungsverzeichnis", new Recorded(true, Set.of())),
+ Map.entry("Haus", new Recorded(true, Set.of("haus", "Haus"))),
+ Map.entry("Baum", new Recorded(true, Set.of("baum", "Baum"))),
+ Map.entry("Buch", new Recorded(true, Set.of("buch", "Buch"))),
+ Map.entry("schnell", new Recorded(true, Set.of("schnell"))),
+ Map.entry(UNKNOWN, new Recorded(false, Set.of())));
+ case HUNGARIAN -> Map.ofEntries(
+ Map.entry("kutyák", new Recorded(true, Set.of("kutya"))),
+ Map.entry("asztalon", new Recorded(true, Set.of("asztal"))),
+ Map.entry("könyveket", new Recorded(true, Set.of("könyv"))),
+ Map.entry("házak", new Recorded(true, Set.of("ház"))),
+ Map.entry("emberek", new Recorded(true, Set.of("ember"))),
+ Map.entry("kutyáknak", new Recorded(true, Set.of("kutya"))),
+ Map.entry(UNKNOWN, new Recorded(false, Set.of())));
+ };
+ }
+}