You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@opennlp.apache.org by "krickert (via GitHub)" <gi...@apache.org> on 2026/07/21 10:40:16 UTC
[PR] OPENNLP-1893: Support Hunspell affix dictionaries for stemming (opennlp)
krickert opened a new pull request, #1190:
URL: https://github.com/apache/opennlp/pull/1190
Adds a stemmer that reads standard Hunspell `.dic`/`.aff` dictionary pairs, built on the `StemmerFactory` seam from OPENNLP-1883.
Supported affix features: alias compression (AF), NEEDAFFIX, ONLYINCOMPOUND, FORBIDDENWORD, CIRCUMFIX, twofold suffix analysis through continuation classes, and compound word positioning including linking forms as used in German compounds. Parsing is regex-free; unsupported dictionary features fail loud at load time rather than degrading silently.
Dictionaries are user-supplied and nothing is bundled: the in-tree tests use a project-authored miniature dictionary, and `dev/README-hunspell-dictionaries.md` documents acquiring published dictionaries together with their license files. The manual gains a Hunspell section whose example is asserted by `HunspellManualExampleTest`.
Additionally verified against published dictionaries for English, Spanish, Hungarian, and German through the gated `HunspellRealDictionaryTest` (runs only when `-Dopennlp.hunspell.dict.dir` is set; skipped otherwise, and no dictionary data enters the tree).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opennlp.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] OPENNLP-1893: Support Hunspell affix dictionaries for stemming (opennlp)
Posted by "rzo1 (via GitHub)" <gi...@apache.org>.
rzo1 merged PR #1190:
URL: https://github.com/apache/opennlp/pull/1190
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opennlp.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] OPENNLP-1893: Support Hunspell affix dictionaries for stemming (opennlp)
Posted by "krickert (via GitHub)" <gi...@apache.org>.
krickert commented on PR #1190:
URL: https://github.com/apache/opennlp/pull/1190#issuecomment-5240442380
While this PR is under re-review we ran an internal bug hunt against the branch and found two holes in the fail-closed policy, both now fixed on the branch in the usual shape (failing tests in fe53f4c99, fix in 39b576e45). Flagging them here so the re-review can cover them.
First, three result-altering directives were silently accepted: COMPOUNDRULE, IGNORE, and KEEPCASE. Per hunspell(5), COMPOUNDRULE defines pattern compounds from flag sequences, IGNORE strips characters from dictionary words, affixes, and input before matching, and KEEPCASE forbids capitalized forms of the flagged words, which matters here because the stemmer analyzes a capitalized surface through its lowercase variant and would otherwise produce analyses hunspell's own analyzer rejects. All three now join ICONV, OCONV, and COMPLEXPREFIXES in the load-time rejection, with the same "unsupported affix directive" IOException.
Second, affix rules that strip the entire stem were applied without the FULLSTRIP directive. Hunspell itself accepts such affix files but refuses to apply the rule unless FULLSTRIP is declared, so the fix gates the rule at match time rather than rejecting at load; with FULLSTRIP declared the rule applies as before. The manual's stemmer chapter documents the extended directive list and the FULLSTRIP behavior.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opennlp.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] OPENNLP-1893: Support Hunspell affix dictionaries for stemming (opennlp)
Posted by "rzo1 (via GitHub)" <gi...@apache.org>.
rzo1 commented on PR #1190:
URL: https://github.com/apache/opennlp/pull/1190#issuecomment-5540126886
> @mawiesne patching a few edge cases I found here.. nothing big but I'll tag you when I push it.
Go for a new branch plz - makes the diff easier. The changes are in now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opennlp.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] OPENNLP-1893: Support Hunspell affix dictionaries for stemming (opennlp)
Posted by "krickert (via GitHub)" <gi...@apache.org>.
krickert commented on PR #1190:
URL: https://github.com/apache/opennlp/pull/1190#issuecomment-5333920425
Heads-up: #1211 is now stacked on top of this PR. Once this merges, #1211 lands the shared installer and, in the same stack, deletes the download code here in favor of ResourceInstaller, so there is no need to re-review the download path in this PR; the adopting commits live in #1211.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opennlp.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] OPENNLP-1893: Support Hunspell affix dictionaries for stemming (opennlp)
Posted by "mawiesne (via GitHub)" <gi...@apache.org>.
mawiesne commented on PR #1190:
URL: https://github.com/apache/opennlp/pull/1190#issuecomment-5489533357
@krickert Can we rebase and resolve this PR next? I think we can make some progress here soon.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opennlp.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] OPENNLP-1893: Support Hunspell affix dictionaries for stemming (opennlp)
Posted by "rzo1 (via GitHub)" <gi...@apache.org>.
rzo1 commented on PR #1190:
URL: https://github.com/apache/opennlp/pull/1190#issuecomment-5318291188
I think that we need to wait with this one until we have discussed the download resource direction from https://github.com/apache/opennlp/pull/1191#issuecomment-5285908363
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opennlp.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] OPENNLP-1893: Support Hunspell affix dictionaries for stemming (opennlp)
Posted by "krickert (via GitHub)" <gi...@apache.org>.
krickert commented on PR #1190:
URL: https://github.com/apache/opennlp/pull/1190#issuecomment-5208823512
Synced the shared DownloadUtil from #1191 in 39afc7e59: the 512 MiB download ceiling is now a default overridable at JVM startup via `opennlp.download.max.bytes` (absent or invalid values fall back). No behavior change for the catalog dictionaries here, which are far below the ceiling.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opennlp.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] OPENNLP-1893: Support Hunspell affix dictionaries for stemming (opennlp)
Posted by "krickert (via GitHub)" <gi...@apache.org>.
krickert commented on PR #1190:
URL: https://github.com/apache/opennlp/pull/1190#issuecomment-5236015276
Description updated: it now states the fail-loud/skip split per directive class instead of the blanket claim.
The three items without a commit, deliberately:
- unescapedSlash: backslash-slash reads as an escaped slash, matching hunspell(5), which defines \/ as the way to put a literal slash in a word. Under that grammar a word ending in a literal backslash directly before a flag run is not expressible, in the reference parser either, so there is no divergence to fix.
- morphologyIndex: the two-letter-tag-plus-colon rule is the morphological field grammar hunspell(5) defines, scanned only after a space or tabulator, which are the two separators hashmgr.cxx splits on. A multi-word entry containing a bare "xy:" run after a space is truncated, but such an entry is ambiguous under the format grammar itself; the tabulator layout expresses it unambiguously and loads correctly here.
- The .aff double decode (ASCII pass to find SET, then the declared charset): kept. Both passes now run under the 64 MiB stream ceiling, so the transient second copy is bounded; folding the two passes into one buys little and costs the clean split between charset discovery and parsing.
PART_CHECK_BUDGET stays a fixed constant: it is documented in the public class javadoc now, and no real dictionary has come close to it in testing. If a compound-heavy dictionary ever does, promoting it to a startup override like the download ceilings is a small follow-up.
That covers every item from the 2026-08-04 review; ready for another look.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opennlp.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] OPENNLP-1893: Support Hunspell affix dictionaries for stemming (opennlp)
Posted by "krickert (via GitHub)" <gi...@apache.org>.
krickert commented on PR #1190:
URL: https://github.com/apache/opennlp/pull/1190#issuecomment-5539968521
@mawiesne patching a few edge cases I found here.. nothing big but I'll tag you when I push it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opennlp.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] OPENNLP-1893: Support Hunspell affix dictionaries for stemming (opennlp)
Posted by "mawiesne (via GitHub)" <gi...@apache.org>.
mawiesne commented on code in PR #1190:
URL: https://github.com/apache/opennlp/pull/1190#discussion_r3850720565
##########
opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/hunspell/HunspellDictionary.java:
##########
@@ -0,0 +1,1254 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package opennlp.tools.stemmer.hunspell;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.charset.Charset;
+import java.nio.charset.IllegalCharsetNameException;
+import java.nio.charset.StandardCharsets;
+import java.nio.charset.UnsupportedCharsetException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import opennlp.tools.commons.ThreadSafe;
+import opennlp.tools.util.StringUtil;
+
+/**
+ * An immutable, in-memory Hunspell-format dictionary: the word list of a {@code .dic}
+ * file and the prefix and suffix rules of its {@code .aff} companion, loaded from
+ * user-supplied files. The engine implements the documented format directly; no
+ * dictionary data is bundled, dictionaries are supplied by the user.
+ *
+ * <p>Supported affix features: {@code PFX} and {@code SFX} rules with strip strings,
+ * character-class conditions, and cross-product combination of one prefix with one
+ * suffix; twofold suffixes through the continuation classes on suffix rules;
+ * {@code FLAG} modes {@code char} (default), {@code UTF-8}, {@code long}, and
+ * {@code num}; the {@code AF} flag alias table; the {@code SET} encoding declaration;
+ * compound decomposition under {@code COMPOUNDFLAG}, the positional
+ * {@code COMPOUNDBEGIN}/{@code COMPOUNDMIDDLE}/{@code COMPOUNDEND} flags,
+ * {@code COMPOUNDMIN}, {@code COMPOUNDWORDMAX}, {@code COMPOUNDPERMITFLAG},
+ * {@code COMPOUNDFORBIDFLAG}, and the {@code CHECKCOMPOUNDDUP},
+ * {@code CHECKCOMPOUNDCASE}, and {@code CHECKCOMPOUNDTRIPLE} declarations, with
+ * compound parts standing on their entries alone or on an entry plus one affix; the
+ * blocking flags
+ * {@code NEEDAFFIX} (with its historical alias {@code PSEUDOROOT}),
+ * {@code ONLYINCOMPOUND}, and {@code FORBIDDENWORD}, which suppress analyses the
+ * dictionary marks as virtual stems, compound-only parts, or forbidden words; and
+ * {@code CIRCUMFIX}, which binds marked prefix and suffix halves to one another; and
+ * the {@code FULLSTRIP} declaration, without which a rule that strips a whole stem is
+ * not applied, matching hunspell.
+ * Directives that would change stems when ignored ({@code ICONV}, {@code OCONV},
+ * {@code COMPLEXPREFIXES}, {@code COMPOUNDRULE}, {@code IGNORE},
+ * {@code KEEPCASE}) are rejected at load time. Cosmetic tables such as
+ * {@code REP}, {@code MAP}, and {@code KEY} are skipped, so analyses that would need
+ * them are missed rather than invented.</p>
+ *
+ * <p>Instances are immutable and safe to share between threads.</p>
+ *
+ * @see HunspellStemmer
+ * @see HunspellStemmerFactory
+ * @since 3.0.0
+ */
+@ThreadSafe
+public final class HunspellDictionary {
+
+ /**
+ * Inclusive upper bound on bytes buffered from one affix or dictionary stream
+ * during {@link #load(InputStream, InputStream)}. Larger streams fail with
+ * {@link IOException}.
+ */
+ public static final int MAX_STREAM_BYTES = 64 * 1024 * 1024;
+
+ /**
+ * One parsed affix rule of a {@code PFX} or {@code SFX} block.
+ *
+ * @param flag The flag naming the rule's block, which an entry carries to accept it.
+ * @param crossProduct Whether the rule may combine with an affix of the opposite kind.
+ * @param strip The stem material the rule replaces, restored during analysis.
+ * @param affix The surface material the rule adds to the stem.
+ * @param condition The condition the stem must satisfy for the rule to apply.
+ * @param continuation The flags of the further affixes that may stack on this one.
+ */
+ record Affix(int flag, boolean crossProduct, String strip, String affix,
+ AffixCondition condition, int[] continuation) {
+
+ /**
+ * Checks whether a further affix may stack on this one.
+ *
+ * @param otherFlag The stacking affix's flag.
+ * @return {@code true} if this affix's continuation classes allow it.
+ */
+ boolean allowsContinuation(int otherFlag) {
+ for (final int candidate : continuation) {
+ if (candidate == otherFlag) {
+ return true;
+ }
+ }
+ return false;
+ }
+ }
+
+ /** The place a part takes in a compound, deciding which positional flag admits it. */
+ enum CompoundPosition {
+ /** The first part. */
+ BEGIN,
+ /** Any part between the first and the last. */
+ MIDDLE,
+ /** The last part. */
+ END
+ }
+
+ /** The shared empty bucket answered for characters no affix rule is keyed under. */
+ private static final List<Affix> NO_AFFIXES = List.of();
+
+ /** The line tag of a prefix block and of every rule line inside it. */
+ private static final String PREFIX_TAG = "PFX";
+
+ /** The line tag of a suffix block and of every rule line inside it. */
+ private static final String SUFFIX_TAG = "SFX";
+
+ /** The affix format's marker for absent strip or affix material. */
+ private static final String NO_MATERIAL = "0";
+
+ private final Map<String, List<int[]>> entries;
+ private final BoundaryIndex suffixesByLast;
+ private final List<Affix> suffixesWithoutMaterial;
+ private final BoundaryIndex prefixesByFirst;
+ private final List<Affix> prefixesWithoutMaterial;
+ private final int compoundFlag;
+ private final int compoundBegin;
+ private final int compoundEnd;
+ private final int compoundMin;
+ private final int needAffix;
+ private final int onlyInCompound;
+ private final int forbiddenWord;
+ private final int circumfix;
+ private final int compoundMiddle;
+ private final int compoundPermit;
+ private final int compoundForbid;
+ private final int compoundWordMax;
+ private final boolean checkCompoundDup;
+ private final boolean checkCompoundCase;
+ private final boolean checkCompoundTriple;
+ private final boolean fullStrip;
+
+ /**
+ * Initializes the dictionary from the two parsed files.
+ *
+ * @param entries The words mapped to the flag sets of their entries.
+ * @param affix The parsed affix file.
+ */
+ private HunspellDictionary(Map<String, List<int[]>> entries, AffixFile affix) {
+ this.compoundFlag = affix.compoundFlag;
+ this.compoundBegin = affix.compoundBegin;
+ this.compoundEnd = affix.compoundEnd;
+ this.compoundMin = affix.compoundMin;
+ this.needAffix = affix.needAffix;
+ this.onlyInCompound = affix.onlyInCompound;
+ this.forbiddenWord = affix.forbiddenWord;
+ this.circumfix = affix.circumfix;
+ this.compoundMiddle = affix.compoundMiddle;
+ this.compoundPermit = affix.compoundPermit;
+ this.compoundForbid = affix.compoundForbid;
+ this.compoundWordMax = affix.compoundWordMax;
+ this.checkCompoundDup = affix.checkCompoundDup;
+ this.checkCompoundCase = affix.checkCompoundCase;
+ this.checkCompoundTriple = affix.checkCompoundTriple;
+ this.fullStrip = affix.fullStrip;
+ this.entries = entries;
+ // A material-bearing rule can only be undone from a word whose boundary
+ // character matches its affix material, so bucketing by that character
+ // narrows each scan to one bucket plus the strip-only rules.
+ final List<Affix> suffixesWithout = new ArrayList<>();
+ this.suffixesByLast = bucketByBoundary(affix.suffixes, true, suffixesWithout);
+ this.suffixesWithoutMaterial = List.copyOf(suffixesWithout);
+ final List<Affix> prefixesWithout = new ArrayList<>();
+ this.prefixesByFirst = bucketByBoundary(affix.prefixes, false, prefixesWithout);
+ this.prefixesWithoutMaterial = List.copyOf(prefixesWithout);
+ }
+
+ /**
+ * An immutable index of affix rules keyed by the boundary code point of their affix
+ * material, answering each lookup by binary search so the per-word scans in
+ * {@link HunspellStemmer} allocate nothing.
+ */
+ private static final class BoundaryIndex {
+
+ /** The boundary code points, sorted ascending. */
+ private final int[] boundaries;
+ /** The rule bucket for each boundary, aligned with {@link #boundaries}. */
+ private final List<List<Affix>> buckets;
+
+ /**
+ * Initializes the index from mutable buckets, freezing each one.
+ *
+ * @param byBoundary The rule buckets keyed by boundary code point.
+ */
+ private BoundaryIndex(Map<Integer, List<Affix>> byBoundary) {
+ this.boundaries = new int[byBoundary.size()];
+ int b = 0;
+ for (final Integer boundary : byBoundary.keySet()) {
+ boundaries[b++] = boundary;
+ }
+ Arrays.sort(boundaries);
+ this.buckets = new ArrayList<>(boundaries.length);
+ for (final int boundary : boundaries) {
+ buckets.add(List.copyOf(byBoundary.get(boundary)));
+ }
+ }
+
+ /**
+ * The rules bucketed under a boundary code point.
+ *
+ * @param codePoint The boundary code point to look up.
+ * @return The bucket, possibly empty. Never {@code null}.
+ */
+ List<Affix> bucket(int codePoint) {
+ final int index = Arrays.binarySearch(boundaries, codePoint);
+ return index >= 0 ? buckets.get(index) : NO_AFFIXES;
+ }
+ }
+
+ /**
+ * Buckets affix rules by the boundary code point of their affix material, the last
+ * code point for a suffix rule and the first for a prefix rule.
+ *
+ * @param rules The rules of one kind, in file order.
+ * @param suffix Whether the rules are suffix rules.
+ * @param withoutMaterial Collects the rules with empty affix material, which no
+ * boundary code point keys.
+ * @return The rules indexed by their boundary code point. Never {@code null}.
+ */
+ private static BoundaryIndex bucketByBoundary(List<Affix> rules,
+ boolean suffix, List<Affix> withoutMaterial) {
+ final Map<Integer, List<Affix>> byBoundary = new HashMap<>();
+ for (final Affix rule : rules) {
+ final String material = rule.affix();
+ if (material.isEmpty()) {
+ withoutMaterial.add(rule);
+ } else {
+ final int boundary = suffix
+ ? material.codePointBefore(material.length())
+ : material.codePointAt(0);
+ byBoundary.computeIfAbsent(boundary, key -> new ArrayList<>()).add(rule);
+ }
+ }
+ return new BoundaryIndex(byBoundary);
+ }
+
+ /**
+ * Loads a dictionary from its two files.
+ *
+ * @param affixFile The {@code .aff} affix file. Must not be {@code null}.
+ * @param dictionaryFile The {@code .dic} word list. Must not be {@code null}.
+ * @return The loaded dictionary. Never {@code null}.
+ * @throws IOException Thrown if reading fails or a file is malformed.
+ * @throws IllegalArgumentException Thrown if a parameter is {@code null}.
+ */
+ public static HunspellDictionary load(Path affixFile, Path dictionaryFile)
+ throws IOException {
+ if (affixFile == null) {
+ throw new IllegalArgumentException("affixFile must not be null");
+ }
+ if (dictionaryFile == null) {
+ throw new IllegalArgumentException("dictionaryFile must not be null");
+ }
+ try (InputStream affix = Files.newInputStream(affixFile);
+ InputStream dictionary = Files.newInputStream(dictionaryFile)) {
+ return load(affix, dictionary);
+ }
+ }
+
+ /**
+ * Loads a dictionary from its two streams. Each stream is buffered up to
+ * {@link #MAX_STREAM_BYTES} bytes; a larger stream fails with {@link IOException}.
+ *
+ * @param affixStream The {@code .aff} affix content. Must not be {@code null}. Not
+ * closed.
+ * @param dictionaryStream The {@code .dic} word list content. Must not be
+ * {@code null}. Not closed.
+ * @return The loaded dictionary. Never {@code null}.
+ * @throws IOException Thrown if reading fails, a stream exceeds
+ * {@link #MAX_STREAM_BYTES}, or the content is malformed.
+ * @throws IllegalArgumentException Thrown if a parameter is {@code null}.
+ */
+ public static HunspellDictionary load(InputStream affixStream,
+ InputStream dictionaryStream) throws IOException {
+ if (affixStream == null) {
+ throw new IllegalArgumentException("affixStream must not be null");
+ }
+ if (dictionaryStream == null) {
+ throw new IllegalArgumentException("dictionaryStream must not be null");
+ }
+ final byte[] affixBytes = readBounded(affixStream, MAX_STREAM_BYTES, "affix stream");
+ final Charset charset = declaredCharset(affixBytes);
+ final AffixFile affix = parseAffix(new String(affixBytes, charset));
+ final Map<String, List<int[]>> entries = parseWordList(
+ new String(readBounded(dictionaryStream, MAX_STREAM_BYTES, "dictionary stream"),
+ charset),
+ affix.flagMode, affix.flagAliases);
+ return new HunspellDictionary(entries, affix);
+ }
+
+ /**
+ * Reads an input stream into a byte array, failing when more than {@code maxBytes}
+ * arrive.
+ *
+ * @param in The stream to read. Not closed.
+ * @param maxBytes The inclusive upper bound on buffered bytes.
+ * @param label The stream name used in the error message.
+ * @return The buffered bytes. Never {@code null}.
+ * @throws IOException Thrown if reading fails or the stream exceeds {@code maxBytes}.
+ */
+ static byte[] readBounded(InputStream in, int maxBytes, String label)
+ throws IOException {
+ final byte[] chunk = new byte[8192];
+ byte[] buffer = new byte[Math.min(8192, maxBytes)];
+ int size = 0;
+ int n;
+ while ((n = in.read(chunk)) >= 0) {
+ if (size + n > maxBytes) {
+ throw new IOException(label + " size exceeds safe limit of " + maxBytes);
+ }
+ if (size + n > buffer.length) {
+ buffer = Arrays.copyOf(buffer, Math.min(maxBytes, Math.max(buffer.length * 2, size + n)));
+ }
+ System.arraycopy(chunk, 0, buffer, size, n);
+ size += n;
+ }
+ return size == buffer.length ? buffer : Arrays.copyOf(buffer, size);
+ }
+
+ /**
+ * Looks up a word's flag sets.
+ *
+ * @param word The word exactly as listed.
+ * @return The flag sets of all matching entries, or {@code null} when absent.
+ */
+ List<int[]> lookup(String word) {
+ final List<int[]> found = entries.get(word);
+ if (found == null) {
+ return null;
+ }
+ final List<int[]> copy = new ArrayList<>(found.size());
+ for (final int[] flags : found) {
+ copy.add(flags.clone());
+ }
+ return copy;
+ }
+
+ /**
+ * The suffix rules whose affix material ends in the given code point, which are the
+ * only material-bearing rules that can be undone from a word ending in it.
+ *
+ * @param last The word's last code point.
+ * @return The bucket, possibly empty. Never {@code null}.
+ */
+ List<Affix> suffixesEndingWith(int last) {
+ return suffixesByLast.bucket(last);
+ }
+
+ /** {@return the strip-only suffix rules, applicable to any word} Never {@code null}. */
+ List<Affix> suffixesWithoutMaterial() {
+ return suffixesWithoutMaterial;
+ }
+
+ /**
+ * The prefix rules whose affix material starts with the given code point, which are
+ * the only material-bearing rules that can be undone from a word starting with it.
+ *
+ * @param first The word's first code point.
+ * @return The bucket, possibly empty. Never {@code null}.
+ */
+ List<Affix> prefixesStartingWith(int first) {
+ return prefixesByFirst.bucket(first);
+ }
+
+ /** {@return the strip-only prefix rules, applicable to any word} Never {@code null}. */
+ List<Affix> prefixesWithoutMaterial() {
+ return prefixesWithoutMaterial;
+ }
+
+ /** {@return whether the affix file declares any compounding flag at all} */
+ boolean compoundsDeclared() {
+ return compoundFlag != 0 || compoundBegin != 0 || compoundEnd != 0
+ || compoundMiddle != 0;
+ }
+
+ /** {@return the smallest length a compound part may have} At least {@code 1}. */
+ int compoundMin() {
+ return compoundMin;
+ }
+
+ /** {@return the largest number of parts a compound may have} {@code 0} is unbounded. */
+ int compoundWordMax() {
+ return compoundWordMax;
+ }
+
+ /** {@return whether {@code CHECKCOMPOUNDDUP} forbids a part repeating its neighbor} */
+ boolean checkCompoundDup() {
+ return checkCompoundDup;
+ }
+
+ /** {@return whether {@code CHECKCOMPOUNDCASE} forbids uppercase at part boundaries} */
+ boolean checkCompoundCase() {
+ return checkCompoundCase;
+ }
+
+ /** {@return whether {@code CHECKCOMPOUNDTRIPLE} forbids triple letters at boundaries} */
+ boolean checkCompoundTriple() {
+ return checkCompoundTriple;
+ }
+
+ /** {@return whether {@code FULLSTRIP} allows an affix rule to strip a whole stem} */
+ boolean fullStrip() {
+ return fullStrip;
+ }
+
+ /**
+ * The flag admitting a part at a compound position, next to the general
+ * compounding flag.
+ *
+ * @param position The part's place in the compound.
+ * @return The dedicated positional flag, or {@code 0} when undeclared.
+ */
+ private int positionalFlag(CompoundPosition position) {
+ return switch (position) {
+ case BEGIN -> compoundBegin;
+ case MIDDLE -> compoundMiddle;
+ case END -> compoundEnd;
+ };
+ }
+
+ /**
+ * Checks whether a listed word may stand at a compound position: some homonym's
+ * flag set carries the general compounding flag or the position's dedicated flag
+ * and is not forbidden. A compound-only or virtual-stem homonym may take the
+ * position; that is what those flags permit.
+ *
+ * @param flagSets The word's flag sets from {@link #lookup(String)}.
+ * @param position The part's place in the compound.
+ * @return {@code true} if the word may stand at the position.
+ */
+ boolean mayStand(List<int[]> flagSets, CompoundPosition position) {
+ final int positional = positionalFlag(position);
+ for (final int[] flags : flagSets) {
+ if ((contains(flags, compoundFlag) || contains(flags, positional))
+ && !contains(flags, forbiddenWord) && !contains(flags, needAffix)) {
+ return true;
+ }
+ }
+ return false;
+ }
+
+ /**
+ * Checks whether some homonym supports an affixed compound part: its flag set
+ * carries the removed affix's flag, is not forbidden, and either the affix itself
+ * admits the position or the set carries the compounding or positional flag.
+ *
+ * @param flagSets The part stem's flag sets from {@link #lookup(String)}.
+ * @param affixFlag The removed affix's flag.
+ * @param position The part's place in the compound.
+ * @param affixAdmits Whether the affix's continuation classes admit the position,
+ * from {@link #affixAdmits(Affix, CompoundPosition)}.
+ * @return {@code true} if some homonym stands affixed at the position.
+ */
+ boolean supportsPart(List<int[]> flagSets, int affixFlag, CompoundPosition position,
+ boolean affixAdmits) {
+ final int positional = positionalFlag(position);
+ for (final int[] flags : flagSets) {
+ if (contains(flags, affixFlag) && !contains(flags, forbiddenWord)
+ && (affixAdmits || contains(flags, compoundFlag)
+ || contains(flags, positional))) {
+ return true;
+ }
+ }
+ return false;
+ }
+
+ /**
+ * Checks whether an affix admits its derived form at a compound position: its
+ * continuation classes carry the general compounding flag or the position's
+ * dedicated flag. Published dictionaries position their linking forms this way,
+ * through zero or dash suffixes whose continuation classes hold the positional
+ * flags.
+ *
+ * @param affix The affix rule applied to the part.
+ * @param position The part's place in the compound.
+ * @return {@code true} if the affixed form may stand at the position.
+ */
+ boolean affixAdmits(Affix affix, CompoundPosition position) {
+ return (compoundFlag != 0 && affix.allowsContinuation(compoundFlag))
+ || (positionalFlag(position) != 0
+ && affix.allowsContinuation(positionalFlag(position)));
+ }
+
+ /**
+ * Checks whether an affix may sit at a compound-internal boundary: it carries the
+ * {@code COMPOUNDPERMITFLAG} among its continuation classes. Without the flag a
+ * suffix fits only the last part and a prefix only the first.
+ *
+ * @param affix The affix rule applied to the part.
+ * @return {@code true} if the affix may face another part.
+ */
+ boolean permitsInside(Affix affix) {
+ return compoundPermit != 0 && affix.allowsContinuation(compoundPermit);
+ }
+
+ /**
+ * Checks whether an affix bars its derived form from compounds altogether: it
+ * carries the {@code COMPOUNDFORBIDFLAG} among its continuation classes.
+ *
+ * @param affix The affix rule applied to the part.
+ * @return {@code true} if the affixed form may not join a compound.
+ */
+ boolean forbidsInCompound(Affix affix) {
+ return compoundForbid != 0 && affix.allowsContinuation(compoundForbid);
+ }
+
+ /**
+ * Checks whether any of a word's flag sets is forbidden, which a dictionary uses
+ * to block one specific ill-formed compound while its parts stay productive.
+ *
+ * @param flagSets The word's flag sets from {@link #lookup(String)}.
+ * @return {@code true} if some homonym carries the forbidden-word flag.
+ */
+ boolean anyForbidden(List<int[]> flagSets) {
+ return hasFlag(flagSets, forbiddenWord);
+ }
+
+ /**
+ * Checks whether any of a word's flag sets carries a flag.
+ *
+ * @param flagSets The flag sets from {@link #lookup(String)}.
+ * @param flag The flag to look for.
+ * @return {@code true} if some flag set contains the flag.
+ */
+ static boolean hasFlag(List<int[]> flagSets, int flag) {
+ for (final int[] flags : flagSets) {
+ if (contains(flags, flag)) {
+ return true;
+ }
+ }
+ return false;
+ }
+
+ /**
+ * Checks one flag set for a flag. An undeclared flag, encoded as {@code 0}, is
+ * carried by no entry.
+ *
+ * @param flags One entry's flag set.
+ * @param flag The flag to look for.
+ * @return {@code true} if the set contains the flag.
+ */
+ private static boolean contains(int[] flags, int flag) {
+ if (flag == 0) {
+ return false;
+ }
+ for (final int candidate : flags) {
+ if (candidate == flag) {
+ return true;
+ }
+ }
+ return false;
+ }
+
+ /**
+ * Checks whether a listed word is valid on its own: some homonym's flag set carries
+ * none of the blocking flags. An entry whose every flag set is marked
+ * {@code NEEDAFFIX} is a virtual stem that exists only to be affixed, one marked
+ * {@code ONLYINCOMPOUND} appears only inside compounds, and one marked
+ * {@code FORBIDDENWORD} is listed to be blocked; none of them is a word by itself.
+ *
+ * @param flagSets The word's flag sets from {@link #lookup(String)}.
+ * @return {@code true} if some homonym stands on its own.
+ */
+ boolean validStandalone(List<int[]> flagSets) {
+ for (final int[] flags : flagSets) {
+ if (!contains(flags, needAffix) && !contains(flags, onlyInCompound)
+ && !contains(flags, forbiddenWord)) {
+ return true;
+ }
+ }
+ return false;
+ }
+
+ /**
+ * Checks whether some homonym supports an affix analysis: its flag set carries the
+ * affix's flag and is neither compound-only nor forbidden. A {@code NEEDAFFIX} set
+ * does support the analysis, because the removed affix is exactly what the virtual
+ * stem needs.
+ *
+ * @param flagSets The stem's flag sets from {@link #lookup(String)}.
+ * @param flag The removed affix's flag.
+ * @return {@code true} if some homonym carries the flag and may stand affixed.
+ */
+ boolean supports(List<int[]> flagSets, int flag) {
+ for (final int[] flags : flagSets) {
+ if (contains(flags, flag) && !contains(flags, onlyInCompound)
+ && !contains(flags, forbiddenWord)) {
+ return true;
+ }
+ }
+ return false;
+ }
+
+ /**
+ * Checks whether some homonym supports a cross-product analysis: one flag set
+ * carries both removed affixes' flags and is neither compound-only nor forbidden.
+ * The two flags must sit in the same set, because homonyms are separate words and
+ * each removal must be licensed by the same one.
+ *
+ * @param flagSets The stem's flag sets from {@link #lookup(String)}.
+ * @param prefixFlag The removed prefix's flag.
+ * @param suffixFlag The removed suffix's flag.
+ * @return {@code true} if some homonym carries both flags and may stand affixed.
+ */
+ boolean supports(List<int[]> flagSets, int prefixFlag, int suffixFlag) {
+ for (final int[] flags : flagSets) {
+ if (contains(flags, prefixFlag) && contains(flags, suffixFlag)
+ && !contains(flags, onlyInCompound) && !contains(flags, forbiddenWord)) {
+ return true;
+ }
+ }
+ return false;
+ }
+
+ /**
+ * Checks whether a form made with this affix alone is still a virtual stem: the
+ * affix carries the {@code NEEDAFFIX} flag among its continuation classes, so a
+ * further affix must join before the form is a word.
+ *
+ * @param affix The affix rule to inspect.
+ * @return {@code true} if the affix alone does not finish a word.
+ */
+ boolean needsFurtherAffix(Affix affix) {
+ return needAffix != 0 && affix.allowsContinuation(needAffix);
+ }
+
+ /**
+ * Checks whether an affix applies only inside compounds: it carries the
+ * {@code ONLYINCOMPOUND} flag among its continuation classes.
+ *
+ * @param affix The affix rule to inspect.
+ * @return {@code true} if the affix never applies to a standalone word.
+ */
+ boolean compoundOnly(Affix affix) {
+ return onlyInCompound != 0 && affix.allowsContinuation(onlyInCompound);
+ }
+
+ /**
+ * Checks whether an affix is one half of a circumfix: it carries the
+ * {@code CIRCUMFIX} flag among its continuation classes, so it is only valid on a
+ * word that also carries a circumfix-marked affix of the other kind, the German
+ * {@code ge...t} participle being the model.
+ *
+ * @param affix The affix rule to inspect.
+ * @return {@code true} if the affix never applies without its other half.
+ */
+ boolean circumfixOnly(Affix affix) {
+ return circumfix != 0 && affix.allowsContinuation(circumfix);
+ }
+
+ /**
+ * Finds the {@code SET} declaration by scanning the raw affix bytes as ASCII, which
+ * is safe because the declaration itself is ASCII in every supported encoding. Both
+ * files are then decoded with the declared charset.
+ *
+ * @param affixBytes The raw affix file content.
+ * @return The declared charset, or UTF-8 when no declaration is present.
+ * @throws IOException Thrown if the declared encoding name is not supported.
+ */
+ private static Charset declaredCharset(byte[] affixBytes) throws IOException {
+ final String ascii = new String(affixBytes, StandardCharsets.US_ASCII);
+ for (final String line : splitLines(ascii)) {
+ final String trimmed = trim(line);
+ if (trimmed.startsWith("SET ") || trimmed.startsWith("SET\t")) {
Review Comment:
Please declare `"SET"` and `"SET\t"`as a named constant for better re-use (in loops etc.)
##########
opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/hunspell/HunspellDictionaryDownload.java:
##########
@@ -0,0 +1,89 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package opennlp.tools.stemmer.hunspell;
+
+import java.io.IOException;
+import java.nio.file.Path;
+
+import opennlp.tools.util.DictionaryCatalog;
+
+/**
+ * Opt-in download of Hunspell {@code .aff}/{@code .dic} pairs (and their license
+ * readme) from {@link DictionaryCatalog}. Requires
+ * {@code -Dopennlp.download.remote=true}. OpenNLP never bundles dictionary data.
+ *
+ * @since 3.0.0
+ */
+public final class HunspellDictionaryDownload {
+
+ private HunspellDictionaryDownload() {
+ }
+
+ /**
+ * Downloads the cataloged {@code .aff}, {@code .dic}, and readme files for
+ * {@code dictionaryId} into {@code targetDirectory}.
+ *
+ * @param dictionaryId The catalog dictionary name, for example {@code en_US}.
+ * Must not be {@code null}.
+ * @param targetDirectory The directory to write into; created when absent. Must not
+ * be {@code null}.
+ * @throws IOException Thrown if remote downloads are disabled, a catalog entry is
+ * missing, or verification fails.
+ * @throws IllegalArgumentException Thrown if a parameter is {@code null}.
+ */
+ public static void downloadFromCatalog(String dictionaryId, Path targetDirectory)
+ throws IOException {
+ if (dictionaryId == null) {
+ throw new IllegalArgumentException("dictionaryId must not be null");
+ }
+ if (targetDirectory == null) {
+ throw new IllegalArgumentException("targetDirectory must not be null");
+ }
+ final DictionaryCatalog catalog = DictionaryCatalog.loadDefault();
+ final String prefix = "hunspell." + dictionaryId + ".";
+ download(catalog, prefix + "aff", targetDirectory);
Review Comment:
Can we declare a constant for "aff" and "dic" suffixes somewhere and refer to it in the Hunspell related classes and tests? Would the class `HunspellDictionary` be a good place to declare those public constants? Do we instead require an _opennlp-api_ interface for `HunspellDictionary` and make the class the `DefaultHunspellDictionary` its impl?
##########
opennlp-core/opennlp-runtime/src/main/resources/opennlp/tools/util/dictionary-catalog.properties:
##########
Review Comment:
Should this be under /src/main/test ?? This shouldn't be bundled with opennlp-runtime as it might outdate over time and be inconsistent with future code changes... Better document it via a test that uses it "now" and refer to the way users can setup there custom properties locally....
##########
opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/DownloadUtil.java:
##########
@@ -63,6 +70,34 @@ public class DownloadUtil {
System.getProperty("OPENNLP_DOWNLOAD_MODEL_PATH", "models/ud-models-1.3/");
private static final String OPENNLP_DOWNLOAD_HOME = "OPENNLP_DOWNLOAD_HOME";
+ /**
+ * System property that must be {@code true} before a
+ * {@link DictionaryCatalog} entry may be fetched. Explicit
+ * {@link #download(URI, Path, String)} calls do not require it: the caller already
+ * supplied the URI and digest.
+ */
+ public static final String REMOTE_DOWNLOAD_PROPERTY = "opennlp.download.remote";
+
+ /**
+ * System property for overriding {@link #MAX_DOWNLOAD_BYTES}. Set at JVM startup,
+ * e.g. {@code -Dopennlp.download.max.bytes=2147483648} for dictionaries larger than
+ * the default ceiling. Falls back to the default if absent, non-numeric, or not
+ * positive.
+ */
+ public static final String MAX_DOWNLOAD_BYTES_PROPERTY = "opennlp.download.max.bytes";
+
+ /**
+ * Inclusive ceiling on bytes buffered for one {@link #download(URI, Path, String)},
+ * 512 MiB unless overridden via {@link #MAX_DOWNLOAD_BYTES_PROPERTY}.
Review Comment:
How is this default value (512 MB) motivated here? Wouldn't 64 MB be sufficient? Please re-consider this value and change it accordingly if no one finds a proper rationale for that huge value.
##########
opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/hunspell/HunspellDictionary.java:
##########
@@ -0,0 +1,1254 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package opennlp.tools.stemmer.hunspell;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.charset.Charset;
+import java.nio.charset.IllegalCharsetNameException;
+import java.nio.charset.StandardCharsets;
+import java.nio.charset.UnsupportedCharsetException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import opennlp.tools.commons.ThreadSafe;
+import opennlp.tools.util.StringUtil;
+
+/**
+ * An immutable, in-memory Hunspell-format dictionary: the word list of a {@code .dic}
+ * file and the prefix and suffix rules of its {@code .aff} companion, loaded from
+ * user-supplied files. The engine implements the documented format directly; no
+ * dictionary data is bundled, dictionaries are supplied by the user.
+ *
+ * <p>Supported affix features: {@code PFX} and {@code SFX} rules with strip strings,
+ * character-class conditions, and cross-product combination of one prefix with one
+ * suffix; twofold suffixes through the continuation classes on suffix rules;
+ * {@code FLAG} modes {@code char} (default), {@code UTF-8}, {@code long}, and
+ * {@code num}; the {@code AF} flag alias table; the {@code SET} encoding declaration;
+ * compound decomposition under {@code COMPOUNDFLAG}, the positional
+ * {@code COMPOUNDBEGIN}/{@code COMPOUNDMIDDLE}/{@code COMPOUNDEND} flags,
+ * {@code COMPOUNDMIN}, {@code COMPOUNDWORDMAX}, {@code COMPOUNDPERMITFLAG},
+ * {@code COMPOUNDFORBIDFLAG}, and the {@code CHECKCOMPOUNDDUP},
+ * {@code CHECKCOMPOUNDCASE}, and {@code CHECKCOMPOUNDTRIPLE} declarations, with
+ * compound parts standing on their entries alone or on an entry plus one affix; the
+ * blocking flags
+ * {@code NEEDAFFIX} (with its historical alias {@code PSEUDOROOT}),
+ * {@code ONLYINCOMPOUND}, and {@code FORBIDDENWORD}, which suppress analyses the
+ * dictionary marks as virtual stems, compound-only parts, or forbidden words; and
+ * {@code CIRCUMFIX}, which binds marked prefix and suffix halves to one another; and
+ * the {@code FULLSTRIP} declaration, without which a rule that strips a whole stem is
+ * not applied, matching hunspell.
+ * Directives that would change stems when ignored ({@code ICONV}, {@code OCONV},
+ * {@code COMPLEXPREFIXES}, {@code COMPOUNDRULE}, {@code IGNORE},
+ * {@code KEEPCASE}) are rejected at load time. Cosmetic tables such as
+ * {@code REP}, {@code MAP}, and {@code KEY} are skipped, so analyses that would need
+ * them are missed rather than invented.</p>
+ *
+ * <p>Instances are immutable and safe to share between threads.</p>
+ *
+ * @see HunspellStemmer
+ * @see HunspellStemmerFactory
+ * @since 3.0.0
+ */
+@ThreadSafe
+public final class HunspellDictionary {
+
+ /**
+ * Inclusive upper bound on bytes buffered from one affix or dictionary stream
+ * during {@link #load(InputStream, InputStream)}. Larger streams fail with
+ * {@link IOException}.
Review Comment:
Please add a sentence on what the (current) default value is: "64 MB"
##########
opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/hunspell/HunspellDictionary.java:
##########
@@ -0,0 +1,1254 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package opennlp.tools.stemmer.hunspell;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.charset.Charset;
+import java.nio.charset.IllegalCharsetNameException;
+import java.nio.charset.StandardCharsets;
+import java.nio.charset.UnsupportedCharsetException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import opennlp.tools.commons.ThreadSafe;
+import opennlp.tools.util.StringUtil;
+
+/**
+ * An immutable, in-memory Hunspell-format dictionary: the word list of a {@code .dic}
+ * file and the prefix and suffix rules of its {@code .aff} companion, loaded from
+ * user-supplied files. The engine implements the documented format directly; no
+ * dictionary data is bundled, dictionaries are supplied by the user.
+ *
+ * <p>Supported affix features: {@code PFX} and {@code SFX} rules with strip strings,
+ * character-class conditions, and cross-product combination of one prefix with one
+ * suffix; twofold suffixes through the continuation classes on suffix rules;
+ * {@code FLAG} modes {@code char} (default), {@code UTF-8}, {@code long}, and
+ * {@code num}; the {@code AF} flag alias table; the {@code SET} encoding declaration;
+ * compound decomposition under {@code COMPOUNDFLAG}, the positional
+ * {@code COMPOUNDBEGIN}/{@code COMPOUNDMIDDLE}/{@code COMPOUNDEND} flags,
+ * {@code COMPOUNDMIN}, {@code COMPOUNDWORDMAX}, {@code COMPOUNDPERMITFLAG},
+ * {@code COMPOUNDFORBIDFLAG}, and the {@code CHECKCOMPOUNDDUP},
+ * {@code CHECKCOMPOUNDCASE}, and {@code CHECKCOMPOUNDTRIPLE} declarations, with
+ * compound parts standing on their entries alone or on an entry plus one affix; the
+ * blocking flags
+ * {@code NEEDAFFIX} (with its historical alias {@code PSEUDOROOT}),
+ * {@code ONLYINCOMPOUND}, and {@code FORBIDDENWORD}, which suppress analyses the
+ * dictionary marks as virtual stems, compound-only parts, or forbidden words; and
+ * {@code CIRCUMFIX}, which binds marked prefix and suffix halves to one another; and
+ * the {@code FULLSTRIP} declaration, without which a rule that strips a whole stem is
+ * not applied, matching hunspell.
+ * Directives that would change stems when ignored ({@code ICONV}, {@code OCONV},
+ * {@code COMPLEXPREFIXES}, {@code COMPOUNDRULE}, {@code IGNORE},
+ * {@code KEEPCASE}) are rejected at load time. Cosmetic tables such as
+ * {@code REP}, {@code MAP}, and {@code KEY} are skipped, so analyses that would need
+ * them are missed rather than invented.</p>
+ *
+ * <p>Instances are immutable and safe to share between threads.</p>
+ *
+ * @see HunspellStemmer
+ * @see HunspellStemmerFactory
+ * @since 3.0.0
+ */
+@ThreadSafe
+public final class HunspellDictionary {
+
+ /**
+ * Inclusive upper bound on bytes buffered from one affix or dictionary stream
+ * during {@link #load(InputStream, InputStream)}. Larger streams fail with
+ * {@link IOException}.
+ */
+ public static final int MAX_STREAM_BYTES = 64 * 1024 * 1024;
+
+ /**
+ * One parsed affix rule of a {@code PFX} or {@code SFX} block.
+ *
+ * @param flag The flag naming the rule's block, which an entry carries to accept it.
+ * @param crossProduct Whether the rule may combine with an affix of the opposite kind.
+ * @param strip The stem material the rule replaces, restored during analysis.
+ * @param affix The surface material the rule adds to the stem.
+ * @param condition The condition the stem must satisfy for the rule to apply.
+ * @param continuation The flags of the further affixes that may stack on this one.
+ */
+ record Affix(int flag, boolean crossProduct, String strip, String affix,
+ AffixCondition condition, int[] continuation) {
+
+ /**
+ * Checks whether a further affix may stack on this one.
+ *
+ * @param otherFlag The stacking affix's flag.
+ * @return {@code true} if this affix's continuation classes allow it.
+ */
+ boolean allowsContinuation(int otherFlag) {
+ for (final int candidate : continuation) {
+ if (candidate == otherFlag) {
+ return true;
+ }
+ }
+ return false;
+ }
+ }
+
+ /** The place a part takes in a compound, deciding which positional flag admits it. */
+ enum CompoundPosition {
+ /** The first part. */
+ BEGIN,
+ /** Any part between the first and the last. */
+ MIDDLE,
+ /** The last part. */
+ END
+ }
+
+ /** The shared empty bucket answered for characters no affix rule is keyed under. */
+ private static final List<Affix> NO_AFFIXES = List.of();
+
+ /** The line tag of a prefix block and of every rule line inside it. */
+ private static final String PREFIX_TAG = "PFX";
+
+ /** The line tag of a suffix block and of every rule line inside it. */
+ private static final String SUFFIX_TAG = "SFX";
+
+ /** The affix format's marker for absent strip or affix material. */
+ private static final String NO_MATERIAL = "0";
+
+ private final Map<String, List<int[]>> entries;
+ private final BoundaryIndex suffixesByLast;
+ private final List<Affix> suffixesWithoutMaterial;
+ private final BoundaryIndex prefixesByFirst;
+ private final List<Affix> prefixesWithoutMaterial;
+ private final int compoundFlag;
+ private final int compoundBegin;
+ private final int compoundEnd;
+ private final int compoundMin;
+ private final int needAffix;
+ private final int onlyInCompound;
+ private final int forbiddenWord;
+ private final int circumfix;
+ private final int compoundMiddle;
+ private final int compoundPermit;
+ private final int compoundForbid;
+ private final int compoundWordMax;
+ private final boolean checkCompoundDup;
+ private final boolean checkCompoundCase;
+ private final boolean checkCompoundTriple;
+ private final boolean fullStrip;
+
+ /**
+ * Initializes the dictionary from the two parsed files.
+ *
+ * @param entries The words mapped to the flag sets of their entries.
+ * @param affix The parsed affix file.
+ */
+ private HunspellDictionary(Map<String, List<int[]>> entries, AffixFile affix) {
+ this.compoundFlag = affix.compoundFlag;
+ this.compoundBegin = affix.compoundBegin;
+ this.compoundEnd = affix.compoundEnd;
+ this.compoundMin = affix.compoundMin;
+ this.needAffix = affix.needAffix;
+ this.onlyInCompound = affix.onlyInCompound;
+ this.forbiddenWord = affix.forbiddenWord;
+ this.circumfix = affix.circumfix;
+ this.compoundMiddle = affix.compoundMiddle;
+ this.compoundPermit = affix.compoundPermit;
+ this.compoundForbid = affix.compoundForbid;
+ this.compoundWordMax = affix.compoundWordMax;
+ this.checkCompoundDup = affix.checkCompoundDup;
+ this.checkCompoundCase = affix.checkCompoundCase;
+ this.checkCompoundTriple = affix.checkCompoundTriple;
+ this.fullStrip = affix.fullStrip;
+ this.entries = entries;
+ // A material-bearing rule can only be undone from a word whose boundary
+ // character matches its affix material, so bucketing by that character
+ // narrows each scan to one bucket plus the strip-only rules.
+ final List<Affix> suffixesWithout = new ArrayList<>();
+ this.suffixesByLast = bucketByBoundary(affix.suffixes, true, suffixesWithout);
+ this.suffixesWithoutMaterial = List.copyOf(suffixesWithout);
+ final List<Affix> prefixesWithout = new ArrayList<>();
+ this.prefixesByFirst = bucketByBoundary(affix.prefixes, false, prefixesWithout);
+ this.prefixesWithoutMaterial = List.copyOf(prefixesWithout);
+ }
+
+ /**
+ * An immutable index of affix rules keyed by the boundary code point of their affix
+ * material, answering each lookup by binary search so the per-word scans in
+ * {@link HunspellStemmer} allocate nothing.
+ */
+ private static final class BoundaryIndex {
+
+ /** The boundary code points, sorted ascending. */
+ private final int[] boundaries;
+ /** The rule bucket for each boundary, aligned with {@link #boundaries}. */
+ private final List<List<Affix>> buckets;
+
+ /**
+ * Initializes the index from mutable buckets, freezing each one.
+ *
+ * @param byBoundary The rule buckets keyed by boundary code point.
+ */
+ private BoundaryIndex(Map<Integer, List<Affix>> byBoundary) {
+ this.boundaries = new int[byBoundary.size()];
+ int b = 0;
+ for (final Integer boundary : byBoundary.keySet()) {
+ boundaries[b++] = boundary;
+ }
+ Arrays.sort(boundaries);
+ this.buckets = new ArrayList<>(boundaries.length);
+ for (final int boundary : boundaries) {
+ buckets.add(List.copyOf(byBoundary.get(boundary)));
+ }
+ }
+
+ /**
+ * The rules bucketed under a boundary code point.
+ *
+ * @param codePoint The boundary code point to look up.
+ * @return The bucket, possibly empty. Never {@code null}.
+ */
+ List<Affix> bucket(int codePoint) {
+ final int index = Arrays.binarySearch(boundaries, codePoint);
+ return index >= 0 ? buckets.get(index) : NO_AFFIXES;
+ }
+ }
+
+ /**
+ * Buckets affix rules by the boundary code point of their affix material, the last
+ * code point for a suffix rule and the first for a prefix rule.
+ *
+ * @param rules The rules of one kind, in file order.
+ * @param suffix Whether the rules are suffix rules.
+ * @param withoutMaterial Collects the rules with empty affix material, which no
+ * boundary code point keys.
+ * @return The rules indexed by their boundary code point. Never {@code null}.
+ */
+ private static BoundaryIndex bucketByBoundary(List<Affix> rules,
+ boolean suffix, List<Affix> withoutMaterial) {
+ final Map<Integer, List<Affix>> byBoundary = new HashMap<>();
+ for (final Affix rule : rules) {
+ final String material = rule.affix();
+ if (material.isEmpty()) {
+ withoutMaterial.add(rule);
+ } else {
+ final int boundary = suffix
+ ? material.codePointBefore(material.length())
+ : material.codePointAt(0);
+ byBoundary.computeIfAbsent(boundary, key -> new ArrayList<>()).add(rule);
+ }
+ }
+ return new BoundaryIndex(byBoundary);
+ }
+
+ /**
+ * Loads a dictionary from its two files.
+ *
+ * @param affixFile The {@code .aff} affix file. Must not be {@code null}.
+ * @param dictionaryFile The {@code .dic} word list. Must not be {@code null}.
+ * @return The loaded dictionary. Never {@code null}.
+ * @throws IOException Thrown if reading fails or a file is malformed.
+ * @throws IllegalArgumentException Thrown if a parameter is {@code null}.
+ */
+ public static HunspellDictionary load(Path affixFile, Path dictionaryFile)
+ throws IOException {
+ if (affixFile == null) {
+ throw new IllegalArgumentException("affixFile must not be null");
+ }
+ if (dictionaryFile == null) {
+ throw new IllegalArgumentException("dictionaryFile must not be null");
+ }
+ try (InputStream affix = Files.newInputStream(affixFile);
+ InputStream dictionary = Files.newInputStream(dictionaryFile)) {
+ return load(affix, dictionary);
+ }
+ }
+
+ /**
+ * Loads a dictionary from its two streams. Each stream is buffered up to
+ * {@link #MAX_STREAM_BYTES} bytes; a larger stream fails with {@link IOException}.
+ *
+ * @param affixStream The {@code .aff} affix content. Must not be {@code null}. Not
+ * closed.
+ * @param dictionaryStream The {@code .dic} word list content. Must not be
+ * {@code null}. Not closed.
+ * @return The loaded dictionary. Never {@code null}.
+ * @throws IOException Thrown if reading fails, a stream exceeds
+ * {@link #MAX_STREAM_BYTES}, or the content is malformed.
+ * @throws IllegalArgumentException Thrown if a parameter is {@code null}.
+ */
+ public static HunspellDictionary load(InputStream affixStream,
+ InputStream dictionaryStream) throws IOException {
+ if (affixStream == null) {
+ throw new IllegalArgumentException("affixStream must not be null");
+ }
+ if (dictionaryStream == null) {
+ throw new IllegalArgumentException("dictionaryStream must not be null");
+ }
+ final byte[] affixBytes = readBounded(affixStream, MAX_STREAM_BYTES, "affix stream");
+ final Charset charset = declaredCharset(affixBytes);
+ final AffixFile affix = parseAffix(new String(affixBytes, charset));
+ final Map<String, List<int[]>> entries = parseWordList(
+ new String(readBounded(dictionaryStream, MAX_STREAM_BYTES, "dictionary stream"),
+ charset),
+ affix.flagMode, affix.flagAliases);
+ return new HunspellDictionary(entries, affix);
+ }
+
+ /**
+ * Reads an input stream into a byte array, failing when more than {@code maxBytes}
+ * arrive.
+ *
+ * @param in The stream to read. Not closed.
+ * @param maxBytes The inclusive upper bound on buffered bytes.
+ * @param label The stream name used in the error message.
+ * @return The buffered bytes. Never {@code null}.
+ * @throws IOException Thrown if reading fails or the stream exceeds {@code maxBytes}.
+ */
+ static byte[] readBounded(InputStream in, int maxBytes, String label)
+ throws IOException {
+ final byte[] chunk = new byte[8192];
+ byte[] buffer = new byte[Math.min(8192, maxBytes)];
+ int size = 0;
+ int n;
+ while ((n = in.read(chunk)) >= 0) {
+ if (size + n > maxBytes) {
+ throw new IOException(label + " size exceeds safe limit of " + maxBytes);
+ }
+ if (size + n > buffer.length) {
+ buffer = Arrays.copyOf(buffer, Math.min(maxBytes, Math.max(buffer.length * 2, size + n)));
+ }
+ System.arraycopy(chunk, 0, buffer, size, n);
+ size += n;
+ }
+ return size == buffer.length ? buffer : Arrays.copyOf(buffer, size);
+ }
+
+ /**
+ * Looks up a word's flag sets.
+ *
+ * @param word The word exactly as listed.
+ * @return The flag sets of all matching entries, or {@code null} when absent.
+ */
+ List<int[]> lookup(String word) {
+ final List<int[]> found = entries.get(word);
+ if (found == null) {
+ return null;
+ }
+ final List<int[]> copy = new ArrayList<>(found.size());
+ for (final int[] flags : found) {
+ copy.add(flags.clone());
+ }
+ return copy;
+ }
+
+ /**
+ * The suffix rules whose affix material ends in the given code point, which are the
+ * only material-bearing rules that can be undone from a word ending in it.
+ *
+ * @param last The word's last code point.
+ * @return The bucket, possibly empty. Never {@code null}.
+ */
+ List<Affix> suffixesEndingWith(int last) {
+ return suffixesByLast.bucket(last);
+ }
+
+ /** {@return the strip-only suffix rules, applicable to any word} Never {@code null}. */
+ List<Affix> suffixesWithoutMaterial() {
+ return suffixesWithoutMaterial;
+ }
+
+ /**
+ * The prefix rules whose affix material starts with the given code point, which are
+ * the only material-bearing rules that can be undone from a word starting with it.
+ *
+ * @param first The word's first code point.
+ * @return The bucket, possibly empty. Never {@code null}.
+ */
+ List<Affix> prefixesStartingWith(int first) {
+ return prefixesByFirst.bucket(first);
+ }
+
+ /** {@return the strip-only prefix rules, applicable to any word} Never {@code null}. */
+ List<Affix> prefixesWithoutMaterial() {
+ return prefixesWithoutMaterial;
+ }
+
+ /** {@return whether the affix file declares any compounding flag at all} */
+ boolean compoundsDeclared() {
+ return compoundFlag != 0 || compoundBegin != 0 || compoundEnd != 0
+ || compoundMiddle != 0;
+ }
+
+ /** {@return the smallest length a compound part may have} At least {@code 1}. */
+ int compoundMin() {
+ return compoundMin;
+ }
+
+ /** {@return the largest number of parts a compound may have} {@code 0} is unbounded. */
+ int compoundWordMax() {
+ return compoundWordMax;
+ }
+
+ /** {@return whether {@code CHECKCOMPOUNDDUP} forbids a part repeating its neighbor} */
+ boolean checkCompoundDup() {
+ return checkCompoundDup;
+ }
+
+ /** {@return whether {@code CHECKCOMPOUNDCASE} forbids uppercase at part boundaries} */
+ boolean checkCompoundCase() {
+ return checkCompoundCase;
+ }
+
+ /** {@return whether {@code CHECKCOMPOUNDTRIPLE} forbids triple letters at boundaries} */
+ boolean checkCompoundTriple() {
+ return checkCompoundTriple;
+ }
+
+ /** {@return whether {@code FULLSTRIP} allows an affix rule to strip a whole stem} */
+ boolean fullStrip() {
+ return fullStrip;
+ }
+
+ /**
+ * The flag admitting a part at a compound position, next to the general
+ * compounding flag.
+ *
+ * @param position The part's place in the compound.
+ * @return The dedicated positional flag, or {@code 0} when undeclared.
+ */
+ private int positionalFlag(CompoundPosition position) {
+ return switch (position) {
+ case BEGIN -> compoundBegin;
+ case MIDDLE -> compoundMiddle;
+ case END -> compoundEnd;
+ };
+ }
+
+ /**
+ * Checks whether a listed word may stand at a compound position: some homonym's
+ * flag set carries the general compounding flag or the position's dedicated flag
+ * and is not forbidden. A compound-only or virtual-stem homonym may take the
+ * position; that is what those flags permit.
+ *
+ * @param flagSets The word's flag sets from {@link #lookup(String)}.
+ * @param position The part's place in the compound.
+ * @return {@code true} if the word may stand at the position.
+ */
+ boolean mayStand(List<int[]> flagSets, CompoundPosition position) {
+ final int positional = positionalFlag(position);
+ for (final int[] flags : flagSets) {
+ if ((contains(flags, compoundFlag) || contains(flags, positional))
+ && !contains(flags, forbiddenWord) && !contains(flags, needAffix)) {
+ return true;
+ }
+ }
+ return false;
+ }
+
+ /**
+ * Checks whether some homonym supports an affixed compound part: its flag set
+ * carries the removed affix's flag, is not forbidden, and either the affix itself
+ * admits the position or the set carries the compounding or positional flag.
+ *
+ * @param flagSets The part stem's flag sets from {@link #lookup(String)}.
+ * @param affixFlag The removed affix's flag.
+ * @param position The part's place in the compound.
+ * @param affixAdmits Whether the affix's continuation classes admit the position,
+ * from {@link #affixAdmits(Affix, CompoundPosition)}.
+ * @return {@code true} if some homonym stands affixed at the position.
+ */
+ boolean supportsPart(List<int[]> flagSets, int affixFlag, CompoundPosition position,
+ boolean affixAdmits) {
+ final int positional = positionalFlag(position);
+ for (final int[] flags : flagSets) {
+ if (contains(flags, affixFlag) && !contains(flags, forbiddenWord)
+ && (affixAdmits || contains(flags, compoundFlag)
+ || contains(flags, positional))) {
+ return true;
+ }
+ }
+ return false;
+ }
+
+ /**
+ * Checks whether an affix admits its derived form at a compound position: its
+ * continuation classes carry the general compounding flag or the position's
+ * dedicated flag. Published dictionaries position their linking forms this way,
+ * through zero or dash suffixes whose continuation classes hold the positional
+ * flags.
+ *
+ * @param affix The affix rule applied to the part.
+ * @param position The part's place in the compound.
+ * @return {@code true} if the affixed form may stand at the position.
+ */
+ boolean affixAdmits(Affix affix, CompoundPosition position) {
+ return (compoundFlag != 0 && affix.allowsContinuation(compoundFlag))
+ || (positionalFlag(position) != 0
+ && affix.allowsContinuation(positionalFlag(position)));
+ }
+
+ /**
+ * Checks whether an affix may sit at a compound-internal boundary: it carries the
+ * {@code COMPOUNDPERMITFLAG} among its continuation classes. Without the flag a
+ * suffix fits only the last part and a prefix only the first.
+ *
+ * @param affix The affix rule applied to the part.
+ * @return {@code true} if the affix may face another part.
+ */
+ boolean permitsInside(Affix affix) {
+ return compoundPermit != 0 && affix.allowsContinuation(compoundPermit);
+ }
+
+ /**
+ * Checks whether an affix bars its derived form from compounds altogether: it
+ * carries the {@code COMPOUNDFORBIDFLAG} among its continuation classes.
+ *
+ * @param affix The affix rule applied to the part.
+ * @return {@code true} if the affixed form may not join a compound.
+ */
+ boolean forbidsInCompound(Affix affix) {
+ return compoundForbid != 0 && affix.allowsContinuation(compoundForbid);
+ }
+
+ /**
+ * Checks whether any of a word's flag sets is forbidden, which a dictionary uses
+ * to block one specific ill-formed compound while its parts stay productive.
+ *
+ * @param flagSets The word's flag sets from {@link #lookup(String)}.
+ * @return {@code true} if some homonym carries the forbidden-word flag.
+ */
+ boolean anyForbidden(List<int[]> flagSets) {
+ return hasFlag(flagSets, forbiddenWord);
+ }
+
+ /**
+ * Checks whether any of a word's flag sets carries a flag.
+ *
+ * @param flagSets The flag sets from {@link #lookup(String)}.
+ * @param flag The flag to look for.
+ * @return {@code true} if some flag set contains the flag.
+ */
+ static boolean hasFlag(List<int[]> flagSets, int flag) {
+ for (final int[] flags : flagSets) {
+ if (contains(flags, flag)) {
+ return true;
+ }
+ }
+ return false;
+ }
+
+ /**
+ * Checks one flag set for a flag. An undeclared flag, encoded as {@code 0}, is
+ * carried by no entry.
+ *
+ * @param flags One entry's flag set.
+ * @param flag The flag to look for.
+ * @return {@code true} if the set contains the flag.
+ */
+ private static boolean contains(int[] flags, int flag) {
+ if (flag == 0) {
+ return false;
+ }
+ for (final int candidate : flags) {
+ if (candidate == flag) {
+ return true;
+ }
+ }
+ return false;
+ }
+
+ /**
+ * Checks whether a listed word is valid on its own: some homonym's flag set carries
+ * none of the blocking flags. An entry whose every flag set is marked
+ * {@code NEEDAFFIX} is a virtual stem that exists only to be affixed, one marked
+ * {@code ONLYINCOMPOUND} appears only inside compounds, and one marked
+ * {@code FORBIDDENWORD} is listed to be blocked; none of them is a word by itself.
+ *
+ * @param flagSets The word's flag sets from {@link #lookup(String)}.
+ * @return {@code true} if some homonym stands on its own.
+ */
+ boolean validStandalone(List<int[]> flagSets) {
+ for (final int[] flags : flagSets) {
+ if (!contains(flags, needAffix) && !contains(flags, onlyInCompound)
+ && !contains(flags, forbiddenWord)) {
+ return true;
+ }
+ }
+ return false;
+ }
+
+ /**
+ * Checks whether some homonym supports an affix analysis: its flag set carries the
+ * affix's flag and is neither compound-only nor forbidden. A {@code NEEDAFFIX} set
+ * does support the analysis, because the removed affix is exactly what the virtual
+ * stem needs.
+ *
+ * @param flagSets The stem's flag sets from {@link #lookup(String)}.
+ * @param flag The removed affix's flag.
+ * @return {@code true} if some homonym carries the flag and may stand affixed.
+ */
+ boolean supports(List<int[]> flagSets, int flag) {
+ for (final int[] flags : flagSets) {
+ if (contains(flags, flag) && !contains(flags, onlyInCompound)
+ && !contains(flags, forbiddenWord)) {
+ return true;
+ }
+ }
+ return false;
+ }
+
+ /**
+ * Checks whether some homonym supports a cross-product analysis: one flag set
+ * carries both removed affixes' flags and is neither compound-only nor forbidden.
+ * The two flags must sit in the same set, because homonyms are separate words and
+ * each removal must be licensed by the same one.
+ *
+ * @param flagSets The stem's flag sets from {@link #lookup(String)}.
+ * @param prefixFlag The removed prefix's flag.
+ * @param suffixFlag The removed suffix's flag.
+ * @return {@code true} if some homonym carries both flags and may stand affixed.
+ */
+ boolean supports(List<int[]> flagSets, int prefixFlag, int suffixFlag) {
+ for (final int[] flags : flagSets) {
+ if (contains(flags, prefixFlag) && contains(flags, suffixFlag)
+ && !contains(flags, onlyInCompound) && !contains(flags, forbiddenWord)) {
+ return true;
+ }
+ }
+ return false;
+ }
+
+ /**
+ * Checks whether a form made with this affix alone is still a virtual stem: the
+ * affix carries the {@code NEEDAFFIX} flag among its continuation classes, so a
+ * further affix must join before the form is a word.
+ *
+ * @param affix The affix rule to inspect.
+ * @return {@code true} if the affix alone does not finish a word.
+ */
+ boolean needsFurtherAffix(Affix affix) {
+ return needAffix != 0 && affix.allowsContinuation(needAffix);
+ }
+
+ /**
+ * Checks whether an affix applies only inside compounds: it carries the
+ * {@code ONLYINCOMPOUND} flag among its continuation classes.
+ *
+ * @param affix The affix rule to inspect.
+ * @return {@code true} if the affix never applies to a standalone word.
+ */
+ boolean compoundOnly(Affix affix) {
+ return onlyInCompound != 0 && affix.allowsContinuation(onlyInCompound);
+ }
+
+ /**
+ * Checks whether an affix is one half of a circumfix: it carries the
+ * {@code CIRCUMFIX} flag among its continuation classes, so it is only valid on a
+ * word that also carries a circumfix-marked affix of the other kind, the German
+ * {@code ge...t} participle being the model.
+ *
+ * @param affix The affix rule to inspect.
+ * @return {@code true} if the affix never applies without its other half.
+ */
+ boolean circumfixOnly(Affix affix) {
+ return circumfix != 0 && affix.allowsContinuation(circumfix);
+ }
+
+ /**
+ * Finds the {@code SET} declaration by scanning the raw affix bytes as ASCII, which
+ * is safe because the declaration itself is ASCII in every supported encoding. Both
+ * files are then decoded with the declared charset.
+ *
+ * @param affixBytes The raw affix file content.
+ * @return The declared charset, or UTF-8 when no declaration is present.
+ * @throws IOException Thrown if the declared encoding name is not supported.
+ */
+ private static Charset declaredCharset(byte[] affixBytes) throws IOException {
+ final String ascii = new String(affixBytes, StandardCharsets.US_ASCII);
+ for (final String line : splitLines(ascii)) {
+ final String trimmed = trim(line);
+ if (trimmed.startsWith("SET ") || trimmed.startsWith("SET\t")) {
+ final String name = trim(trimmed.substring(4));
+ try {
+ return Charset.forName(name);
+ } catch (IllegalCharsetNameException | UnsupportedCharsetException e) {
+ throw new IOException("unsupported SET encoding: " + name, e);
+ }
+ }
+ }
+ return StandardCharsets.UTF_8;
+ }
+
+ /** The flag encodings a dictionary may declare with the {@code FLAG} directive. */
+ private enum FlagMode {
+ /**
+ * The default: each single character is one flag. Also what {@code FLAG UTF-8}
+ * declares, which asks for single-character flags in a file the {@code SET}
+ * declaration already had decoded.
+ */
+ CHAR,
+ /** Declared as {@code FLAG long}: each pair of characters is one flag. */
+ LONG,
+ /** Declared as {@code FLAG num}: comma-separated decimal numbers are flags. */
+ NUM
+ }
+
+ /** The parsed affix file content. */
+ private static final class AffixFile {
+ private final List<Affix> prefixes = new ArrayList<>();
+ private final List<Affix> suffixes = new ArrayList<>();
+ private final List<int[]> flagAliases = new ArrayList<>();
+ private boolean aliasHeaderSeen;
+ private FlagMode flagMode = FlagMode.CHAR;
+ private int compoundFlag;
+ private int compoundBegin;
+ private int compoundEnd;
+ private int compoundMin = 3;
+ private int needAffix;
+ private int onlyInCompound;
+ private int forbiddenWord;
+ private int circumfix;
+ private int compoundMiddle;
+ private int compoundPermit;
+ private int compoundForbid;
+ private int compoundWordMax;
+ private boolean checkCompoundDup;
+ private boolean checkCompoundCase;
+ private boolean checkCompoundTriple;
+ private boolean fullStrip;
+ }
+
+ /**
+ * Parses the affix file: the {@code FLAG} declaration, the {@code AF} flag alias
+ * table, the compound and blocking flag declarations, and the {@code PFX} and
+ * {@code SFX} blocks. Result-altering unsupported directives fail loud;
+ * cosmetic ones are skipped.
+ *
+ * @param content The decoded affix file content.
+ * @return The parsed rules and flag mode. Never {@code null}.
+ * @throws IOException Thrown if a supported directive is malformed, or if
+ * {@code ICONV}, {@code OCONV}, {@code COMPLEXPREFIXES}, {@code COMPOUNDRULE},
+ * {@code IGNORE}, or {@code KEEPCASE} appears.
+ */
+ private static AffixFile parseAffix(String content) throws IOException {
+ final AffixFile result = new AffixFile();
+ final String[] lines = splitLines(content);
+ int i = 0;
+ while (i < lines.length) {
+ final String[] fields = split(lines[i]);
+ if (fields.length == 0 || fields[0].startsWith("#")) {
Review Comment:
Please declare `"#"` as a named constant for better re-use (in loops etc.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opennlp.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] OPENNLP-1893: Support Hunspell affix dictionaries for stemming (opennlp)
Posted by "krickert (via GitHub)" <gi...@apache.org>.
krickert commented on PR #1190:
URL: https://github.com/apache/opennlp/pull/1190#issuecomment-5206408356
Same download rework as #1191 in 0b27d5ec1: shared `DownloadUtil.download` plus `dictionary-catalog.properties`, shell helper removed.
`HunspellDictionaryDownload.downloadFromCatalog("en_US", dir)` fetches the `.aff`/`.dic` pair and `README_en_US.txt` (the dictionary's license file), pinned to LibreOffice dictionaries commit 208a9fd8 and verified by SHA-512. Requires `-Dopennlp.download.remote=true`; without the property the call fails with the property name in the message. Nothing is bundled and the tests touch no network.
The `DownloadUtil`/`DictionaryCatalog` files are identical to #1191's; whichever merges second gets rebased onto the other.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opennlp.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] OPENNLP-1893: Support Hunspell affix dictionaries for stemming (opennlp)
Posted by "krickert (via GitHub)" <gi...@apache.org>.
krickert commented on PR #1190:
URL: https://github.com/apache/opennlp/pull/1190#issuecomment-5204105265
1. `ICONV`, `OCONV`, and `COMPLEXPREFIXES` now fail at load. Cosmetic directives (`REP`, `MAP`, `KEY`, …) still skip. Class javadoc, README, and stemmer chapter updated. Tip: ba8ab36bb.
2. Download tooling will follow whatever we land for #1191 — one Java path with digest verification for both, no shell helper.
Also in that commit: defensive copy from `lookup`, `HunspellStemmer` is final, compound search budget called out on the class javadoc.
Stream size ceiling, char vs code-point consistency on affix material, and `@since` consistency still open.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opennlp.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] OPENNLP-1893: Support Hunspell affix dictionaries for stemming (opennlp)
Posted by "krickert (via GitHub)" <gi...@apache.org>.
krickert commented on PR #1190:
URL: https://github.com/apache/opennlp/pull/1190#issuecomment-5204655720
Follow-up in 8cede5e29.
1. Affix and dictionary streams are capped at `HunspellDictionary.MAX_STREAM_BYTES` (64 MiB); larger streams fail at `load`. Tests pin the public `load` path and the inclusive ceiling. Stemmer chapter updated.
2. Affix conditions and boundary bucketing use Unicode code points so supplementary characters agree with `FLAG UTF-8`. Test covers two-dot / one-dot / class conditions on emoji stems.
Download tooling still deferred with #1191 (one Java digest path, no shell helper).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opennlp.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org