You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@opennlp.apache.org by kr...@apache.org on 2026/09/04 00:40:50 UTC
(opennlp) 02/08: Merge refreshed PR #1190 into OPENNLP-1833 gRPC helper
This is an automated email from the ASF dual-hosted git repository.
krickert pushed a commit to branch OPENNLP-1833-grpc-helper
in repository https://gitbox.apache.org/repos/asf/opennlp.git
commit 2fb1ed6304f942474387c73aab88b2022ea37808
Merge: f2e804906 c833264c1
Author: Kristian Rickert <kr...@apache.org>
AuthorDate: Wed Sep 2 08:55:30 2026 -0400
Merge refreshed PR #1190 into OPENNLP-1833 gRPC helper
# Conflicts:
# dev/README-hunspell-dictionaries.md
# opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/hunspell/HunspellDictionary.java
# opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/hunspell/HunspellDictionaryDownload.java
# opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/DictionaryCatalog.java
# opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/stemmer/hunspell/HunspellDictionaryDownloadTest.java
# opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/stemmer/hunspell/HunspellRealDictionaryTest.java
# opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/stemmer/hunspell/HunspellStemmerFactoryTest.java
# opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/util/DictionaryCatalogTest.java
# opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/util/DigestTestUtil.java
dev/README-hunspell-dictionaries.md | 24 +++-
dev/README-mecab-dictionaries.md | 8 +-
.../tools/stemmer/hunspell/HunspellDictionary.java | 28 ++++-
.../hunspell/HunspellDictionaryDownload.java | 24 ++--
.../tokenize/lattice/MecabDictionaryInstaller.java | 16 ++-
.../java/opennlp/tools/util/DictionaryCatalog.java | 41 +++----
.../java/opennlp/tools/util/ResourceInstaller.java | 33 +++++-
.../hunspell/HunspellDictionaryDownloadTest.java | 117 ++++++++++++++++----
.../hunspell/HunspellRealDictionaryTest.java | 4 +-
.../hunspell/HunspellStemmerFactoryTest.java | 8 +-
.../tokenize/lattice/MecabCatalogEndToEndTest.java | 122 ---------------------
.../lattice/MecabDictionaryInstallerTest.java | 21 +++-
.../opennlp/tools/util/DictionaryCatalogTest.java | 39 +++++--
.../tools/util/dictionary-catalog.properties | 4 +-
opennlp-docs/src/docbkx/stemmer.xml | 56 +++-------
opennlp-docs/src/docbkx/tokenizer.xml | 92 +---------------
16 files changed, 286 insertions(+), 351 deletions(-)
diff --cc dev/README-mecab-dictionaries.md
index 5d31094d7,000000000..c98467cf3
mode 100644,000000..100644
--- a/dev/README-mecab-dictionaries.md
+++ b/dev/README-mecab-dictionaries.md
@@@ -1,126 -1,0 +1,128 @@@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+# CJK dictionaries for the lattice tokenizer
+
+The lattice tokenizer (`opennlp.tools.tokenize.lattice`) segments Japanese and Korean over a MeCab-format dictionary, and the unigram segmenter handles Chinese over a plain word-frequency lexicon. Apache OpenNLP bundles no dictionary data: you download a dictionary from the project of your choice, and each dictionary carries its own license. Read the license file inside the archive before use.
+
+## Known MeCab-format dictionary projects
+
+| Catalog id | Dictionary | Language | Encoding |
+|---|---|---|---|
+| `mecab.ipadic` | IPADIC 2.7.0 | Japanese | EUC-JP |
+| `mecab.ko-dic` | mecab-ko-dic 2.1.1 | Korean | UTF-8 |
+
+Pinned download URLs and SHA-512 digests for those ids live in
+`opennlp/tools/util/dictionary-catalog.properties`. Both archives are
+gzip-compressed tars; `MecabDictionaryInstaller` reads the ustar, pax, and GNU
+formats through `ResourceInstaller`.
+
+The installer extracts only the dictionary payload: the `*.csv` and `*.def` files a
+`MecabDictionary` reads, plus the `dicrc` configuration file the distributions ship
+alongside them. It flattens the entries into the target directory, and by the same
+flattening makes it impossible for an archive path to escape that directory. The
+returned count is the number of dictionary files extracted.
+
+## Option A: opt-in catalog install
+
+Catalog URLs stay inactive until you set `-Dopennlp.download.remote=true` (or the
- equivalent system property in code). That flag is the explicit user action that
- enables the built-in URLs; OpenNLP never fetches them by default.
++equivalent system property in code). Applications supply the catalog, and OpenNLP
++never fetches its entries by default.
+
+```java
+import java.nio.file.Path;
+import opennlp.tools.tokenize.lattice.MecabDictionaryInstaller;
++import opennlp.tools.util.DictionaryCatalog;
+
+// JVM flag: -Dopennlp.download.remote=true
++DictionaryCatalog catalog = DictionaryCatalog.load(catalogProperties);
+int files = MecabDictionaryInstaller.installFromCatalog(
- "mecab.ipadic", Path.of("ipadic"));
++ catalog, "mecab.ipadic", Path.of("ipadic"));
+```
+
+## Option B: your own URL and digest
+
+```java
+import java.net.URI;
+import java.nio.file.Path;
+import opennlp.tools.tokenize.lattice.MecabDictionaryInstaller;
+
+String expectedSha512 = "..."; // the 128-hex SHA-512 of the archive
+int files = MecabDictionaryInstaller.install(
+ URI.create("https://example.example/dict.tar.gz"),
+ Path.of("dict"),
+ expectedSha512);
+```
+
+A local `file:` URI may omit the digest:
+`MecabDictionaryInstaller.install(localArchive.toUri(), targetDirectory)`.
+Any other URI scheme requires the digest.
+
+## Size budgets for larger dictionaries
+
+Fetching and unpacking go through `ResourceInstaller` and are bounded so a crafted
+archive cannot fill the disk: by default one download is capped at 1 GiB, the
+unpacked payload at 4 GiB, and the archive at 100000 entries. IPADIC and
+mecab-ko-dic fit comfortably. For larger dictionaries, such as UniDic, raise the
+ceilings at JVM startup:
+
+```bash
+-Dopennlp.download.max.bytes=4294967296 \
+-Dopennlp.install.max.total.bytes=8589934592 \
+-Dopennlp.install.max.entries=200000
+```
+
+Values must be positive counts; anything absent or invalid falls back to the
+default.
+
+## Load and tokenize
+
+`MecabDictionary.load(Path)` assumes UTF-8. IPADIC needs the two-argument overload:
+
+```java
+import java.nio.charset.Charset;
+import java.nio.file.Path;
+import opennlp.tools.tokenize.lattice.LatticeTokenizer;
+import opennlp.tools.tokenize.lattice.MecabDictionary;
+
+MecabDictionary dictionary =
+ MecabDictionary.load(Path.of("ipadic"), Charset.forName("EUC-JP"));
+LatticeTokenizer tokenizer = new LatticeTokenizer(dictionary);
+// "Tokyo-to ni iku" (go to the Tokyo metropolis), escaped to keep this file ASCII
+String[] tokens = tokenizer.tokenize("\u6771\u4EAC\u90FD\u306B\u884C\u304F");
+```
+
+For a UTF-8 dictionary such as mecab-ko-dic, `MecabDictionary.load(Path.of("ko-dic"))`
+is enough. Loaded dictionaries and tokenizers are immutable and safe to share between
+threads, so load once and reuse.
+
+## Chinese: the unigram segmenter needs only a frequency lexicon
+
+`opennlp.tools.tokenize.lattice.UnigramSegmenter` does not use MeCab dictionaries. It
+loads a plain text lexicon, one entry per line: the word, its count, and optionally a
+tag, separated by whitespace. Any word-frequency list you have the rights to use works:
+
+```java
+import java.nio.file.Path;
+import opennlp.tools.tokenize.lattice.UnigramSegmenter;
+
+UnigramSegmenter segmenter = UnigramSegmenter.load(Path.of("words.txt"));
+// "wo laidao Beijing Tian'anmen" (I arrive at Beijing Tiananmen), escaped as above
+String[] tokens = segmenter.tokenize("\u6211\u6765\u5230\u5317\u4EAC\u5929\u5B89\u95E8");
+```
+
+As with the dictionaries, the lexicon carries its own license; nothing is bundled.
diff --cc opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/hunspell/HunspellDictionaryDownload.java
index a6cfefb8e,3775df4ac..1d5527e71
--- a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/hunspell/HunspellDictionaryDownload.java
+++ b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/hunspell/HunspellDictionaryDownload.java
@@@ -36,34 -37,57 +37,37 @@@ public final class HunspellDictionaryDo
/**
* Downloads the cataloged {@code .aff}, {@code .dic}, and readme files for
- * {@code dictionaryId} into {@code targetDirectory}.
+ * {@code dictionaryId} into {@code targetDirectory}, each stored under its source
+ * name, for example {@code en_US.aff}. A file that already exists in the target is
+ * refused, so refreshing a dictionary means removing its old files first.
*
+ * @param catalog The application-supplied catalog. Must not be {@code null}.
* @param dictionaryId The catalog dictionary name, for example {@code en_US}.
* Must not be {@code null}.
* @param targetDirectory The directory to write into; created when absent. Must not
* be {@code null}.
* @throws IOException Thrown if remote downloads are disabled, a catalog entry is
- * missing, or verification fails.
+ * missing, verification fails, or the target already contains one of the
+ * files.
* @throws IllegalArgumentException Thrown if a parameter is {@code null}.
*/
- public static void downloadFromCatalog(String dictionaryId, Path targetDirectory)
- throws IOException {
+ public static void downloadFromCatalog(DictionaryCatalog catalog, String dictionaryId,
+ Path targetDirectory) throws IOException {
+ if (catalog == null) {
+ throw new IllegalArgumentException("catalog must not be null");
+ }
if (dictionaryId == null) {
throw new IllegalArgumentException("dictionaryId must not be null");
}
if (targetDirectory == null) {
throw new IllegalArgumentException("targetDirectory must not be null");
}
- final DictionaryCatalog catalog = DictionaryCatalog.loadDefault();
- final String prefix = "hunspell." + dictionaryId + ".";
- catalog.install(prefix + "aff", targetDirectory);
- catalog.install(prefix + "dic", targetDirectory);
- final String readmeId = prefix + "readme";
+ final String prefix = "hunspell." + dictionaryId;
- download(catalog, prefix + HunspellDictionary.AFFIX_FILE_SUFFIX, targetDirectory);
- download(catalog, prefix + HunspellDictionary.DICTIONARY_FILE_SUFFIX, targetDirectory);
++ catalog.install(prefix + HunspellDictionary.AFFIX_FILE_SUFFIX, targetDirectory);
++ catalog.install(prefix + HunspellDictionary.DICTIONARY_FILE_SUFFIX, targetDirectory);
+ final String readmeId = prefix + ".readme";
if (catalog.ids().contains(readmeId)) {
- download(catalog, readmeId, targetDirectory);
+ catalog.install(readmeId, targetDirectory);
}
}
-
- /**
- * Downloads one catalog entry into {@code targetDirectory}, named by the entry's
- * preferred file name or, when absent, by the last segment of its URI path.
- *
- * @param catalog The catalog holding {@code id}.
- * @param id The catalog entry id.
- * @param targetDirectory The directory to write into.
- * @throws IOException Thrown if remote downloads are disabled, the entry is missing,
- * or the download fails verification.
- */
- private static void download(DictionaryCatalog catalog, String id, Path targetDirectory)
- throws IOException {
- final DictionaryCatalog.Entry entry = catalog.get(id);
- final String filename;
- if (entry.filename() != null) {
- filename = entry.filename();
- } else {
- final String path = entry.uri().getPath();
- filename = path.substring(path.lastIndexOf('/') + 1);
- }
- catalog.download(id, targetDirectory.resolve(filename));
- }
}
diff --cc opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/tokenize/lattice/MecabDictionaryInstaller.java
index 7110af4e9,000000000..7df9045f4
mode 100644,000000..100644
--- a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/tokenize/lattice/MecabDictionaryInstaller.java
+++ b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/tokenize/lattice/MecabDictionaryInstaller.java
@@@ -1,238 -1,0 +1,244 @@@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package opennlp.tools.tokenize.lattice;
+
+import java.io.IOException;
+import java.net.URI;
+import java.nio.file.FileVisitResult;
+import java.nio.file.Files;
+import java.nio.file.LinkOption;
+import java.nio.file.Path;
+import java.nio.file.SimpleFileVisitor;
+import java.nio.file.attribute.BasicFileAttributes;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.stream.Stream;
+
+import opennlp.tools.util.DictionaryCatalog;
+import opennlp.tools.util.ResourceInstaller;
+
+/**
+ * Fetches and unpacks a MeCab-format dictionary archive into a local directory, so the
+ * dictionary is acquired by the user at install time and never ships with this library.
+ * No dictionary data is bundled. Fetching, verification, and unpacking are done by
+ * {@link ResourceInstaller} under {@link ResourceInstaller.Limits#DEFAULT} and its
+ * startup property overrides: an {@code http} or {@code https} archive requires an
+ * expected checksum, and the ustar, pax, and GNU tar formats are all read. Built-in
- * catalog URLs are opt-in via {@link #installFromCatalog(String, Path)}.
++ * catalog URLs are opt-in via
++ * {@link #installFromCatalog(DictionaryCatalog, String, Path)}.
+ *
+ * <p>Only the dictionary payload is installed: the {@code *.csv} lexicon files and
+ * {@code *.def} definition files that a {@link MecabDictionary} reads, plus the
+ * {@code dicrc} configuration file distributions ship alongside them, taken from the
+ * archive root only (at most one leading directory deep). Deeper entries are skipped:
+ * mecab-ko-dic, for example, nests {@code user-dic} templates whose numeric fields are
+ * empty because they are input for {@code mecab-dict-index}, not loadable lexicon
+ * data. Installed files are flattened to their base names. A file whose base name
+ * already exists in the target is refused, so refreshing a dictionary means removing
+ * its old files first.</p>
+ *
+ * @since 3.0.0
+ */
+public final class MecabDictionaryInstaller {
+
+ /** The deepest entry path, relative to the archive root, that holds payload. */
+ private static final int MAX_PAYLOAD_DEPTH = 2;
+
+ private MecabDictionaryInstaller() {
+ // This class exposes only static methods and is never instantiated.
+ }
+
+ /**
+ * Unpacks a local {@code file:} archive URI. Any other scheme requires
+ * {@link #install(URI, Path, String)} with an expected checksum.
+ *
+ * @param archive The archive location, a gzip-compressed tar. Must not be
+ * {@code null}.
+ * @param targetDirectory The directory to unpack into; created when absent. Must not
+ * be {@code null}.
+ * @return The number of dictionary files installed.
+ * @throws IOException Thrown if reading or writing fails, the archive contains no
+ * dictionary file, an installation limit is exceeded, or the target already
+ * contains one of the files.
+ * @throws IllegalArgumentException Thrown if a parameter is {@code null} or
+ * {@code archive} is not a readable {@code file:} URI.
+ */
+ public static int install(URI archive, Path targetDirectory) throws IOException {
+ return install(archive, targetDirectory, null);
+ }
+
+ /**
+ * Downloads a dictionary archive when needed, verifies its checksum, and unpacks it
+ * through {@link ResourceInstaller#install(URI, Path, String)}. A {@code file:} URI
+ * may omit the checksum and is then opened as trusted caller input.
+ *
+ * @param archive The archive location, a gzip-compressed tar. Must not be
+ * {@code null}.
+ * @param targetDirectory The directory to unpack into; created when absent. Must not
+ * be {@code null}.
+ * @param expectedChecksum The expected digest of the archive bytes as a hex string,
+ * 64 characters for SHA-256 or 128 for SHA-512. Required for
+ * an http or https source; pass {@code null} to skip
+ * verification for a file source.
+ * @return The number of dictionary files installed.
+ * @throws IOException Thrown if fetching, verification, reading, or writing fails,
+ * the archive contains no dictionary file, an installation limit is
+ * exceeded, or the target already contains one of the files.
+ * @throws IllegalArgumentException Thrown if a parameter is {@code null}, the URI is
+ * not supported by {@link ResourceInstaller}, or an http or https source
+ * carries no checksum.
+ */
+ public static int install(URI archive, Path targetDirectory, String expectedChecksum)
+ throws IOException {
+ if (archive == null) {
+ throw new IllegalArgumentException("archive must not be null");
+ }
+ if (targetDirectory == null) {
+ throw new IllegalArgumentException("targetDirectory must not be null");
+ }
+ final Path unpacked = Files.createTempDirectory("mecab-dict-");
+ try {
+ ResourceInstaller.install(archive, unpacked, expectedChecksum);
+ return promoteDictionaryFiles(unpacked, targetDirectory);
+ } finally {
+ deleteRecursively(unpacked);
+ }
+ }
+
+ /**
- * Downloads a dictionary named in {@link DictionaryCatalog} and unpacks it. Requires
++ * Downloads a dictionary named in an application-supplied
++ * {@link DictionaryCatalog} and unpacks it. Requires
+ * {@code -Dopennlp.download.remote=true}.
+ *
++ * @param catalog The application-supplied catalog. Must not be {@code null}.
+ * @param dictionaryId The catalog id, for example {@code mecab.ipadic} or
+ * {@code mecab.ko-dic}. Must not be {@code null}.
+ * @param targetDirectory The directory to unpack into; created when absent. Must not
+ * be {@code null}.
+ * @return The number of dictionary files installed.
+ * @throws IOException Thrown if the catalog entry is missing, remote downloads are
+ * disabled, or install fails.
+ * @throws IllegalArgumentException Thrown if a parameter is {@code null}.
+ */
- public static int installFromCatalog(String dictionaryId, Path targetDirectory)
- throws IOException {
++ public static int installFromCatalog(DictionaryCatalog catalog, String dictionaryId,
++ Path targetDirectory) throws IOException {
++ if (catalog == null) {
++ throw new IllegalArgumentException("catalog must not be null");
++ }
+ if (dictionaryId == null) {
+ throw new IllegalArgumentException("dictionaryId must not be null");
+ }
+ if (targetDirectory == null) {
+ throw new IllegalArgumentException("targetDirectory must not be null");
+ }
+ final Path unpacked = Files.createTempDirectory("mecab-dict-");
+ try {
- DictionaryCatalog.loadDefault().install(dictionaryId, unpacked);
++ catalog.install(dictionaryId, unpacked);
+ return promoteDictionaryFiles(unpacked, targetDirectory);
+ } finally {
+ deleteRecursively(unpacked);
+ }
+ }
+
+ /**
+ * Moves the dictionary payload files from an unpacked archive tree into the target
+ * directory, flattened to their base names. Every destination is checked before the
+ * first move, so a collision leaves the target unchanged.
+ *
+ * @param unpacked The directory the archive was unpacked into.
+ * @param targetDirectory The directory to install into; created when absent.
+ * @return The number of dictionary files installed.
+ * @throws IOException Thrown if the tree holds no dictionary file, two entries
+ * flatten to the same base name, a target file already exists, or moving
+ * fails.
+ */
+ private static int promoteDictionaryFiles(Path unpacked, Path targetDirectory)
+ throws IOException {
+ Files.createDirectories(targetDirectory);
+ final List<Path> candidates;
+ try (Stream<Path> files = Files.walk(unpacked, MAX_PAYLOAD_DEPTH)) {
+ candidates = files.filter(Files::isRegularFile).toList();
+ }
+ final List<Path> payload = new ArrayList<>();
+ final Set<String> baseNames = new HashSet<>();
+ for (final Path file : candidates) {
+ final String baseName = file.getFileName().toString();
+ if (!isDictionaryFile(baseName)) {
+ continue;
+ }
+ if (!baseNames.add(baseName)) {
+ throw new IOException(
+ "the archive flattens two entries to the same name: " + baseName);
+ }
+ payload.add(file);
+ }
+ if (payload.isEmpty()) {
+ throw new IOException("the archive contains no dictionary file");
+ }
+ for (final Path file : payload) {
+ final Path destination = targetDirectory.resolve(file.getFileName().toString());
+ if (Files.exists(destination, LinkOption.NOFOLLOW_LINKS)) {
+ throw new IOException("target already contains: " + destination);
+ }
+ }
+ for (final Path file : payload) {
+ Files.move(file, targetDirectory.resolve(file.getFileName().toString()));
+ }
+ return payload.size();
+ }
+
+ /**
+ * Recognizes the file names a {@link MecabDictionary} loads.
+ *
+ * @param baseName The file name without any directory prefix.
+ * @return {@code true} when the name is dictionary payload.
+ */
+ private static boolean isDictionaryFile(String baseName) {
+ return baseName.endsWith(".csv") || baseName.endsWith(".def")
+ || "dicrc".equals(baseName);
+ }
+
+ /**
+ * Deletes a directory tree, deepest entries first.
+ *
+ * @param root The directory to remove.
+ * @throws IOException Thrown if a deletion fails.
+ */
+ private static void deleteRecursively(Path root) throws IOException {
+ Files.walkFileTree(root, new SimpleFileVisitor<>() {
+ @Override
+ public FileVisitResult visitFile(Path file, BasicFileAttributes attributes)
+ throws IOException {
+ Files.delete(file);
+ return FileVisitResult.CONTINUE;
+ }
+
+ @Override
+ public FileVisitResult postVisitDirectory(Path directory, IOException error)
+ throws IOException {
+ if (error != null) {
+ throw error;
+ }
+ Files.delete(directory);
+ return FileVisitResult.CONTINUE;
+ }
+ });
+ }
+}
diff --cc opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/DictionaryCatalog.java
index 041c027e6,d0704e8c6..1a752aa02
--- a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/DictionaryCatalog.java
+++ b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/DictionaryCatalog.java
@@@ -28,25 -28,14 +28,20 @@@ import java.util.Properties
import java.util.Set;
/**
- * Opt-in catalog of remote dictionary archives and companion files. The catalog
- * ships URLs and SHA-512 digests only; it never bundles the data itself. Fetching
- * an entry requires {@link #REMOTE_DOWNLOAD_PROPERTY} to be {@code true}, so
- * enabling a built-in URL is an explicit user action.
+ * Opt-in catalog of remote dictionary archives and companion files, loaded from
+ * application-supplied properties containing URLs and SHA-512 digests. Fetching an
- * entry requires {@link DownloadUtil#REMOTE_DOWNLOAD_PROPERTY} to be {@code true}.
++ * entry requires {@link #REMOTE_DOWNLOAD_PROPERTY} to be {@code true}.
*
* @since 3.0.0
*/
public final class DictionaryCatalog {
+ /**
+ * System property that must be {@code true} before a catalog entry may be
+ * fetched. Direct {@link ResourceInstaller} calls do not require it: there the
+ * caller already supplied the URI and digest.
+ */
+ public static final String REMOTE_DOWNLOAD_PROPERTY = "opennlp.download.remote";
-
- private static final String DEFAULT_RESOURCE =
- "opennlp/tools/util/dictionary-catalog.properties";
-
private final Properties properties;
private DictionaryCatalog(Properties properties) {
@@@ -124,29 -98,25 +104,34 @@@
}
/**
- * Downloads a catalog entry into {@code target} after checking that remote catalog
- * downloads are enabled.
+ * Installs a catalog entry into {@code targetDirectory} after checking that remote
+ * catalog downloads are enabled. The entry is fetched, digest-verified, and unpacked
+ * by {@link ResourceInstaller#install(URI, Path, String)}: an archive expands into
+ * the directory, and a plain file is stored under its source name.
*
* @param id The entry id. Must not be {@code null}.
- * @param target The local file to create. Must not be {@code null}.
+ * @param targetDirectory The directory to install into; created when absent. Must
+ * not be {@code null}.
* @throws IOException Thrown if the property is not enabled, the entry is missing,
- * or the download fails verification.
+ * the download fails verification, or the target already contains an
+ * installed file.
* @throws IllegalArgumentException Thrown if a parameter is {@code null}.
*/
- public void download(String id, Path target) throws IOException {
- if (target == null) {
- throw new IllegalArgumentException("target must not be null");
+ public void install(String id, Path targetDirectory) throws IOException {
+ if (targetDirectory == null) {
+ throw new IllegalArgumentException("targetDirectory must not be null");
}
- if (!DownloadUtil.isRemoteDownloadEnabled()) {
+ if (!Boolean.getBoolean(REMOTE_DOWNLOAD_PROPERTY)) {
throw new IOException("remote dictionary catalog downloads are disabled; set -D"
- + DownloadUtil.REMOTE_DOWNLOAD_PROPERTY + "=true to enable");
+ + REMOTE_DOWNLOAD_PROPERTY + "=true to enable");
}
final Entry entry = get(id);
- ResourceInstaller.install(entry.uri(), targetDirectory, entry.sha512());
- DownloadUtil.download(entry.uri(), target, entry.sha512());
++ if (entry.filename() == null) {
++ ResourceInstaller.install(entry.uri(), targetDirectory, entry.sha512());
++ } else {
++ ResourceInstaller.installNamed(
++ entry.uri(), targetDirectory, entry.sha512(), entry.filename());
++ }
}
/**
diff --cc opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/ResourceInstaller.java
index f09f0c703,000000000..9df2ef8e1
mode 100644,000000..100644
--- a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/ResourceInstaller.java
+++ b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/ResourceInstaller.java
@@@ -1,1123 -1,0 +1,1154 @@@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package opennlp.tools.util;
+
+import java.io.BufferedInputStream;
+import java.io.FilterInputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.net.HttpURLConnection;
+import java.net.URI;
+import java.nio.file.AtomicMoveNotSupportedException;
+import java.nio.file.FileSystemNotFoundException;
+import java.nio.file.Files;
+import java.nio.file.LinkOption;
+import java.nio.file.Path;
+import java.nio.file.StandardCopyOption;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.time.Duration;
+import java.util.Comparator;
+import java.util.HexFormat;
+import java.util.List;
+import java.util.stream.Stream;
+import java.util.zip.GZIPInputStream;
+import java.util.zip.ZipEntry;
+import java.util.zip.ZipInputStream;
+
+import opennlp.tools.util.archive.TarStream;
+
+/**
+ * Fetches a third-party resource, such as a training corpus, a dictionary archive, or a
+ * lexicon, into a local directory. The caller supplies the location and thereby accepts
+ * that resource's license; no locations are built in and no data is bundled. Only
+ * {@code http}, {@code https}, and {@code file} locations are accepted.
+ *
+ * <p>A checksum is required for http and https sources and optional for file sources.
+ * It is verified against the downloaded bytes before anything is unpacked: a
+ * 64-character hex digest selects SHA-256, a 128-character one SHA-512.
+ * The content format is detected from the bytes, not from the name: gzip-compressed
+ * tar archives and zip archives are unpacked with their relative structure, entries
+ * that would escape the target directory are rejected, plain gzip files are
+ * decompressed, and anything else is stored as a file under its source name. One name
+ * rule overrides byte detection: a {@code *.bin} source is always stored packed,
+ * because an OpenNLP model file is itself a zip archive that its consumers load
+ * packed.</p>
+ *
+ * <p>Every installation is bounded by {@link Limits}: http and https fetches carry
+ * connection and read timeouts, follow at most a fixed number of redirects, refuse
+ * redirects that leave the http and https schemes or downgrade https to http, and
+ * abort once the download or the expanded content crosses its size ceiling or the
+ * archive crosses its entry ceiling. The defaults in {@link Limits#DEFAULT} apply when
+ * no limits are given, and {@link Limits#builder()} starts from them.</p>
+ *
+ * <p>Installation is staged: content is unpacked into a hidden staging directory on
+ * the same filesystem and moved into the target only after the download was verified
+ * and every entry unpacked cleanly. A fetch, verification, or unpacking failure
+ * therefore leaves the target directory as it was, without partially written files.
+ * Promotion refuses to replace a file that already exists in the target and detects
+ * the collision before moving anything, so refreshing a resource means removing its
+ * old files first.</p>
+ *
+ * @see DownloadUtil
+ * @since 3.0.0
+ */
+public final class ResourceInstaller {
+
+ private static final String SHA_256 = "SHA-256";
+ private static final String SHA_512 = "SHA-512";
+ private static final int SHA_256_HEX_LENGTH = 64;
+ private static final int SHA_512_HEX_LENGTH = 128;
+ private static final String GZIP_SUFFIX = ".gz";
+
+ /** OpenNLP model files are packed zip archives; install them packed. */
+ private static final String MODEL_SUFFIX = ".bin";
+ private static final String DEFAULT_RESOURCE_NAME = "resource";
+ private static final String STAGING_PREFIX = ".opennlp-staging";
+ private static final String DOWNLOAD_PREFIX = ".opennlp-download";
+ private static final String DOWNLOAD_SUFFIX = ".part";
+ private static final int BUFFER_SIZE = 8192;
+ private static final int MAGIC_LENGTH = 2;
+ private static final int GZIP_MAGIC_FIRST = 0x1F;
+ private static final int GZIP_MAGIC_SECOND = 0x8B;
+ private static final int ZIP_MAGIC_FIRST = 'P';
+ private static final int ZIP_MAGIC_SECOND = 'K';
+ private static final int HTTP_TEMPORARY_REDIRECT = 307;
+ private static final int HTTP_PERMANENT_REDIRECT = 308;
+ private static final String SCHEME_HTTP = "http";
+ private static final String SCHEME_HTTPS = "https";
+ private static final String SCHEME_FILE = "file";
+
+ /**
+ * Safety ceilings and network behavior for one installation.
+ *
+ * @param connectTimeout How long to wait for a connection to be established. Must
+ * be positive.
+ * @param readTimeout How long to wait for data on an established connection. Must
+ * be positive.
+ * @param maxRedirects How many http redirects to follow before failing. Must not be
+ * negative; zero refuses all redirects.
+ * @param maxDownloadBytes The largest download accepted, in bytes. Must be positive.
+ * @param maxExpandedBytes The largest expanded byte count accepted. For gzip content,
+ * this counts the entire decompressed stream; otherwise, it
+ * counts installed file content. Must be positive.
+ * @param maxEntries The largest number of archive entries accepted, counting every
+ * entry including directories, so an archive of many tiny files
+ * cannot exhaust directory entries while staying under the byte
+ * ceilings. Must be positive.
+ */
+ public record Limits(Duration connectTimeout, Duration readTimeout, int maxRedirects,
+ long maxDownloadBytes, long maxExpandedBytes, long maxEntries) {
+
+ /** The system property overriding the default download ceiling in bytes. */
+ public static final String MAX_DOWNLOAD_BYTES_PROPERTY = "opennlp.download.max.bytes";
+
+ /** The system property overriding the default expansion ceiling in bytes. */
+ public static final String MAX_EXPANDED_BYTES_PROPERTY =
+ "opennlp.install.max.total.bytes";
+
+ /** The system property overriding the default archive entry ceiling. */
+ public static final String MAX_ENTRIES_PROPERTY = "opennlp.install.max.entries";
+
+ /**
+ * The limits applied when none are given: 20 second connect timeout, 60 second
+ * read timeout, at most 5 redirects, a 1 GiB download ceiling, a 4 GiB expansion
+ * ceiling, and 100000 archive entries. Each ceiling can be raised or lowered at
+ * startup through its system property ({@link #MAX_DOWNLOAD_BYTES_PROPERTY},
+ * {@link #MAX_EXPANDED_BYTES_PROPERTY}, {@link #MAX_ENTRIES_PROPERTY}), read once
+ * at class load; a value that is absent, not a number, or not positive falls back
+ * to the built-in default.
+ */
+ public static final Limits DEFAULT = new Limits(Duration.ofSeconds(20),
+ Duration.ofSeconds(60), 5,
+ longProperty(MAX_DOWNLOAD_BYTES_PROPERTY, 1L << 30),
+ longProperty(MAX_EXPANDED_BYTES_PROPERTY, 4L << 30),
+ longProperty(MAX_ENTRIES_PROPERTY, 100_000L));
+
+ /**
+ * Reads a ceiling override from a system property, trimmed before parsing.
+ *
+ * @param name The property name.
+ * @param fallback The built-in default.
+ * @return The property's value, or {@code fallback} when the property is absent,
+ * not a number, or not positive.
+ */
+ static long longProperty(String name, long fallback) {
+ final String value = System.getProperty(name);
+ if (value == null) {
+ return fallback;
+ }
+ final long parsed;
+ try {
+ parsed = Long.parseLong(value.trim());
+ } catch (NumberFormatException e) {
+ return fallback;
+ }
+ return parsed > 0 ? parsed : fallback;
+ }
+
+ /**
+ * Validates the limit values before constructing an instance.
+ *
+ * @param connectTimeout How long to wait for a connection to be established.
+ * @param readTimeout How long to wait for data on an established connection.
+ * @param maxRedirects How many http redirects to follow before failing.
+ * @param maxDownloadBytes The largest download accepted, in bytes.
+ * @param maxExpandedBytes The largest expanded byte count accepted.
+ * @param maxEntries The largest number of archive entries accepted.
+ * @throws IllegalArgumentException Thrown if either timeout is {@code null}, zero,
+ * or negative, a ceiling is not positive, or the redirect limit is
+ * negative.
+ */
+ public Limits(Duration connectTimeout, Duration readTimeout, int maxRedirects,
+ long maxDownloadBytes, long maxExpandedBytes, long maxEntries) {
+ if (connectTimeout == null || connectTimeout.isZero() || connectTimeout.isNegative()) {
+ throw new IllegalArgumentException("connectTimeout must be positive");
+ }
+ if (readTimeout == null || readTimeout.isZero() || readTimeout.isNegative()) {
+ throw new IllegalArgumentException("readTimeout must be positive");
+ }
+ if (maxRedirects < 0) {
+ throw new IllegalArgumentException("maxRedirects must not be negative");
+ }
+ if (maxDownloadBytes <= 0) {
+ throw new IllegalArgumentException("maxDownloadBytes must be positive");
+ }
+ if (maxExpandedBytes <= 0) {
+ throw new IllegalArgumentException("maxExpandedBytes must be positive");
+ }
+ if (maxEntries <= 0) {
+ throw new IllegalArgumentException("maxEntries must be positive");
+ }
+ this.connectTimeout = connectTimeout;
+ this.readTimeout = readTimeout;
+ this.maxRedirects = maxRedirects;
+ this.maxDownloadBytes = maxDownloadBytes;
+ this.maxExpandedBytes = maxExpandedBytes;
+ this.maxEntries = maxEntries;
+ }
+
+ /**
+ * Starts from {@link #DEFAULT} so a caller can state only the limits that differ
+ * from it, instead of repeating all six in the canonical constructor.
+ *
+ * @return A builder holding the default limits. Never {@code null}.
+ */
+ public static Builder builder() {
+ return new Builder();
+ }
+
+ /**
+ * Collects limit values and validates them on {@link #build()}. Each setter returns
+ * this builder. Not thread safe; the {@link Limits} it builds is immutable.
+ */
+ public static final class Builder {
+
+ private Duration connectTimeout = DEFAULT.connectTimeout();
+ private Duration readTimeout = DEFAULT.readTimeout();
+ private int maxRedirects = DEFAULT.maxRedirects();
+ private long maxDownloadBytes = DEFAULT.maxDownloadBytes();
+ private long maxExpandedBytes = DEFAULT.maxExpandedBytes();
+ private long maxEntries = DEFAULT.maxEntries();
+
+ private Builder() {
+ }
+
+ /**
+ * Sets how long to wait for a connection to be established.
+ *
+ * @param connectTimeout How long to wait for a connection to be established.
+ * Must be positive.
+ * @return This builder. Never {@code null}.
+ */
+ public Builder connectTimeout(Duration connectTimeout) {
+ this.connectTimeout = connectTimeout;
+ return this;
+ }
+
+ /**
+ * Sets how long to wait for data on an established connection.
+ *
+ * @param readTimeout How long to wait for data on an established connection.
+ * Must be positive.
+ * @return This builder. Never {@code null}.
+ */
+ public Builder readTimeout(Duration readTimeout) {
+ this.readTimeout = readTimeout;
+ return this;
+ }
+
+ /**
+ * Sets how many http redirects to follow before failing.
+ *
+ * @param maxRedirects How many http redirects to follow before failing. Must not
+ * be negative; zero refuses all redirects.
+ * @return This builder. Never {@code null}.
+ */
+ public Builder maxRedirects(int maxRedirects) {
+ this.maxRedirects = maxRedirects;
+ return this;
+ }
+
+ /**
+ * Sets the largest download accepted.
+ *
+ * @param maxDownloadBytes The largest download accepted, in bytes. Must be
+ * positive.
+ * @return This builder. Never {@code null}.
+ */
+ public Builder maxDownloadBytes(long maxDownloadBytes) {
+ this.maxDownloadBytes = maxDownloadBytes;
+ return this;
+ }
+
+ /**
+ * Sets the largest expanded byte count accepted.
+ *
+ * @param maxExpandedBytes The largest expanded byte count accepted. For gzip
+ * content, this counts the entire decompressed stream;
+ * otherwise, it counts installed file content. Must be
+ * positive.
+ * @return This builder. Never {@code null}.
+ */
+ public Builder maxExpandedBytes(long maxExpandedBytes) {
+ this.maxExpandedBytes = maxExpandedBytes;
+ return this;
+ }
+
+ /**
+ * Sets the largest number of archive entries accepted.
+ *
+ * @param maxEntries The largest number of archive entries accepted, counting
+ * every entry including directories. Must be positive.
+ * @return This builder. Never {@code null}.
+ */
+ public Builder maxEntries(long maxEntries) {
+ this.maxEntries = maxEntries;
+ return this;
+ }
+
+ /**
+ * Builds the limits.
+ *
+ * @return The limits collected so far. Never {@code null}.
+ * @throws IllegalArgumentException Thrown if any value is outside its documented
+ * range, exactly as the canonical constructor rejects it.
+ */
+ public Limits build() {
+ return new Limits(connectTimeout, readTimeout, maxRedirects, maxDownloadBytes,
+ maxExpandedBytes, maxEntries);
+ }
+ }
+ }
+
+ private ResourceInstaller() {
+ }
+
+ /**
+ * Unpacks a resource without checksum verification, under {@link Limits#DEFAULT}.
+ * This overload treats the source as trusted caller input and performs no
+ * cryptographic integrity verification, so it accepts only {@code file} sources; an
+ * http or https source must go through an overload that takes its checksum.
+ *
+ * @param source The resource location, a {@code file} URI. Must not be {@code null}.
+ * @param targetDirectory The directory to install into; created when absent. Must
+ * not be {@code null}.
+ * @return The target directory. Never {@code null}.
+ * @throws IOException Thrown if fetching or unpacking fails.
+ * @throws IllegalArgumentException Thrown if {@code source} or
+ * {@code targetDirectory} is {@code null}, or {@code source} carries a scheme
+ * other than {@code file}.
+ */
+ public static Path install(URI source, Path targetDirectory) throws IOException {
+ return install(source, targetDirectory, null);
+ }
+
+ /**
+ * Fetches, verifies, and unpacks a resource under {@link Limits#DEFAULT}.
+ *
+ * @param source The resource location, an {@code http}, {@code https}, or
+ * {@code file} URI. Must not be {@code null}.
+ * @param targetDirectory The directory to install into; created when absent. Must
+ * not be {@code null}.
+ * @param checksum The expected digest of the downloaded bytes as a hex string,
+ * compared case-insensitively and ignoring leading and trailing
+ * whitespace: 64 characters select SHA-256, 128 characters SHA-512.
+ * Required for an http or https source; pass {@code null} to skip
+ * verification for a {@code file} source.
+ * @return The target directory. Never {@code null}.
+ * @throws IOException Thrown if fetching fails, the checksum does not match, or
+ * unpacking fails.
+ * @throws IllegalArgumentException Thrown if {@code source} or
+ * {@code targetDirectory} is {@code null}, {@code source} carries a scheme
+ * other than {@code http}, {@code https}, or {@code file}, {@code checksum}
+ * is neither a 64-character nor a 128-character hex string, or an http or
+ * https source carries no checksum.
+ */
+ public static Path install(URI source, Path targetDirectory, String checksum)
+ throws IOException {
+ return install(source, targetDirectory, checksum, Limits.DEFAULT);
+ }
+
+ /**
+ * Fetches, verifies, and unpacks a resource under the given {@link Limits}.
+ *
+ * @param source The resource location, an {@code http}, {@code https}, or
+ * {@code file} URI. Must not be {@code null}.
+ * @param targetDirectory The directory to install into; created when absent. Must
+ * not be {@code null}.
+ * @param checksum The expected digest of the downloaded bytes as a hex string,
+ * compared case-insensitively and ignoring leading and trailing
+ * whitespace: 64 characters select SHA-256, 128 characters SHA-512.
+ * Required for an http or https source; pass {@code null} to skip
+ * verification for a {@code file} source.
+ * @param limits The timeouts, redirect allowance, and size and entry ceilings to
+ * enforce. Must not be {@code null}.
+ * @return The target directory. Never {@code null}.
+ * @throws IOException Thrown if fetching fails, a limit is exceeded, the checksum
+ * does not match, or unpacking fails.
+ * @throws IllegalArgumentException Thrown if {@code source}, {@code targetDirectory},
+ * or {@code limits} is {@code null}, {@code source} carries a scheme other
+ * than {@code http}, {@code https}, or {@code file}, {@code checksum} is
+ * neither a 64-character nor a 128-character hex string, or an http or https
+ * source carries no checksum.
+ */
+ public static Path install(URI source, Path targetDirectory, String checksum,
+ Limits limits) throws IOException {
++ return install(source, targetDirectory, checksum, null, limits);
++ }
++
++ /**
++ * Installs a catalog entry under its preferred file name when it is not an archive.
++ */
++ static Path installNamed(URI source, Path targetDirectory, String checksum,
++ String name) throws IOException {
++ if (name == null) {
++ throw new IllegalArgumentException("name must not be null");
++ }
++ return install(source, targetDirectory, checksum, name, Limits.DEFAULT);
++ }
++
++ private static Path install(URI source, Path targetDirectory, String checksum,
++ String name, Limits limits) throws IOException {
+ if (source == null) {
+ throw new IllegalArgumentException("source must not be null");
+ }
+ if (targetDirectory == null) {
+ throw new IllegalArgumentException("targetDirectory must not be null");
+ }
+ if (limits == null) {
+ throw new IllegalArgumentException("limits must not be null");
+ }
+ validateSource(source);
+ final String expected = validateChecksum(checksum);
+ if (expected == null && isHttp(source.getScheme())) {
+ throw new IllegalArgumentException(
+ "checksum must be given for an http or https source: " + source);
+ }
+ Files.createDirectories(targetDirectory);
+ final Path downloaded = createDownloadFile(targetDirectory);
+ try {
+ download(source, downloaded, limits);
+ if (expected != null) {
+ verify(downloaded, expected);
+ }
- installStaged(downloaded, sourceName(source), targetDirectory, limits);
++ installStaged(downloaded,
++ name == null ? sourceName(source) : validateSourceName(name),
++ targetDirectory, limits);
+ return targetDirectory;
+ } finally {
+ Files.deleteIfExists(downloaded);
+ }
+ }
+
+ /**
+ * Creates the file the download is written to. It is placed on the target's
+ * filesystem, not in the system temporary directory, so a large download cannot
+ * exhaust the system temporary directory while the target has room. It is hidden, so
+ * a leaked one is visible as residue rather than as an installed file.
+ *
+ * @param targetDirectory The directory to install into. Must already exist.
+ * @return The newly created, empty download file. Never {@code null}.
+ * @throws IOException Thrown if the file cannot be created.
+ */
+ static Path createDownloadFile(Path targetDirectory) throws IOException {
+ return Files.createTempFile(targetDirectory, DOWNLOAD_PREFIX, DOWNLOAD_SUFFIX);
+ }
+
+ /**
+ * Validates the checksum argument and normalizes it for comparison.
+ *
+ * @param checksum The digest as given by the caller, or {@code null} to skip.
+ * @return The trimmed digest, or {@code null} when verification is skipped.
+ * @throws IllegalArgumentException Thrown if the digest is neither a 64-character
+ * nor a 128-character hex string.
+ */
+ private static String validateChecksum(String checksum) {
+ if (checksum == null) {
+ return null;
+ }
+ final String trimmed = checksum.trim();
+ if ((trimmed.length() == SHA_256_HEX_LENGTH || trimmed.length() == SHA_512_HEX_LENGTH)
+ && isHex(trimmed)) {
+ return trimmed;
+ }
+ throw new IllegalArgumentException(
+ "checksum must be 64 (SHA-256) or 128 (SHA-512) hex characters; pass null to skip");
+ }
+
+ /**
+ * Checks whether a string is made up entirely of hexadecimal digits.
+ *
+ * @param value The string to inspect.
+ * @return {@code true} if every character is a hexadecimal digit.
+ */
+ private static boolean isHex(String value) {
+ for (int i = 0; i < value.length(); i++) {
+ final char c = value.charAt(i);
+ final boolean digit = c >= '0' && c <= '9';
+ final boolean lower = c >= 'a' && c <= 'f';
+ final boolean upper = c >= 'A' && c <= 'F';
+ if (!digit && !lower && !upper) {
+ return false;
+ }
+ }
+ return true;
+ }
+
+ /**
+ * Fetches the source into the given file, bounded by the download ceiling. Http and
+ * https locations are fetched with timeouts and the redirect policy; a {@code file}
+ * location is read directly. The scheme was accepted by {@link #validateSource(URI)}
+ * at the public boundary.
+ *
+ * @param source The resource location.
+ * @param file The file receiving the downloaded bytes.
+ * @param limits The limits to enforce.
+ * @throws IOException Thrown if fetching fails or a limit is exceeded.
+ */
+ private static void download(URI source, Path file, Limits limits) throws IOException {
+ final Budget budget = new Budget(limits.maxDownloadBytes(),
+ "download exceeds the ceiling of " + limits.maxDownloadBytes() + " bytes");
+ if (isHttp(source.getScheme())) {
+ downloadHttp(source, file, limits, budget);
+ } else {
+ try (InputStream in = Files.newInputStream(localFile(source))) {
+ copyBounded(in, file, budget);
+ }
+ }
+ }
+
+ /**
+ * Rejects a source the installer will not fetch. Only {@code http}, {@code https}, and
+ * {@code file} are accepted: any other scheme would be handed to whichever URL handler
+ * the runtime happens to have installed, outside the connection timeout, read timeout,
+ * and redirect policy this class enforces.
+ *
+ * @param source The resource location as given by the caller.
+ * @throws IllegalArgumentException Thrown if the scheme is absent or unsupported.
+ */
+ private static void validateSource(URI source) {
+ final String scheme = source.getScheme();
+ if (!isHttp(scheme) && !SCHEME_FILE.equalsIgnoreCase(scheme)) {
+ throw new IllegalArgumentException(
+ "source scheme must be http, https, or file, but was: " + source);
+ }
+ }
+
+ /**
+ * Classifies a scheme as one the http fetch path handles.
+ *
+ * @param scheme The URI scheme, or {@code null} when the location has none.
+ * @return {@code true} for {@code http} and {@code https}, ignoring case.
+ */
+ private static boolean isHttp(String scheme) {
+ return SCHEME_HTTP.equalsIgnoreCase(scheme) || SCHEME_HTTPS.equalsIgnoreCase(scheme);
+ }
+
+ /**
+ * Resolves a {@code file} location to a path on the default filesystem.
+ *
+ * @param source The {@code file} location, already validated as such.
+ * @return The local path. Never {@code null}.
+ * @throws IOException Thrown if the location does not name a file this runtime can
+ * open, such as a {@code file} URI naming a remote host.
+ */
+ private static Path localFile(URI source) throws IOException {
+ try {
+ return Path.of(source);
+ } catch (IllegalArgumentException | FileSystemNotFoundException e) {
+ throw new IOException("not a readable local file location: " + source, e);
+ }
+ }
+
+ /**
+ * Fetches an http or https source with connection and read timeouts, following at
+ * most the allowed number of redirects under the redirect policy, checking any
+ * declared content length against the download ceiling before reading the body, and
+ * bounding the transferred bytes against the same ceiling.
+ *
+ * @param source The resource location as requested by the caller.
+ * @param file The file receiving the downloaded bytes.
+ * @param limits The limits to enforce.
+ * @param budget The download budget shared with the caller.
+ * @throws IOException Thrown if fetching fails, the server answers with a status
+ * other than 200, the redirect policy is violated, or a limit is exceeded.
+ */
+ private static void downloadHttp(URI source, Path file, Limits limits, Budget budget)
+ throws IOException {
+ URI current = source;
+ int redirects = 0;
+ while (true) {
+ final HttpURLConnection connection =
+ (HttpURLConnection) current.toURL().openConnection();
+ connection.setInstanceFollowRedirects(false);
+ connection.setConnectTimeout(timeoutMillis(limits.connectTimeout()));
+ connection.setReadTimeout(timeoutMillis(limits.readTimeout()));
+ try {
+ final int status = connection.getResponseCode();
+ if (isRedirect(status)) {
+ if (redirects >= limits.maxRedirects()) {
+ throw new IOException(
+ "more than " + limits.maxRedirects() + " redirects: " + source);
+ }
+ current = resolveRedirect(current, connection.getHeaderField("Location"));
+ redirects++;
+ continue;
+ }
+ if (status != HttpURLConnection.HTTP_OK) {
+ throw new IOException(
+ "download failed with HTTP status " + status + ": " + current);
+ }
+ final long declared = connection.getContentLengthLong();
+ if (declared > limits.maxDownloadBytes()) {
+ throw new IOException("declared content length " + declared
+ + " exceeds the download ceiling of " + limits.maxDownloadBytes()
+ + " bytes");
+ }
+ try (InputStream in = connection.getInputStream()) {
+ copyBounded(in, file, budget);
+ }
+ return;
+ } finally {
+ connection.disconnect();
+ }
+ }
+ }
+
+ /**
+ * Classifies a response status as a redirect the installer follows.
+ *
+ * @param status The HTTP response status.
+ * @return {@code true} if the status is one of the redirect statuses 301, 302, 303,
+ * 307, or 308.
+ */
+ private static boolean isRedirect(int status) {
+ return status == HttpURLConnection.HTTP_MOVED_PERM
+ || status == HttpURLConnection.HTTP_MOVED_TEMP
+ || status == HttpURLConnection.HTTP_SEE_OTHER
+ || status == HTTP_TEMPORARY_REDIRECT
+ || status == HTTP_PERMANENT_REDIRECT;
+ }
+
+ /**
+ * Resolves a redirect location against the redirected request and enforces the
+ * redirect policy: the target must be an http or https location, and an https
+ * request must not be redirected to plain http.
+ *
+ * @param from The location that answered with the redirect.
+ * @param location The Location header value, absolute or relative, or {@code null}
+ * when the header is absent.
+ * @return The resolved redirect target. Never {@code null}.
+ * @throws IOException Thrown if the location is absent or malformed, leaves the
+ * http and https schemes, or downgrades https to http.
+ */
+ static URI resolveRedirect(URI from, String location) throws IOException {
+ if (location == null || location.isEmpty()) {
+ throw new IOException("redirect from " + from + " carries no Location header");
+ }
+ final URI target;
+ try {
+ target = from.resolve(location);
+ } catch (IllegalArgumentException e) {
+ throw new IOException(
+ "redirect from " + from + " carries a malformed Location: " + location, e);
+ }
+ final String scheme = target.getScheme();
+ final boolean https = SCHEME_HTTPS.equalsIgnoreCase(scheme);
+ if (!https && !SCHEME_HTTP.equalsIgnoreCase(scheme)) {
+ throw new IOException(
+ "redirect target is not an http or https location: " + target);
+ }
+ if (SCHEME_HTTPS.equalsIgnoreCase(from.getScheme()) && !https) {
+ throw new IOException("redirect downgrades https to http: " + target);
+ }
+ return target;
+ }
+
+ /**
+ * Converts a timeout to the millisecond form the connection setters take. A positive
+ * timeout shorter than a millisecond becomes one millisecond rather than zero, because
+ * {@link HttpURLConnection#setReadTimeout(int) zero means no timeout at all}, and a
+ * timeout too large for the int range is capped instead of overflowing.
+ *
+ * @param timeout The timeout as a duration. Must be positive.
+ * @return The timeout in milliseconds, at least {@code 1} and at most
+ * {@link Integer#MAX_VALUE}.
+ */
+ private static int timeoutMillis(Duration timeout) {
+ final long millis;
+ try {
+ millis = timeout.toMillis();
+ } catch (ArithmeticException e) {
+ return Integer.MAX_VALUE;
+ }
+ return Math.clamp(millis, 1, Integer.MAX_VALUE);
+ }
+
+ /**
+ * Computes the file's digest and compares it with the expected hex digest, ignoring
+ * hex letter case. The digest length selects the algorithm: 64 characters SHA-256,
+ * 128 characters SHA-512.
+ *
+ * @param file The file to digest.
+ * @param expected The expected hex digest, already trimmed.
+ * @throws IOException Thrown if the file cannot be read or the digests differ.
+ */
+ private static void verify(Path file, String expected) throws IOException {
+ final String algorithm =
+ expected.length() == SHA_512_HEX_LENGTH ? SHA_512 : SHA_256;
+ final MessageDigest digest;
+ try {
+ digest = MessageDigest.getInstance(algorithm);
+ } catch (NoSuchAlgorithmException e) {
+ throw new IOException(algorithm + " is unavailable in this runtime", e);
+ }
+ try (InputStream in = Files.newInputStream(file)) {
+ final byte[] buffer = new byte[BUFFER_SIZE];
+ int read;
+ while ((read = in.read(buffer)) >= 0) {
+ digest.update(buffer, 0, read);
+ }
+ }
+ final String actual = HexFormat.of().formatHex(digest.digest());
+ if (!actual.equalsIgnoreCase(expected)) {
+ throw new IOException(
+ "checksum mismatch: expected " + expected + " but downloaded " + actual);
+ }
+ }
+
+ /**
+ * Unpacks the downloaded content into a hidden staging directory beneath the target
+ * and promotes it into the target only after every entry unpacked cleanly. The
+ * staging directory lives on the target's filesystem so promotion is a sequence of
+ * renames, and it is removed whether the installation succeeds or fails.
+ *
+ * @param downloaded The fetched and verified file.
+ * @param name The file name derived from the source location.
+ * @param target The directory to install into.
+ * @param limits The limits to enforce while unpacking.
+ * @throws IOException Thrown if unpacking fails, a limit is exceeded, or promotion
+ * or staging cleanup fails.
+ */
+ private static void installStaged(Path downloaded, String name, Path target,
+ Limits limits) throws IOException {
+ final Path staging = Files.createTempDirectory(target, STAGING_PREFIX);
+ try {
+ unpack(downloaded, name, staging, limits);
+ promote(staging, target);
+ } catch (IOException e) {
+ try {
+ deleteRecursively(staging);
+ } catch (IOException cleanup) {
+ e.addSuppressed(cleanup);
+ }
+ throw e;
+ }
+ deleteRecursively(staging);
+ }
+
+ /**
+ * Moves every staged regular file to its relative location beneath the target,
+ * refusing to replace anything that already exists there. All destinations are
+ * checked before the first move, so a collision leaves the target without a mix of
+ * old and new files. Moves are attempted atomically and fall back to a plain move
+ * where the filesystem does not support it.
+ *
+ * @param staging The staging directory holding the fully unpacked content.
+ * @param target The directory to install into.
+ * @throws IOException Thrown if a destination already exists, a move fails, or a
+ * directory on the way to a destination is an existing symbolic link.
+ */
+ private static void promote(Path staging, Path target) throws IOException {
+ final List<Path> files;
+ try (Stream<Path> walk = Files.walk(staging)) {
+ files = walk.filter(Files::isRegularFile).toList();
+ }
+ for (final Path file : files) {
+ ensureVacant(target, staging.relativize(file));
+ }
+ for (final Path file : files) {
+ final Path destination = destination(target, staging.relativize(file));
+ try {
+ Files.move(file, destination, StandardCopyOption.ATOMIC_MOVE);
+ } catch (AtomicMoveNotSupportedException e) {
+ Files.move(file, destination);
+ }
+ }
+ }
+
+ /**
+ * Checks that one staged file's destination is free to receive it, without creating
+ * anything: an existing filesystem object at the destination is a collision, and a
+ * symbolic link or a non-directory on the way to it is refused exactly as
+ * {@link #destination(Path, Path)} refuses it during the move pass. A missing
+ * directory on the way proves the destination vacant.
+ *
+ * @param target The directory to install into.
+ * @param relative The staged file's path relative to the staging directory.
+ * @throws IOException Thrown if the destination already exists, or a directory on
+ * the way is a symbolic link or exists as something other than a directory.
+ */
+ private static void ensureVacant(Path target, Path relative) throws IOException {
+ Path directory = target;
+ for (int i = 0; i < relative.getNameCount() - 1; i++) {
+ directory = directory.resolve(relative.getName(i));
+ if (Files.isSymbolicLink(directory)) {
+ throw new IOException(
+ "installation path crosses a symbolic link: " + directory);
+ }
+ if (!Files.exists(directory)) {
+ return;
+ }
+ if (!Files.isDirectory(directory)) {
+ throw new IOException(
+ "installation path crosses an existing file: " + directory);
+ }
+ }
+ final Path destination = directory.resolve(relative.getFileName());
+ if (Files.exists(destination, LinkOption.NOFOLLOW_LINKS)) {
+ throw new IOException("target already contains: " + destination);
+ }
+ }
+
+ /**
+ * Resolves one staged file's destination beneath the target, creating the directories
+ * leading to it one at a time and refusing to descend through a symbolic link that is
+ * already there. An entry name that stays inside the staging directory can still land
+ * outside the target if a directory below the target is a link to somewhere else.
+ *
+ * <p>This covers links present when the installation runs. It is not a defense against
+ * a link created concurrently, between the check here and the move that follows.</p>
+ *
+ * @param target The directory to install into.
+ * @param relative The staged file's path relative to the staging directory.
+ * @return The destination path beneath the target. Never {@code null}.
+ * @throws IOException Thrown if a directory on the way is a symbolic link or exists as
+ * something other than a directory, or if a directory cannot be created.
+ */
+ private static Path destination(Path target, Path relative) throws IOException {
+ Path directory = target;
+ for (int i = 0; i < relative.getNameCount() - 1; i++) {
+ directory = directory.resolve(relative.getName(i));
+ if (Files.isSymbolicLink(directory)) {
+ throw new IOException(
+ "installation path crosses a symbolic link: " + directory);
+ }
+ if (!Files.exists(directory)) {
+ Files.createDirectory(directory);
+ } else if (!Files.isDirectory(directory)) {
+ throw new IOException(
+ "installation path crosses an existing file: " + directory);
+ }
+ }
+ return directory.resolve(relative.getFileName());
+ }
+
+ /**
+ * Removes the given directory tree, deepest entries first.
+ *
+ * @param root The directory to remove.
+ * @throws IOException Thrown if a deletion fails.
+ */
+ private static void deleteRecursively(Path root) throws IOException {
+ final List<Path> paths;
+ try (Stream<Path> walk = Files.walk(root)) {
+ paths = walk.sorted(Comparator.reverseOrder()).toList();
+ }
+ for (final Path path : paths) {
+ Files.deleteIfExists(path);
+ }
+ }
+
+ /**
+ * Detects the content format from its leading bytes and unpacks accordingly,
+ * bounding the total expanded bytes against the expansion ceiling. One exception: a
+ * source named {@code *.bin} is stored verbatim even when its bytes are a zip
+ * archive, because that is exactly what an OpenNLP model file is: a zipped artifact
+ * that consumers load packed. Unpacking it would deliver its innards
+ * ({@code manifest.properties}, {@code *.model}) where the operator asked for the
+ * model.
+ *
+ * @param downloaded The fetched file.
+ * @param name The file name derived from the source location.
+ * @param staging The staging directory to unpack into.
+ * @param limits The limits to enforce.
+ * @throws IOException Thrown if reading or unpacking fails or the expansion ceiling
+ * is exceeded.
+ */
+ private static void unpack(Path downloaded, String name, Path staging, Limits limits)
+ throws IOException {
+ final Budget budget = new Budget(limits.maxExpandedBytes(),
+ "expanded content exceeds the ceiling of " + limits.maxExpandedBytes()
+ + " bytes");
+ final Budget entryBudget = new Budget(limits.maxEntries(),
+ "archive entry count exceeds the ceiling of " + limits.maxEntries()
+ + " entries");
+ try (InputStream raw = new BufferedInputStream(Files.newInputStream(downloaded))) {
+ raw.mark(MAGIC_LENGTH);
+ final int first = raw.read();
+ final int second = raw.read();
+ raw.reset();
+ if (name.endsWith(MODEL_SUFFIX)) {
+ copyBounded(raw, safeChild(staging, name), budget);
+ } else if (first == GZIP_MAGIC_FIRST && second == GZIP_MAGIC_SECOND) {
+ unpackGzip(raw, name, staging, budget, entryBudget);
+ } else if (first == ZIP_MAGIC_FIRST && second == ZIP_MAGIC_SECOND) {
+ unpackZip(raw, staging, budget, entryBudget);
+ } else {
+ copyBounded(raw, safeChild(staging, name), budget);
+ }
+ }
+ }
+
+ /**
+ * Unpacks gzip content: a tar archive inside when present, a plain file otherwise. A
+ * plain file loses the {@code .gz} suffix of its source name.
+ *
+ * @param raw The gzip-compressed content.
+ * @param name The file name derived from the source location.
+ * @param staging The staging directory to unpack into.
+ * @param budget The expansion budget.
+ * @param entryBudget The entry-count budget.
+ * @throws IOException Thrown if decompressing or unpacking fails or a ceiling is
+ * exceeded.
+ */
+ private static void unpackGzip(InputStream raw, String name, Path staging,
+ Budget budget, Budget entryBudget) throws IOException {
+ final InputStream decompressed = new BufferedInputStream(
+ new BudgetInputStream(new GZIPInputStream(raw), budget), BUFFER_SIZE);
+ if (TarStream.startsWithHeader(decompressed)) {
+ unpackTar(decompressed, staging, entryBudget);
+ } else {
+ final String plainName = name.endsWith(GZIP_SUFFIX)
+ ? name.substring(0, name.length() - GZIP_SUFFIX.length()) : name;
+ copy(decompressed, safeChild(staging, plainName));
+ }
+ }
+
+ /**
+ * Unpacks every regular tar entry to its relative location beneath the staging
+ * directory.
+ *
+ * @param decompressed The uncompressed tar content.
+ * @param staging The staging directory to unpack into.
+ * @param entryBudget The entry-count budget, charged for every entry including
+ * directories.
+ * @throws IOException Thrown if the archive is malformed, an entry escapes the
+ * staging directory, or the entry ceiling is exceeded.
+ */
+ private static void unpackTar(InputStream decompressed, Path staging,
+ Budget entryBudget) throws IOException {
+ final TarStream entries = new TarStream(decompressed);
+ while (entries.next()) {
+ entryBudget.spend(1);
+ if (!entries.isFile()) {
+ continue;
+ }
+ final Path file = safeChild(staging, entries.name());
+ Files.createDirectories(file.getParent());
+ copy(entries.entryStream(), file);
+ }
+ }
+
+ /**
+ * Unpacks every regular zip entry to its relative location beneath the staging
+ * directory.
+ *
+ * @param raw The zip content.
+ * @param staging The staging directory to unpack into.
+ * @param budget The expansion budget.
+ * @param entryBudget The entry-count budget, charged for every entry including
+ * directories.
+ * @throws IOException Thrown if the archive is malformed, an entry escapes the
+ * staging directory, or a ceiling is exceeded.
+ */
+ private static void unpackZip(InputStream raw, Path staging, Budget budget,
+ Budget entryBudget) throws IOException {
+ final ZipInputStream zip = new ZipInputStream(raw);
+ ZipEntry entry;
+ while ((entry = zip.getNextEntry()) != null) {
+ entryBudget.spend(1);
+ if (entry.isDirectory()) {
+ continue;
+ }
+ final Path file = safeChild(staging, entry.getName());
+ Files.createDirectories(file.getParent());
+ copyBounded(zip, file, budget);
+ }
+ }
+
+ /**
+ * Copies the stream into the file, charging every byte against the budget before it
+ * is written, so an oversized transfer aborts within one buffer of its ceiling.
+ *
+ * @param in The content to copy.
+ * @param file The file to write.
+ * @param budget The byte budget to charge.
+ * @throws IOException Thrown if reading or writing fails or the budget is exceeded.
+ */
+ private static void copyBounded(InputStream in, Path file, Budget budget)
+ throws IOException {
+ try (OutputStream out = Files.newOutputStream(file)) {
+ final byte[] buffer = new byte[BUFFER_SIZE];
+ int read;
+ while ((read = in.read(buffer)) >= 0) {
+ budget.spend(read);
+ out.write(buffer, 0, read);
+ }
+ }
+ }
+
+ /**
+ * Copies the stream into the file. The caller is responsible for applying any byte
+ * ceiling to the input stream.
+ *
+ * @param in The content to copy.
+ * @param file The file to write.
+ * @throws IOException Thrown if reading or writing fails.
+ */
+ private static void copy(InputStream in, Path file) throws IOException {
+ try (OutputStream out = Files.newOutputStream(file)) {
+ in.transferTo(out);
+ }
+ }
+
+ /**
+ * Resolves an archive entry inside the staging directory, rejecting escaping paths.
+ *
+ * @param staging The staging directory to unpack into.
+ * @param entryName The entry name as stored in the archive.
+ * @return The resolved path beneath the staging directory. Never {@code null}.
+ * @throws IOException Thrown if the entry resolves outside the staging directory.
+ */
+ private static Path safeChild(Path staging, String entryName) throws IOException {
+ final Path resolved = staging.resolve(entryName).normalize();
+ if (!resolved.startsWith(staging.normalize())) {
+ throw new IOException("archive entry escapes the target directory: " + entryName);
+ }
+ return resolved;
+ }
+
+ /**
+ * Derives a file name from the source URI for non-archive content.
+ *
+ * @param source The resource location.
+ * @return The last path segment, or {@code resource} if the location has none.
+ */
+ private static String sourceName(URI source) {
+ final String path = source.getPath();
+ if (path == null || path.isEmpty()) {
+ return DEFAULT_RESOURCE_NAME;
+ }
+ final int slash = path.lastIndexOf('/');
+ final String name = slash < 0 ? path : path.substring(slash + 1);
+ return name.isEmpty() ? DEFAULT_RESOURCE_NAME : name;
+ }
+
++ private static String validateSourceName(String name) {
++ if (name.isEmpty() || ".".equals(name) || "..".equals(name)) {
++ throw new IllegalArgumentException("name must be a file name");
++ }
++ for (int i = 0; i < name.length(); i++) {
++ final char c = name.charAt(i);
++ if (c == '/' || c == '\\' || c == 0) {
++ throw new IllegalArgumentException("name must be a file name");
++ }
++ }
++ return name;
++ }
++
+ /**
+ * A unit budget, counting bytes or archive entries: {@link #spend(long)} accumulates
+ * spent units and fails once the ceiling is crossed.
+ */
+ private static final class Budget {
+
+ private final long ceiling;
+ private final String message;
+ private long used;
+
+ /**
+ * Creates a budget that has spent nothing yet.
+ *
+ * @param ceiling The largest total number of units accepted.
+ * @param message The failure message raised when the ceiling is crossed.
+ */
+ Budget(long ceiling, String message) {
+ this.ceiling = ceiling;
+ this.message = message;
+ }
+
+ /**
+ * Charges the given number of units against the budget.
+ *
+ * @param units The number of units to charge.
+ * @throws IOException Thrown if the total charged units exceed the ceiling.
+ */
+ void spend(long units) throws IOException {
+ used += units;
+ if (used > ceiling) {
+ throw new IOException(message);
+ }
+ }
+ }
+
+ /**
+ * Charges every byte read or skipped from an expanded stream against a shared budget.
+ */
+ private static final class BudgetInputStream extends FilterInputStream {
+
+ private final Budget budget;
+
+ /**
+ * Initializes a budgeted stream.
+ *
+ * @param in The expanded stream to read.
+ * @param budget The budget to charge.
+ */
+ BudgetInputStream(InputStream in, Budget budget) {
+ super(in);
+ this.budget = budget;
+ }
+
+ /**
+ * {@inheritDoc}
+ */
+ @Override
+ public int read() throws IOException {
+ final int value = super.read();
+ if (value >= 0) {
+ budget.spend(1);
+ }
+ return value;
+ }
+
+ /**
+ * {@inheritDoc}
+ */
+ @Override
+ public int read(byte[] buffer, int offset, int length) throws IOException {
+ final int read = super.read(buffer, offset, length);
+ if (read > 0) {
+ budget.spend(read);
+ }
+ return read;
+ }
+
+ /**
+ * {@inheritDoc}
+ */
+ @Override
+ public long skip(long bytes) throws IOException {
+ final long skipped = super.skip(bytes);
+ budget.spend(skipped);
+ return skipped;
+ }
+ }
+}
diff --cc opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/stemmer/hunspell/HunspellDictionaryDownloadTest.java
index 23ab31dd0,67d85af69..b5719a0d0
--- a/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/stemmer/hunspell/HunspellDictionaryDownloadTest.java
+++ b/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/stemmer/hunspell/HunspellDictionaryDownloadTest.java
@@@ -25,6 -28,8 +28,7 @@@ import org.junit.jupiter.api.Test
import org.junit.jupiter.api.io.TempDir;
import opennlp.tools.util.DictionaryCatalog;
+ import opennlp.tools.util.DigestTestUtil;
-import opennlp.tools.util.DownloadUtil;
/**
* Pins the Hunspell catalog download gate; network fetches are not exercised here.
@@@ -36,36 -41,108 +40,111 @@@ public class HunspellDictionaryDownload
* the property name in the message, leaving the previous property value restored.
*
* @param target A scratch directory managed by the test framework.
- */
+ * @throws IOException Thrown if the local catalog cannot be prepared.
- */
++ */
@Test
- void testDownloadRequiresRemoteProperty(@TempDir Path target) {
- final String previous = System.getProperty(DictionaryCatalog.REMOTE_DOWNLOAD_PROPERTY);
+ void testDownloadRequiresRemoteProperty(@TempDir Path target) throws IOException {
+ final DictionaryCatalog catalog = localCatalog(target);
- final String previous = System.getProperty(DownloadUtil.REMOTE_DOWNLOAD_PROPERTY);
- System.clearProperty(DownloadUtil.REMOTE_DOWNLOAD_PROPERTY);
++ final String previous =
++ System.getProperty(DictionaryCatalog.REMOTE_DOWNLOAD_PROPERTY);
+ System.clearProperty(DictionaryCatalog.REMOTE_DOWNLOAD_PROPERTY);
try {
final IOException e = Assertions.assertThrows(IOException.class,
- () -> HunspellDictionaryDownload.downloadFromCatalog("en_US", target));
- Assertions.assertTrue(e.getMessage().contains(DictionaryCatalog.REMOTE_DOWNLOAD_PROPERTY));
+ () -> HunspellDictionaryDownload.downloadFromCatalog(catalog, "demo", target));
- Assertions.assertTrue(e.getMessage().contains(DownloadUtil.REMOTE_DOWNLOAD_PROPERTY));
++ Assertions.assertTrue(
++ e.getMessage().contains(DictionaryCatalog.REMOTE_DOWNLOAD_PROPERTY));
+ } finally {
+ restore(previous);
+ }
+ }
+
+ /**
+ * Verifies that an application-supplied catalog downloads a Hunspell pair and its
+ * license readme under the names declared by that catalog.
+ *
+ * @param target A scratch directory managed by the test framework.
+ * @throws IOException Thrown if the local catalog cannot be prepared or downloaded.
+ */
+ @Test
+ void testDownloadsFromApplicationCatalog(@TempDir Path target) throws IOException {
+ final DictionaryCatalog catalog = localCatalog(target);
+ final Path output = target.resolve("output");
- final String previous = System.getProperty(DownloadUtil.REMOTE_DOWNLOAD_PROPERTY);
- System.setProperty(DownloadUtil.REMOTE_DOWNLOAD_PROPERTY, "true");
++ final String previous =
++ System.getProperty(DictionaryCatalog.REMOTE_DOWNLOAD_PROPERTY);
++ System.setProperty(DictionaryCatalog.REMOTE_DOWNLOAD_PROPERTY, "true");
+ try {
+ HunspellDictionaryDownload.downloadFromCatalog(catalog, "demo", output);
+ Assertions.assertEquals("SET UTF-8\n",
+ Files.readString(output.resolve("demo" + HunspellDictionary.AFFIX_FILE_SUFFIX)));
+ Assertions.assertEquals("1\nword\n",
+ Files.readString(output.resolve("demo" + HunspellDictionary.DICTIONARY_FILE_SUFFIX)));
+ Assertions.assertEquals("license\n", Files.readString(output.resolve("README.txt")));
} finally {
- if (previous == null) {
- System.clearProperty(DictionaryCatalog.REMOTE_DOWNLOAD_PROPERTY);
- } else {
- System.setProperty(DictionaryCatalog.REMOTE_DOWNLOAD_PROPERTY, previous);
- }
+ restore(previous);
}
}
/**
- * Verifies that the shipped catalog holds the {@code en_US} pair and its license
- * readme, each with a full-length SHA-512 digest.
+ * Verifies that each required parameter is checked before a download begins.
*
- * @throws IOException Thrown if the shipped catalog fails to load.
+ * @param target A scratch directory managed by the test framework.
+ * @throws IOException Thrown if the local catalog cannot be prepared.
*/
@Test
- void testCatalogContainsEnUsPair() throws IOException {
- final DictionaryCatalog catalog = DictionaryCatalog.loadDefault();
- Assertions.assertTrue(catalog.ids().contains("hunspell.en_US.aff"));
- Assertions.assertTrue(catalog.ids().contains("hunspell.en_US.dic"));
- Assertions.assertTrue(catalog.ids().contains("hunspell.en_US.readme"));
- Assertions.assertEquals(128, catalog.get("hunspell.en_US.aff").sha512().length());
+ void testRejectsNullParameters(@TempDir Path target) throws IOException {
+ final DictionaryCatalog catalog = localCatalog(target);
+ Assertions.assertThrows(IllegalArgumentException.class,
+ () -> HunspellDictionaryDownload.downloadFromCatalog(null, "demo", target));
+ Assertions.assertThrows(IllegalArgumentException.class,
+ () -> HunspellDictionaryDownload.downloadFromCatalog(catalog, null, target));
+ Assertions.assertThrows(IllegalArgumentException.class,
+ () -> HunspellDictionaryDownload.downloadFromCatalog(catalog, "demo", null));
+ }
+
+ /**
+ * Creates an application-supplied catalog backed by local files.
+ *
+ * @param directory The directory to hold the source files.
+ * @return The loaded catalog. Never {@code null}.
+ * @throws IOException Thrown if the source files cannot be written.
+ */
+ private static DictionaryCatalog localCatalog(Path directory) throws IOException {
+ final byte[] affix = "SET UTF-8\n".getBytes(StandardCharsets.UTF_8);
+ final byte[] dictionary = "1\nword\n".getBytes(StandardCharsets.UTF_8);
+ final byte[] readme = "license\n".getBytes(StandardCharsets.UTF_8);
+ final Path affixSource = directory.resolve(
+ "source" + HunspellDictionary.AFFIX_FILE_SUFFIX);
+ final Path dictionarySource = directory.resolve(
+ "source" + HunspellDictionary.DICTIONARY_FILE_SUFFIX);
+ final Path readmeSource = directory.resolve("source-readme.txt");
+ Files.write(affixSource, affix);
+ Files.write(dictionarySource, dictionary);
+ Files.write(readmeSource, readme);
+
+ final String prefix = "hunspell.demo";
+ final String catalog = entry(prefix + HunspellDictionary.AFFIX_FILE_SUFFIX,
+ affixSource, affix, "demo" + HunspellDictionary.AFFIX_FILE_SUFFIX)
+ + entry(prefix + HunspellDictionary.DICTIONARY_FILE_SUFFIX,
+ dictionarySource, dictionary,
+ "demo" + HunspellDictionary.DICTIONARY_FILE_SUFFIX)
+ + entry(prefix + ".readme", readmeSource, readme, "README.txt");
+ return DictionaryCatalog.load(
+ new ByteArrayInputStream(catalog.getBytes(StandardCharsets.UTF_8)));
+ }
+
+ /** Builds one properties entry for a local source file. */
+ private static String entry(String id, Path source, byte[] content, String filename) {
+ return id + ".url=" + source.toUri() + "\n"
+ + id + ".sha512=" + DigestTestUtil.sha512(content) + "\n"
+ + id + ".filename=" + filename + "\n";
+ }
+
+ /** Restores the remote-download property to its previous value. */
+ private static void restore(String previous) {
+ if (previous == null) {
- System.clearProperty(DownloadUtil.REMOTE_DOWNLOAD_PROPERTY);
++ System.clearProperty(DictionaryCatalog.REMOTE_DOWNLOAD_PROPERTY);
+ } else {
- System.setProperty(DownloadUtil.REMOTE_DOWNLOAD_PROPERTY, previous);
++ System.setProperty(DictionaryCatalog.REMOTE_DOWNLOAD_PROPERTY, previous);
+ }
}
}
diff --cc opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/tokenize/lattice/MecabDictionaryInstallerTest.java
index adbfe26f3,000000000..5fa043602
mode 100644,000000..100644
--- a/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/tokenize/lattice/MecabDictionaryInstallerTest.java
+++ b/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/tokenize/lattice/MecabDictionaryInstallerTest.java
@@@ -1,254 -1,0 +1,265 @@@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package opennlp.tools.tokenize.lattice;
+
++import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.net.URI;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.Path;
+
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+
+import opennlp.tools.util.DictionaryCatalog;
+import opennlp.tools.util.DigestTestUtil;
+import opennlp.tools.util.archive.TarArchives;
+
+/**
+ * Tests the installer against project-authored, in-memory archives; no external
+ * dictionary data and no network access are involved. Fetch, verification, and
+ * unpacking limits are exercised in {@code opennlp.tools.util.ResourceInstallerTest},
+ * which tests the shared installation path this installer delegates to.
+ */
+public class MecabDictionaryInstallerTest {
+
+ @Test
+ void testInstallsDictionaryFilesAndFlattensPaths(@TempDir Path source,
+ @TempDir Path target) throws IOException {
+ final Path archiveFile = archive(source, new String[][] {
+ {"dict-1.0/lexicon.csv", "cat,0,0,100,noun\n"},
+ {"dict-1.0/matrix.def", "1 1\n0 0 0\n"},
+ {"dict-1.0/char.def", "DEFAULT 0 1 0\n"},
+ {"dict-1.0/unk.def", "DEFAULT,0,0,10000,unknown\n"},
+ {"dict-1.0/README", "not a dictionary file"},
+ {"dict-1.0/dicrc", "config"}});
+
+ final int installed =
+ MecabDictionaryInstaller.install(archiveFile.toUri(), target);
+
+ Assertions.assertEquals(5, installed);
+ Assertions.assertTrue(Files.exists(target.resolve("lexicon.csv")));
+ Assertions.assertTrue(Files.exists(target.resolve("matrix.def")));
+ Assertions.assertTrue(Files.exists(target.resolve("char.def")));
+ Assertions.assertTrue(Files.exists(target.resolve("unk.def")));
+ Assertions.assertTrue(Files.exists(target.resolve("dicrc")));
+ Assertions.assertTrue(Files.notExists(target.resolve("README")));
+ Assertions.assertEquals("cat,0,0,100,noun\n",
+ Files.readString(target.resolve("lexicon.csv")));
+ }
+
+ /**
+ * Verifies that only files at the archive root count as dictionary payload.
+ * mecab-ko-dic ships template user dictionaries under {@code user-dic/} whose
+ * numeric fields are empty, input for {@code mecab-dict-index} rather than loadable
+ * lexicon data. Flattening them next to the real lexicon fails the subsequent load,
+ * and on a case-insensitive file system a template can silently overwrite a real
+ * lexicon file of the same base name.
+ */
+ @Test
+ void testNestedTemplateFilesAreNotInstalled(@TempDir Path source,
+ @TempDir Path target) throws IOException {
+ final Path archiveFile = archive(source, new String[][] {
+ {"dict-1.0/NNP.csv", "cat,1786,3546,2953,noun\n"},
+ {"dict-1.0/matrix.def", "1 1\n0 0 0\n"},
+ {"dict-1.0/user-dic/person.csv", "template,,,,noun\n"}});
+
+ final int installed =
+ MecabDictionaryInstaller.install(archiveFile.toUri(), target);
+
+ Assertions.assertEquals(2, installed);
+ Assertions.assertTrue(Files.exists(target.resolve("NNP.csv")));
+ Assertions.assertTrue(Files.notExists(target.resolve("person.csv")));
+ }
+
+ /**
+ * Verifies that two payload entries that flatten to the same base name are refused,
+ * because keeping either one silently would install an ambiguous dictionary.
+ */
+ @Test
+ void testEntriesFlatteningToTheSameNameAreRefused(@TempDir Path source,
+ @TempDir Path target) throws IOException {
+ final Path archiveFile = archive(source, new String[][] {
+ {"words.csv", "cat,0,0,100,noun\n"},
+ {"d/words.csv", "dog,0,0,100,noun\n"}});
+
+ final IOException e = Assertions.assertThrows(IOException.class,
+ () -> MecabDictionaryInstaller.install(archiveFile.toUri(), target));
+ Assertions.assertEquals(
+ "the archive flattens two entries to the same name: words.csv",
+ e.getMessage());
+ Assertions.assertTrue(Files.notExists(target.resolve("words.csv")));
+ }
+
+ @Test
+ void testInstallReadsAFileUri(@TempDir Path source, @TempDir Path target)
+ throws IOException {
+ final Path archiveFile = archive(source, new String[][] {
+ {"d/words.csv", "cat,0,0,100,noun\n"},
+ {"d/matrix.def", "1 1\n0 0 0\n"}});
+
+ final int installed =
+ MecabDictionaryInstaller.install(archiveFile.toUri(), target);
+
+ Assertions.assertEquals(2, installed);
+ Assertions.assertTrue(Files.exists(target.resolve("words.csv")));
+ }
+
+ /**
+ * Verifies that a pax long-name entry installs under its real name. The common
+ * distributions ship plain ustar today, but {@code bsdtar} and
+ * {@code tar --format=posix} write pax archives, whose over-100-byte names live in an
+ * extension header rather than the header name field.
+ */
+ @Test
+ void testPaxLongNamedEntryInstallsUnderItsRealName(@TempDir Path source,
+ @TempDir Path target) throws IOException {
+ final String longBaseName = "a".repeat(110) + ".csv";
+ final ByteArrayOutputStream tar = new ByteArrayOutputStream();
+ TarArchives.entry(tar, "PaxHeaders.0/lexicon",
+ TarArchives.paxRecord("path", "dict-1.0/" + longBaseName), 'x');
+ TarArchives.entry(tar, "dict-1.0/" + "a".repeat(88),
+ "cat,0,0,100,noun\n".getBytes(StandardCharsets.UTF_8));
+ tar.write(new byte[TarArchives.TERMINATOR_SIZE]);
+ final Path archiveFile = source.resolve("dict.tar.gz");
+ Files.write(archiveFile, TarGzArchives.gzip(tar.toByteArray()));
+
+ final int installed =
+ MecabDictionaryInstaller.install(archiveFile.toUri(), target);
+
+ Assertions.assertEquals(1, installed);
+ Assertions.assertEquals("cat,0,0,100,noun\n",
+ Files.readString(target.resolve(longBaseName)));
+ }
+
+ /**
+ * Verifies that installing into a target that already holds a dictionary file of the
+ * same name is refused, leaving the first installation in place. Refreshing a
+ * dictionary means removing its old files first.
+ */
+ @Test
+ void testReinstallOverAnExistingDictionaryIsRefused(@TempDir Path source,
+ @TempDir Path target) throws IOException {
+ final Path archiveFile = archive(source, new String[][] {
+ {"d/words.csv", "cat,0,0,100,noun\n"}});
+ Assertions.assertEquals(1,
+ MecabDictionaryInstaller.install(archiveFile.toUri(), target));
+
+ final IOException e = Assertions.assertThrows(IOException.class,
+ () -> MecabDictionaryInstaller.install(archiveFile.toUri(), target));
+ Assertions.assertTrue(e.getMessage().contains("target already contains: "));
+ Assertions.assertEquals("cat,0,0,100,noun\n",
+ Files.readString(target.resolve("words.csv")));
+ }
+
+ @Test
+ void testRemoteInstallWithoutDigestFailsLoud(@TempDir Path target) {
+ final IllegalArgumentException e =
+ Assertions.assertThrows(IllegalArgumentException.class,
+ () -> MecabDictionaryInstaller.install(
+ URI.create("https://example.invalid/dict.tar.gz"), target));
+ Assertions.assertTrue(
+ e.getMessage().contains("checksum must be given for an http or https source"));
+ }
+
+ @Test
+ void testInstallVerifiesDigest(@TempDir Path source, @TempDir Path target)
+ throws Exception {
+ final Path archiveFile = archive(source, new String[][] {
+ {"d/words.csv", "cat,0,0,100,noun\n"},
+ {"d/matrix.def", "1 1\n0 0 0\n"}});
+ final byte[] archive = Files.readAllBytes(archiveFile);
+
+ final int installed = MecabDictionaryInstaller.install(
+ archiveFile.toUri(), target, DigestTestUtil.sha512(archive));
+ Assertions.assertEquals(2, installed);
+
+ final IOException e = Assertions.assertThrows(IOException.class,
+ () -> MecabDictionaryInstaller.install(archiveFile.toUri(),
+ target.resolve("other"), DigestTestUtil.sha512(new byte[] {1})));
+ Assertions.assertTrue(e.getMessage().contains("checksum mismatch: expected"));
+ }
+
+ @Test
- void testInstallFromCatalogRequiresRemoteProperty(@TempDir Path target) {
++ void testInstallFromCatalogRequiresRemoteProperty(@TempDir Path target)
++ throws IOException {
++ final DictionaryCatalog catalog = emptyCatalog();
+ final String previous =
+ System.getProperty(DictionaryCatalog.REMOTE_DOWNLOAD_PROPERTY);
+ System.clearProperty(DictionaryCatalog.REMOTE_DOWNLOAD_PROPERTY);
+ try {
+ final IOException e = Assertions.assertThrows(IOException.class,
- () -> MecabDictionaryInstaller.installFromCatalog("mecab.ipadic", target));
++ () -> MecabDictionaryInstaller.installFromCatalog(
++ catalog, "mecab.ipadic", target));
+ Assertions.assertTrue(
+ e.getMessage().contains(DictionaryCatalog.REMOTE_DOWNLOAD_PROPERTY));
+ } finally {
+ if (previous == null) {
+ System.clearProperty(DictionaryCatalog.REMOTE_DOWNLOAD_PROPERTY);
+ } else {
+ System.setProperty(DictionaryCatalog.REMOTE_DOWNLOAD_PROPERTY, previous);
+ }
+ }
+ }
+
+ @Test
+ void testArchivesWithoutDictionaryFilesFailLoud(@TempDir Path source,
+ @TempDir Path target) throws IOException {
+ final Path archiveFile =
+ archive(source, new String[][] {{"readme.txt", "nothing here"}});
+ final IOException e = Assertions.assertThrows(IOException.class,
+ () -> MecabDictionaryInstaller.install(archiveFile.toUri(), target));
+ Assertions.assertEquals("the archive contains no dictionary file", e.getMessage());
+ }
+
+ @Test
- void testInvalidArguments(@TempDir Path target) {
++ void testInvalidArguments(@TempDir Path target) throws IOException {
++ final DictionaryCatalog catalog = emptyCatalog();
+ Assertions.assertThrows(IllegalArgumentException.class,
+ () -> MecabDictionaryInstaller.install(null, target));
+ Assertions.assertThrows(IllegalArgumentException.class,
+ () -> MecabDictionaryInstaller.install(target.toUri(), null));
+ Assertions.assertThrows(IllegalArgumentException.class,
- () -> MecabDictionaryInstaller.installFromCatalog(null, target));
++ () -> MecabDictionaryInstaller.installFromCatalog(null, "mecab.ipadic", target));
++ Assertions.assertThrows(IllegalArgumentException.class,
++ () -> MecabDictionaryInstaller.installFromCatalog(catalog, null, target));
+ Assertions.assertThrows(IllegalArgumentException.class,
- () -> MecabDictionaryInstaller.installFromCatalog("mecab.ipadic", null));
++ () -> MecabDictionaryInstaller.installFromCatalog(catalog, "mecab.ipadic", null));
++ }
++
++ private static DictionaryCatalog emptyCatalog() throws IOException {
++ return DictionaryCatalog.load(new ByteArrayInputStream(new byte[0]));
+ }
+
+ /**
+ * Writes a gzip-compressed tar archive of the given entries to a file.
+ *
+ * @param directory The directory to write the archive into.
+ * @param entries The entries as {@code {name, content}} pairs.
+ * @return The archive file. Never {@code null}.
+ * @throws IOException Thrown if writing fails.
+ */
+ private static Path archive(Path directory, String[][] entries) throws IOException {
+ final Path archiveFile = directory.resolve("dict.tar.gz");
+ Files.write(archiveFile, TarGzArchives.gzippedTar(entries));
+ return archiveFile;
+ }
+}
diff --cc opennlp-docs/src/docbkx/stemmer.xml
index e523b5d76,d777842fd..5ba7b9d96
--- a/opennlp-docs/src/docbkx/stemmer.xml
+++ b/opennlp-docs/src/docbkx/stemmer.xml
@@@ -124,12 -94,9 +94,12 @@@ stemmer.stem("table"); // "table" (
published dictionary yields for a given form is decided by that dictionary.
Acquisition helpers and the supported affix feature set live in
<code>dev/README-hunspell-dictionaries.md</code>.
- An opt-in catalog download
- (<code>HunspellDictionaryDownload.downloadFromCatalog</code>) needs
- <code>-Dopennlp.download.remote=true</code> and fetches through the
- digest-verified <code>ResourceInstaller</code> path, storing each file under
- its source name; a file that already exists in the target is refused, so
- refreshing a dictionary means removing its old files first.
+ An opt-in catalog download accepts an application-supplied
+ <code>DictionaryCatalog</code>, needs
- <code>-Dopennlp.download.remote=true</code>, and verifies SHA-512 digests.
++ <code>-Dopennlp.download.remote=true</code>, and fetches through the
++ digest-verified <code>ResourceInstaller</code> path. A file that already exists
++ in the target is refused, so refreshing a dictionary means removing its old
++ files first.
Directives that would change stems when ignored
(<code>ICONV</code>, <code>OCONV</code>, <code>COMPLEXPREFIXES</code>,
<code>COMPOUNDRULE</code>, <code>IGNORE</code>, <code>KEEPCASE</code>)
diff --cc opennlp-docs/src/docbkx/tokenizer.xml
index 85ada889e,cd1d8a2dd..9a513f795
--- a/opennlp-docs/src/docbkx/tokenizer.xml
+++ b/opennlp-docs/src/docbkx/tokenizer.xml
@@@ -538,139 -538,5 +538,49 @@@ wordTokenizer.tokenize("The quick brow
</programlisting>
</para>
</section>
+ <section xml:id="tools.tokenizer.lattice">
+ <title>Lattice tokenization for CJK</title>
+ <para>
+ Languages written without spaces need a dictionary-backed segmenter.
+ <code>LatticeTokenizer</code> scores paths over a MeCab-format dictionary and
+ emits the cheapest segmentation with spans in original text coordinates.
+ <code>UnigramSegmenter</code> does the same from a plain frequency lexicon.
+ Install a dictionary archive with <code>MecabDictionaryInstaller</code>, load it
+ as a <code>MecabDictionary</code>, and tokenize. Remote archives require an
- expected SHA-512 digest; the opt-in catalog
++ expected SHA-512 digest; an application-supplied opt-in catalog
+ (<code>installFromCatalog</code>) also needs
+ <code>-Dopennlp.download.remote=true</code>.
+ <code>matrix.def</code> must list a cost for every declared cell; matrix
+ dimensions plus lexicon size are bounded by the shared
+ <code>ResourceLimits.MAX_ENTRIES</code> limit and the matrix cell count
+ by <code>ResourceLimits.MAX_MATRIX_CELLS</code>.
+ Fetching, verification, and unpacking go through the bounded
+ <code>ResourceInstaller</code> path described in the model loading chapter,
+ whose ceilings default to a 1 GiB download, 4 GiB unpacked, and 100000
+ archive entries and can be raised at JVM startup via
+ <code>opennlp.download.max.bytes</code>,
+ <code>opennlp.install.max.total.bytes</code>, and
+ <code>opennlp.install.max.entries</code> for larger dictionaries such as
+ UniDic. Installed files are flattened to their base names; a file that
+ already exists in the target is refused, so refreshing a dictionary means
+ removing its old files first.
+ Load also rejects an <code>unk.def</code> template for a category
+ <code>char.def</code> never defined, and accepts MeCab-quoted CSV fields.
+ <code>LatticeUsageExampleTest</code> asserts the install-load-tokenize and
+ lexicon flows shown here.
+ <programlisting language="java">
+<![CDATA[MecabDictionaryInstaller.install(archiveUri, dictionaryDirectory, sha512);
+LatticeTokenizer tokenizer =
+ new LatticeTokenizer(MecabDictionary.load(dictionaryDirectory));
+
+String[] tokens = tokenizer.tokenize(text);
+Span[] spans = tokenizer.tokenizePos(text);
+List<Morpheme> morphemes = tokenizer.analyze(text);
+
+UnigramSegmenter segmenter = UnigramSegmenter.load(Path.of("words.txt"));
+String[] words = segmenter.tokenize(text);]]>
+ </programlisting>
+ </para>
+ </section>
</section>
- <section xml:id="tools.tokenizer.subword">
- <title>Subword Tokenization</title>
- <para>
- Neural models usually operate on subword units drawn from a fixed vocabulary rather than
- on words. The <code>SubwordTokenizer</code> interface in
- <code>opennlp.tools.tokenize</code> covers this case: <code>encode</code> splits text into
- pieces, and every returned <code>SubwordPiece</code> carries the piece string in the
- model's normalized form, its vocabulary id, and the exact span of the original text it
- came from. Because the segmentation runs after the model's own normalization, a piece is
- generally not a substring of the input; the spans always refer to the caller's original
- text, so annotations computed over the pieces can be mapped back without guesswork. The
- <code>encodeToIds</code> and <code>encodeToPieces</code> methods return just the ids or
- the piece strings when the spans are not needed. Thread safety is implementation specific;
- both implementations described below are immutable and safe for concurrent use by
- multiple threads.
- </para>
- <section xml:id="tools.tokenizer.subword.sentencepiece">
- <title>SentencePiece</title>
- <para>
- <code>SentencePieceTokenizer</code> in the <code>opennlp-subword</code> artifact
- (package <code>opennlp.subword.sentencepiece</code>) runs a trained SentencePiece
- <code>.model</code> file purely in Java, with no native library. The file itself
- carries everything the class needs: the vocabulary with piece scores and types, the
- segmentation algorithm (unigram language model or byte-pair encoding), and the text
- normalizer the model was trained with. Loading is a one-time cost and the resulting
- instance is immutable, so one tokenizer can be shared by any number of threads.
- <programlisting language="java">
- <![CDATA[SentencePieceTokenizer tokenizer = SentencePieceTokenizer.load(Path.of("spiece.model"));
-
- for (SubwordPiece piece : tokenizer.encode("Subword pieces keep their original offsets.")) {
- System.out.println(piece.piece() + " -> id " + piece.id()
- + ", original text [" + piece.start() + ", " + piece.end() + ")");
- }
-
- int[] ids = tokenizer.encodeToIds("Ready for the embedding layer.");]]>
- </programlisting>
- <code>SentencePieceUsageExampleTest</code> asserts the load-and-encode workflow
- shown here. The vocabulary can be inspected through
- <code>vocabularySize</code>, <code>idToPiece</code>, <code>pieceToId</code>, and
- <code>score</code>, and the <code>algorithm</code> method reports whether the
- model uses the unigram or the byte-pair encoding algorithm.
- </para>
- <para>
- The model's own normalizer is also exposed directly:
- <code>SentencePieceTokenizer</code> implements <code>OffsetAwareNormalizer</code>, so
- <code>normalize</code> applies the model's normalization rules to arbitrary text and
- <code>normalizeAligned</code> additionally returns the character alignment described
- in <xref linkend="tools.normalizer"/>. This is useful when other processing must see
- text exactly as the subword model does.
- </para>
- <para>
- Output parity with the reference implementation is pinned by test fixtures whose
- expected pieces, ids, and spans are produced by the reference
- <link xlink:href="https://github.com/google/sentencepiece">sentencepiece</link>
- package rather than by the Java code under test. The fixtures README at
- <code>opennlp-extensions/opennlp-subword/src/test/resources/opennlp/subword/sentencepiece/README.md</code>
- documents how the bundled test models were created and walks through regenerating
- them and validating the implementation, including against real published models.
- </para>
- </section>
- <section xml:id="tools.tokenizer.subword.wordpiece">
- <title>WordPiece</title>
- <para>
- <code>WordpieceEncoder</code> in <code>opennlp.tools.tokenize</code> runs the full
- BERT tokenization pipeline: control character removal, whitespace normalization, CJK
- isolation, optional lower casing with accent stripping, punctuation isolation, and
- greedy longest-match wordpiece segmentation. It is constructed from the model's
- vocabulary, conventionally the lines of a <code>vocab.txt</code> file, where a piece's
- id is its line number.
- <programlisting language="java">
- <![CDATA[List<String> vocabulary = Files.readAllLines(Path.of("vocab.txt"));
-
- WordpieceEncoder encoder = new WordpieceEncoder(vocabulary); // uncased model
- WordpieceEncoder cased = new WordpieceEncoder(vocabulary, false); // cased model
-
- List<SubwordPiece> pieces = encoder.encode("OpenNLP encodes text for BERT models.");
- int[] ids = encoder.encodeToIds("OpenNLP encodes text for BERT models.");]]>
- </programlisting>
- Every encoding is framed by the classification and separator pieces, which carry empty
- spans at the text's boundaries, and words the vocabulary cannot cover become the
- unknown piece. Models with other special tokens or with non-contiguous ids are
- supported through the constructors taking explicit special tokens or a piece-to-id
- map. The older <code>BertTokenizer</code> is deprecated and scheduled for removal:
- its <code>tokenize</code> method returns the same piece sequence that
- <code>encodeToPieces</code> returns. Migrating means passing the vocabulary as a
- <code>List</code> or a piece-to-id map instead of a <code>Set</code>; the output
- is unchanged.
- </para>
- </section>
- </section>
</chapter>