You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@opennlp.apache.org by rz...@apache.org on 2026/09/29 11:42:33 UTC
(opennlp) branch main updated: OPENNLP-1931: Fix JSON vocabulary and config parsing in opennlp-dl (#1277)
This is an automated email from the ASF dual-hosted git repository.
rzo1 pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/opennlp.git
The following commit(s) were added to refs/heads/main by this push:
new e88980c5c OPENNLP-1931: Fix JSON vocabulary and config parsing in opennlp-dl (#1277)
e88980c5c is described below
commit e88980c5c3ee83683758c360fa47e7a6cfa2126b
Author: Kristian Rickert <kr...@apache.org>
AuthorDate: Tue Sep 29 07:42:27 2026 -0400
OPENNLP-1931: Fix JSON vocabulary and config parsing in opennlp-dl (#1277)
Replace regex parsing with JsonScan for JSON vocabularies and categorizer
configuration. Require nonnegative integer token ids and read labels from
the top-level id2label object, decoding escaped keys and labels.
Report malformed files as InvalidFormatException with the file name and
offset. Skip nested values without recursive calls and accept non-finite
numbers only in skipped values.
Handle a leading byte order mark in JSON and plain-text vocabularies and
config.json, fixing OPENNLP-1953. Keep tokenizer.json support separate
under OPENNLP-1988.
Add tests for malformed input, escapes, nesting, token ids and byte order
marks. Document the accepted formats and errors in the manual.
Assisted-by: Claude Code 2.1.281
Reviewed-by: Kristian Rickert <kr...@apache.org>
Signed-off-by: Kristian Rickert <kr...@apache.org>
Co-authored-by: Richard Zowalla <rz...@apache.org>
---
.../main/java/opennlp/tools/util/StringUtil.java | 28 +
.../tools/util/StringUtilByteOrderMarkTest.java | 62 ++
.../src/main/java/opennlp/dl/AbstractDL.java | 128 ++--
.../src/main/java/opennlp/dl/JsonScan.java | 651 +++++++++++++++++++++
.../dl/doccat/DocumentCategorizerConfig.java | 51 +-
.../opennlp/dl/doccat/DocumentCategorizerDL.java | 50 +-
.../java/opennlp/dl/namefinder/NameFinderDL.java | 4 +
.../java/opennlp/dl/vectors/SentenceVectorsDL.java | 4 +
.../src/test/java/opennlp/dl/JsonScanTest.java | 454 ++++++++++++++
.../src/test/java/opennlp/dl/LoadVocabTest.java | 227 ++++++-
.../dl/doccat/DocumentCategorizerConfigTest.java | 175 ++++++
.../dl/doccat/DocumentCategorizerDLTest.java | 33 ++
opennlp-docs/src/docbkx/doccat.xml | 13 +
13 files changed, 1751 insertions(+), 129 deletions(-)
diff --git a/opennlp-api/src/main/java/opennlp/tools/util/StringUtil.java b/opennlp-api/src/main/java/opennlp/tools/util/StringUtil.java
index a91678306..5836b6faf 100644
--- a/opennlp-api/src/main/java/opennlp/tools/util/StringUtil.java
+++ b/opennlp-api/src/main/java/opennlp/tools/util/StringUtil.java
@@ -29,6 +29,8 @@ import opennlp.tools.util.normalizer.UnicodeWhitespace;
public class StringUtil {
+ private static final char BYTE_ORDER_MARK = '\uFEFF';
+
private static final Logger logger = LoggerFactory.getLogger(StringUtil.class);
/**
@@ -227,6 +229,32 @@ public class StringUtil {
return fields.toArray(new String[0]);
}
+ /**
+ * Tests whether {@code input} starts with a byte order mark, U+FEFF, which a UTF-8 or
+ * UTF-16 text file may carry as its first character.
+ *
+ * @param input The text to test. Must not be {@code null}.
+ * @return {@code true} if the first character of {@code input} is U+FEFF.
+ * @throws IllegalArgumentException If {@code input} is {@code null}.
+ */
+ public static boolean startsWithByteOrderMark(CharSequence input) {
+ requireNonNullArg(input, "input");
+ return !input.isEmpty() && input.charAt(0) == BYTE_ORDER_MARK;
+ }
+
+ /**
+ * Removes one leading byte order mark, U+FEFF, from {@code input}. A U+FEFF anywhere else
+ * is left in place, since there it is a zero width no-break space and part of the text.
+ *
+ * @param input The text to strip. Must not be {@code null}.
+ * @return {@code input} without its leading byte order mark, or {@code input} itself if it
+ * has none.
+ * @throws IllegalArgumentException If {@code input} is {@code null}.
+ */
+ public static String stripByteOrderMark(String input) {
+ return startsWithByteOrderMark(input) ? input.substring(1) : input;
+ }
+
/**
* Tests whether {@code input} contains an ASCII capital letter, {@code A} to {@code Z}.
* Capital letters outside ASCII do not count.
diff --git a/opennlp-api/src/test/java/opennlp/tools/util/StringUtilByteOrderMarkTest.java b/opennlp-api/src/test/java/opennlp/tools/util/StringUtilByteOrderMarkTest.java
new file mode 100644
index 000000000..8f34a0a90
--- /dev/null
+++ b/opennlp-api/src/test/java/opennlp/tools/util/StringUtilByteOrderMarkTest.java
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package opennlp.tools.util;
+
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Test;
+
+/**
+ * Tests {@link StringUtil#startsWithByteOrderMark(CharSequence)} and
+ * {@link StringUtil#stripByteOrderMark(String)}.
+ */
+public class StringUtilByteOrderMarkTest {
+
+ private static final String BOM = "\uFEFF";
+
+ @Test
+ void testStripsOneLeadingMark() {
+ Assertions.assertEquals("abc", StringUtil.stripByteOrderMark(BOM + "abc"));
+ Assertions.assertEquals("", StringUtil.stripByteOrderMark(BOM));
+ Assertions.assertEquals(BOM + "abc", StringUtil.stripByteOrderMark(BOM + BOM + "abc"));
+ }
+
+ @Test
+ void testKeepsTextWithoutLeadingMark() {
+ final String text = "a" + BOM + "b";
+ Assertions.assertSame(text, StringUtil.stripByteOrderMark(text));
+ Assertions.assertEquals("", StringUtil.stripByteOrderMark(""));
+ Assertions.assertEquals(" " + BOM, StringUtil.stripByteOrderMark(" " + BOM));
+ }
+
+ @Test
+ void testStartsWithByteOrderMark() {
+ Assertions.assertTrue(StringUtil.startsWithByteOrderMark(BOM));
+ Assertions.assertTrue(StringUtil.startsWithByteOrderMark(new StringBuilder(BOM + "{}")));
+ Assertions.assertFalse(StringUtil.startsWithByteOrderMark(""));
+ Assertions.assertFalse(StringUtil.startsWithByteOrderMark("{}" + BOM));
+ // U+FFFE is the byte-swapped mark, not a mark itself
+ Assertions.assertFalse(StringUtil.startsWithByteOrderMark("\uFFFE{}"));
+ }
+
+ @Test
+ void testRejectsNull() {
+ Assertions.assertThrows(IllegalArgumentException.class,
+ () -> StringUtil.startsWithByteOrderMark(null));
+ Assertions.assertThrows(IllegalArgumentException.class,
+ () -> StringUtil.stripByteOrderMark(null));
+ }
+}
diff --git a/opennlp-core/opennlp-ml/opennlp-dl/src/main/java/opennlp/dl/AbstractDL.java b/opennlp-core/opennlp-ml/opennlp-dl/src/main/java/opennlp/dl/AbstractDL.java
index ecbaaf5b0..919bc5155 100644
--- a/opennlp-core/opennlp-ml/opennlp-dl/src/main/java/opennlp/dl/AbstractDL.java
+++ b/opennlp-core/opennlp-ml/opennlp-dl/src/main/java/opennlp/dl/AbstractDL.java
@@ -29,9 +29,6 @@ import java.util.List;
import java.util.Map;
import java.util.concurrent.atomic.AtomicBoolean;
import java.util.concurrent.atomic.AtomicInteger;
-import java.util.regex.Matcher;
-import java.util.regex.Pattern;
-import java.util.stream.Stream;
import ai.onnxruntime.OrtEnvironment;
import ai.onnxruntime.OrtException;
@@ -41,7 +38,9 @@ import opennlp.tools.tokenize.SubwordPiece;
import opennlp.tools.tokenize.SubwordTokenizer;
import opennlp.tools.tokenize.WordpieceEncoder;
import opennlp.tools.tokenize.WordpieceTokenizer;
+import opennlp.tools.util.InvalidFormatException;
import opennlp.tools.util.Span;
+import opennlp.tools.util.StringUtil;
import opennlp.tools.util.normalizer.AlignedText;
import opennlp.tools.util.normalizer.Alignment;
import opennlp.tools.util.normalizer.CharClass;
@@ -76,9 +75,6 @@ public abstract class AbstractDL implements AutoCloseable {
protected record TextChunk(String text, int start, int end) {
}
- private static final Pattern JSON_ENTRY_PATTERN =
- Pattern.compile("\"((?:[^\"\\\\]|\\\\.)*)\"\\s*:\\s*(\\d+)");
-
/**
* Initializes the shared, immutable inference state: the ONNX environment and session,
* the loaded vocabulary and the configured tokenizer. These fields are {@code final}
@@ -94,6 +90,7 @@ public abstract class AbstractDL implements AutoCloseable {
*
* @throws OrtException Thrown if the {@code model} cannot be loaded.
* @throws IOException Thrown if the {@code model} or {@code vocabulary} cannot be read.
+ * @throws InvalidFormatException Thrown if a JSON {@code vocabulary} is malformed.
*/
protected AbstractDL(final File model, final File vocabulary,
final OrtSession.SessionOptions sessionOptions, final boolean lowerCase)
@@ -162,16 +159,19 @@ public abstract class AbstractDL implements AutoCloseable {
}
/**
- * Loads a vocabulary {@link File} from disk.
- * Supports both plain text files (one token per
- * line) and simple JSON vocabulary files mapping tokens to integer
- * IDs. JSON support is intentionally limited to the HuggingFace vocabulary
- * shape; it is not a general-purpose JSON parser.
+ * Loads a vocabulary {@link File} from disk. A file with an opening brace as its first
+ * non-whitespace character is read as JSON: one object that maps each token to a non-negative
+ * integer ID, as in {@code vocab.json}. Any other file is read as plain text with one token
+ * per line, the line number being the ID, as in {@code vocab.txt}. A byte order mark at the
+ * start of the file is not content in either format. The JSON layout is that of
+ * {@link #loadJsonVocab(String)}.
*
* @param vocabFile The vocabulary file.
* @return A map of vocabulary words to IDs.
- * @throws IOException Thrown if the vocabulary
- * file cannot be opened or read.
+ * @throws IOException Thrown if the vocabulary file cannot be opened or read.
+ * @throws InvalidFormatException Thrown if a JSON vocabulary is malformed, has a layout that
+ * is not supported, or contains a value that is not a non-negative integer. The message
+ * names the file and the offset or the token.
*/
public Map<String, Integer> loadVocab(
final File vocabFile) throws IOException {
@@ -179,18 +179,30 @@ public abstract class AbstractDL implements AutoCloseable {
return loadVocabFile(vocabFile);
}
+ /**
+ * Loads a vocabulary file as {@link #loadVocab(File)} does.
+ *
+ * @param vocabFile The vocabulary file.
+ * @return A map of vocabulary words to IDs.
+ * @throws IOException Thrown if the vocabulary file cannot be opened or read.
+ * @throws InvalidFormatException Thrown if a JSON vocabulary is malformed, has a layout that
+ * is not supported, or contains a value that is not a non-negative integer.
+ */
static Map<String, Integer> loadVocabFile(
final File vocabFile) throws IOException {
- final Path vocabPath =
- Path.of(vocabFile.getPath());
- final String content = Files.readString(
- vocabPath, StandardCharsets.UTF_8);
+ final String read = Files.readString(Path.of(vocabFile.getPath()), StandardCharsets.UTF_8);
+ final String content = StringUtil.stripByteOrderMark(read);
final String trimmed = content.trim();
// Detect JSON format by leading brace
if (trimmed.startsWith("{")) {
- return loadJsonVocab(trimmed);
+ try {
+ return loadJsonVocab(trimmed);
+ } catch (IllegalArgumentException e) {
+ throw new InvalidFormatException(
+ "Vocabulary file " + vocabFile.getName() + ": " + e.getMessage(), e);
+ }
}
final Map<String, Integer> vocab =
@@ -198,12 +210,9 @@ public abstract class AbstractDL implements AutoCloseable {
final AtomicInteger counter =
new AtomicInteger(0);
- try (Stream<String> lines = Files.lines(
- vocabPath, StandardCharsets.UTF_8)) {
- lines.forEach(line ->
- vocab.put(line, counter.getAndIncrement())
- );
- }
+ content.lines().forEach(line ->
+ vocab.put(line, counter.getAndIncrement())
+ );
return vocab;
}
@@ -521,58 +530,31 @@ public abstract class AbstractDL implements AutoCloseable {
return List.copyOf(ranges);
}
- private static Map<String, Integer> loadJsonVocab(final String json) {
-
+ /**
+ * Reads a JSON vocabulary as one object mapping tokens to non-negative integer IDs, as in
+ * {@code vocab.json}. Keys are decoded from their escapes, and a later entry for the same
+ * token overwrites an earlier one.
+ *
+ * @param json The JSON text of the vocabulary.
+ * @return A map of vocabulary tokens to IDs.
+ * @throws IllegalArgumentException Thrown if the text is not a single well-formed JSON object,
+ * or if a value of the vocabulary object is not a
+ * non-negative integer that fits into an {@code int}. The message names the offset or the
+ * token.
+ */
+ static Map<String, Integer> loadJsonVocab(final String json) {
+ final List<JsonScan.Member> document = JsonScan.document(json);
final Map<String, Integer> vocab = new HashMap<>();
- final Matcher matcher = JSON_ENTRY_PATTERN.matcher(json);
-
- while (matcher.find()) {
- final String token = matcher.group(1)
- .transform(AbstractDL::unescapeJsonString);
- final int id = Integer.parseInt(matcher.group(2));
- vocab.put(token, id);
- }
-
- return vocab;
- }
-
- private static String unescapeJsonString(final String value) {
- final StringBuilder result = new StringBuilder(value.length());
- for (int i = 0; i < value.length(); i++) {
- final char ch = value.charAt(i);
- if (ch != '\\') {
- result.append(ch);
- continue;
- }
- if (++i == value.length()) {
- throw new IllegalArgumentException("Invalid JSON string escape.");
- }
- final char escaped = value.charAt(i);
- switch (escaped) {
- case '"' -> result.append('"');
- case '\\' -> result.append('\\');
- case '/' -> result.append('/');
- case 'b' -> result.append('\b');
- case 'f' -> result.append('\f');
- case 'n' -> result.append('\n');
- case 'r' -> result.append('\r');
- case 't' -> result.append('\t');
- case 'u' -> {
- if (i + 4 >= value.length()) {
- throw new IllegalArgumentException("Invalid JSON unicode escape.");
- }
- final String hex = value.substring(i + 1, i + 5);
- try {
- result.append((char) Integer.parseInt(hex, 16));
- } catch (NumberFormatException e) {
- throw new IllegalArgumentException("Invalid JSON unicode escape.", e);
- }
- i += 4;
- }
- default -> throw new IllegalArgumentException("Invalid JSON string escape.");
+ for (JsonScan.Member member : document) {
+ try {
+ vocab.put(member.key(), JsonScan.nonNegativeIntValue(json, member));
+ } catch (IllegalArgumentException e) {
+ throw new IllegalArgumentException(
+ "Expected one object mapping tokens to integer ids, as in vocab.json: "
+ + e.getMessage(), e);
}
}
- return result.toString();
+ return vocab;
}
/**
diff --git a/opennlp-core/opennlp-ml/opennlp-dl/src/main/java/opennlp/dl/JsonScan.java b/opennlp-core/opennlp-ml/opennlp-dl/src/main/java/opennlp/dl/JsonScan.java
new file mode 100644
index 000000000..4a0246a86
--- /dev/null
+++ b/opennlp-core/opennlp-ml/opennlp-dl/src/main/java/opennlp/dl/JsonScan.java
@@ -0,0 +1,651 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package opennlp.dl;
+
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Deque;
+import java.util.HashMap;
+import java.util.HexFormat;
+import java.util.List;
+import java.util.Map;
+
+import opennlp.tools.commons.Internal;
+import opennlp.tools.util.StringUtil;
+
+/**
+ * Reads the JSON files of the deep-learning components, vocabularies and model configurations.
+ * Structure, whitespace, numbers, and string escapes follow
+ * <a href="https://www.rfc-editor.org/rfc/rfc8259">RFC 8259</a>, with three additions:
+ * <ul>
+ * <li>a byte order mark as the first character is skipped, as
+ * <a href="https://www.rfc-editor.org/rfc/rfc8259#section-8.1">section 8.1</a> allows;</li>
+ * <li>the control characters {@code U+0000} to {@code U+001F} are accepted inside a string;</li>
+ * <li>the values {@code NaN}, {@code Infinity}, and {@code -Infinity} are accepted where a
+ * value is skipped, never where one is read.</li>
+ * </ul>
+ * Malformed text is reported as an {@link IllegalArgumentException} whose message names the
+ * offset at which reading stopped.
+ */
+@Internal(since = "3.0.0")
+public final class JsonScan {
+
+ private static final String TRUE = "true";
+ private static final String FALSE = "false";
+ private static final String NULL = "null";
+ private static final String NAN = "NaN";
+ private static final String INFINITY = "Infinity";
+ private static final String NEGATIVE_INFINITY = "-Infinity";
+ private static final String[] LITERALS = {TRUE, FALSE, NULL, NAN, INFINITY, NEGATIVE_INFINITY};
+
+ private static final char OBJECT_OPEN = '{';
+ private static final char OBJECT_CLOSE = '}';
+ private static final char ARRAY_OPEN = '[';
+ private static final char ARRAY_CLOSE = ']';
+ private static final char QUOTE = '"';
+ private static final char BACKSLASH = '\\';
+ private static final char UNICODE_ESCAPE = 'u';
+
+ /** The escape characters other than {@code u} that may follow a backslash. */
+ private static final String SIMPLE_ESCAPES = "\"\\/bfnrt";
+ /** The length of a two-character escape such as {@code \n}. */
+ private static final int ESCAPE_LENGTH = 2;
+ /** The number of hexadecimal digits of a {@code \\u} escape. */
+ private static final int UNICODE_ESCAPE_DIGITS = 4;
+ /** The length of a {@code \\u} escape including the backslash and the {@code u}. */
+ private static final int UNICODE_ESCAPE_LENGTH = ESCAPE_LENGTH + UNICODE_ESCAPE_DIGITS;
+ /** The number of characters quoted in the message for malformed text. */
+ private static final int MESSAGE_CONTEXT_LENGTH = 20;
+
+ private static final String TEXT_MUST_NOT_BE_NULL = "text must not be null";
+ private static final String KEY_MUST_NOT_BE_NULL = "key must not be null";
+
+ private JsonScan() {
+ }
+
+ /**
+ * A member of a JSON object.
+ *
+ * @param key The unescaped key. Must not be {@code null}.
+ * @param valueStart The offset of the first character of the value, not negative.
+ * @param valueEnd The offset after the last character of the value, not before
+ * {@code valueStart}.
+ * @throws IllegalArgumentException Thrown if {@code key} is {@code null} or the offsets are
+ * not such a range.
+ */
+ record Member(String key, int valueStart, int valueEnd) {
+
+ Member {
+ if (key == null) {
+ throw new IllegalArgumentException(KEY_MUST_NOT_BE_NULL);
+ }
+ if (valueStart < 0 || valueEnd < valueStart) {
+ throw new IllegalArgumentException(
+ "value range must be non-negative and ordered: " + valueStart + ".." + valueEnd);
+ }
+ }
+ }
+
+ /**
+ * Reads a top-level member whose value is an object of strings, such as the {@code id2label}
+ * map of a model configuration. Keys and values are decoded from their escapes. A later entry
+ * for the same key replaces an earlier one, and a later member with the same name replaces
+ * an earlier one. A member with that name inside a nested value is not considered.
+ *
+ * @param text The JSON text: one object, or blank text (JSON whitespace only, after an
+ * optional byte order mark), which is read as an empty object. Must not be {@code null}.
+ * @param key The unescaped name of the member. Must not be {@code null}.
+ * @return The entries of that object, or an empty map if there is no such top-level member.
+ * @throws IllegalArgumentException Thrown if {@code text} or {@code key} is {@code null},
+ * if {@code text} is neither blank nor a single well-formed JSON object, or if the member
+ * is not an object whose values are all strings. The message names the offset or the key.
+ */
+ public static Map<String, String> stringObject(String text, String key) {
+ requireText(text);
+ if (key == null) {
+ throw new IllegalArgumentException(KEY_MUST_NOT_BE_NULL);
+ }
+ final Map<String, String> entries = new HashMap<>();
+ if (isBlank(text)) {
+ return entries;
+ }
+ final Member object = member(document(text), key);
+ if (object == null) {
+ return entries;
+ }
+ if (!isObject(text, object)) {
+ throw new IllegalArgumentException("\"" + key + "\" must be an object: "
+ + text.substring(object.valueStart(), object.valueEnd()));
+ }
+ for (Member entry : members(text, object.valueStart())) {
+ entries.put(entry.key(), stringValue(text, entry));
+ }
+ return entries;
+ }
+
+ /**
+ * Reads the members of the single object that is the document.
+ *
+ * @param text The JSON text. Must not be {@code null}.
+ * @return The members in document order, an empty list for an empty object.
+ * @throws IllegalArgumentException Thrown if {@code text} is {@code null}, does not
+ * consist of one object, or is malformed at any position.
+ */
+ static List<Member> document(String text) {
+ requireText(text);
+ final int start = skipWhitespace(text, afterByteOrderMark(text));
+ expect(text, start, OBJECT_OPEN);
+ final List<Member> members = new ArrayList<>();
+ final int end = endOfContainer(text, start, members);
+ final int trailing = skipWhitespace(text, end);
+ if (trailing < text.length()) {
+ throw malformed(text, trailing, "content after the object");
+ }
+ return Collections.unmodifiableList(members);
+ }
+
+ /**
+ * Reads the members of the object that starts at an offset.
+ *
+ * @param text The JSON text. Must not be {@code null}.
+ * @param brace The offset of the opening brace.
+ * @return The members in document order, an empty list for an empty object.
+ * @throws IllegalArgumentException Thrown if {@code text} is {@code null}, {@code brace} is
+ * not the offset of an opening brace inside the text, or the object is malformed.
+ */
+ static List<Member> members(String text, int brace) {
+ requireText(text);
+ if (brace < 0 || brace >= text.length()) {
+ throw new IllegalArgumentException(
+ "brace must be an offset inside the text, not " + brace);
+ }
+ expect(text, brace, OBJECT_OPEN);
+ final List<Member> members = new ArrayList<>();
+ endOfContainer(text, brace, members);
+ return Collections.unmodifiableList(members);
+ }
+
+ /**
+ * Finds the member with a key in a list of members. A later member with the same key
+ * counts, as with a map that is filled in document order.
+ *
+ * @param members The members. Must not be {@code null}.
+ * @param key The unescaped key. Must not be {@code null}.
+ * @return The last member with that key, or {@code null} if there is none.
+ * @throws IllegalArgumentException Thrown if {@code members} or {@code key} is {@code null}.
+ */
+ static Member member(List<Member> members, String key) {
+ if (members == null) {
+ throw new IllegalArgumentException("members must not be null");
+ }
+ if (key == null) {
+ throw new IllegalArgumentException(KEY_MUST_NOT_BE_NULL);
+ }
+ Member found = null;
+ for (Member m : members) {
+ if (m.key().equals(key)) {
+ found = m;
+ }
+ }
+ return found;
+ }
+
+ /**
+ * Tests whether the value of a member is an object.
+ *
+ * @param text The JSON text. Must not be {@code null}.
+ * @param member The member, inside the text. Must not be {@code null}.
+ * @return {@code true} if the value starts with an opening brace.
+ * @throws IllegalArgumentException Thrown if an argument is {@code null} or the member lies
+ * outside the text.
+ */
+ static boolean isObject(String text, Member member) {
+ requireMember(text, member);
+ return text.charAt(member.valueStart()) == OBJECT_OPEN;
+ }
+
+ /**
+ * Reads the value of a member as a string.
+ *
+ * @param text The JSON text. Must not be {@code null}.
+ * @param member The member, inside the text. Must not be {@code null}.
+ * @return The unescaped string.
+ * @throws IllegalArgumentException Thrown if an argument is {@code null}, the member lies
+ * outside the text, or the value is not a string literal.
+ */
+ static String stringValue(String text, Member member) {
+ requireMember(text, member);
+ if (text.charAt(member.valueStart()) != QUOTE) {
+ throw new IllegalArgumentException("Value of \"" + member.key() + "\" must be a string: "
+ + text.substring(member.valueStart(), member.valueEnd()));
+ }
+ return unescape(text, member.valueStart() + 1, member.valueEnd() - 1);
+ }
+
+ /**
+ * Reads the value of a member as a non-negative integer: a run of ASCII digits with no sign,
+ * fraction, or exponent that fits into an {@code int}.
+ *
+ * @param text The JSON text. Must not be {@code null}.
+ * @param member The member, inside the text. Must not be {@code null}.
+ * @return The integer.
+ * @throws IllegalArgumentException Thrown if an argument is {@code null}, the member lies
+ * outside the text, or the value is not such an integer.
+ */
+ static int nonNegativeIntValue(String text, Member member) {
+ requireMember(text, member);
+ final int start = member.valueStart();
+ final int end = member.valueEnd();
+ if (StringUtil.endOfAsciiDigits(text, start) != end) {
+ throw new IllegalArgumentException("Value of \"" + member.key()
+ + "\" must be a non-negative integer: " + text.substring(start, end));
+ }
+ try {
+ return Integer.parseInt(text, start, end, 10);
+ } catch (NumberFormatException e) {
+ throw new IllegalArgumentException("Value of \"" + member.key()
+ + "\" does not fit into an int: " + text.substring(start, end), e);
+ }
+ }
+
+ /**
+ * Tests whether the text holds nothing but JSON whitespace after an optional byte order mark.
+ *
+ * @param text The JSON text.
+ * @return {@code true} if there is no content.
+ */
+ private static boolean isBlank(String text) {
+ return skipWhitespace(text, afterByteOrderMark(text)) == text.length();
+ }
+
+ /**
+ * Skips a byte order mark at the start of the text.
+ *
+ * @param text The JSON text.
+ * @return {@code 1} if the text starts with a byte order mark, {@code 0} otherwise.
+ */
+ private static int afterByteOrderMark(String text) {
+ return StringUtil.startsWithByteOrderMark(text) ? 1 : 0;
+ }
+
+ /**
+ * Finds the closing quote of a string literal, checking each escape on the way: a backslash
+ * must be followed by one of the escape characters of RFC 8259, and {@code \\u} by four
+ * hexadecimal digits.
+ *
+ * @param text The JSON text.
+ * @param openQuote The offset of the opening quote.
+ * @return The offset of the closing quote.
+ * @throws IllegalArgumentException Thrown if the literal is not closed or holds a bad escape.
+ */
+ private static int closingQuote(String text, int openQuote) {
+ int i = openQuote + 1;
+ while (i < text.length()) {
+ final char c = text.charAt(i);
+ if (c == QUOTE) {
+ return i;
+ }
+ if (c != BACKSLASH) {
+ i++;
+ } else if (i + 1 >= text.length()) {
+ break;
+ } else {
+ i = afterEscape(text, i);
+ }
+ }
+ throw malformed(text, openQuote, "unterminated string");
+ }
+
+ /**
+ * Checks the escape that starts at a backslash.
+ *
+ * @param text The JSON text.
+ * @param backslash The offset of the backslash, which is not the last character.
+ * @return The offset after the escape.
+ * @throws IllegalArgumentException Thrown if the escape is not one of RFC 8259.
+ */
+ private static int afterEscape(String text, int backslash) {
+ final char escaped = text.charAt(backslash + 1);
+ if (escaped == UNICODE_ESCAPE) {
+ final int end = backslash + UNICODE_ESCAPE_LENGTH;
+ if (end > text.length() || !isHexDigits(text, backslash + ESCAPE_LENGTH, end)) {
+ throw malformed(text, backslash, "\\u must be followed by four hexadecimal digits");
+ }
+ return end;
+ }
+ if (SIMPLE_ESCAPES.indexOf(escaped) < 0) {
+ throw malformed(text, backslash, "unknown escape \\" + escaped);
+ }
+ return backslash + ESCAPE_LENGTH;
+ }
+
+ /**
+ * Replaces the escapes of a string literal with the characters they stand for.
+ *
+ * @param text The JSON text.
+ * @param start The offset after the opening quote.
+ * @param end The offset of the closing quote.
+ * @return The decoded string.
+ * @throws IllegalArgumentException Thrown if an escape is not one of RFC 8259.
+ */
+ private static String unescape(String text, int start, int end) {
+ final int firstBackslash = text.indexOf(BACKSLASH, start, end);
+ if (firstBackslash < 0) {
+ return text.substring(start, end);
+ }
+ final StringBuilder result = new StringBuilder(end - start);
+ result.append(text, start, firstBackslash);
+ int i = firstBackslash;
+ while (i < end) {
+ final char c = text.charAt(i);
+ if (c != BACKSLASH) {
+ result.append(c);
+ i++;
+ continue;
+ }
+ if (i + 1 >= end) {
+ throw malformed(text, i, "a backslash must be followed by an escape character");
+ }
+ final char escaped = text.charAt(i + 1);
+ switch (escaped) {
+ case QUOTE -> result.append(QUOTE);
+ case BACKSLASH -> result.append(BACKSLASH);
+ case '/' -> result.append('/');
+ case 'b' -> result.append('\b');
+ case 'f' -> result.append('\f');
+ case 'n' -> result.append('\n');
+ case 'r' -> result.append('\r');
+ case 't' -> result.append('\t');
+ case UNICODE_ESCAPE -> {
+ final int escapeEnd = i + UNICODE_ESCAPE_LENGTH;
+ if (escapeEnd > end || !isHexDigits(text, i + ESCAPE_LENGTH, escapeEnd)) {
+ throw malformed(text, i, "\\u must be followed by four hexadecimal digits");
+ }
+ result.append((char) HexFormat.fromHexDigits(text, i + ESCAPE_LENGTH, escapeEnd));
+ i += UNICODE_ESCAPE_DIGITS;
+ }
+ default -> throw malformed(text, i, "unknown escape \\" + escaped);
+ }
+ i += ESCAPE_LENGTH;
+ }
+ return result.toString();
+ }
+
+ /**
+ * Skips JSON whitespace: space, tab, line feed, and carriage return. Other Unicode spaces
+ * are content, not separators.
+ *
+ * @param text The JSON text.
+ * @param from The offset to start at.
+ * @return The offset of the first non-whitespace character at or after {@code from}, or the
+ * length of the text if only whitespace remains.
+ */
+ private static int skipWhitespace(String text, int from) {
+ int i = from;
+ while (i < text.length() && isJsonWhitespace(text.charAt(i))) {
+ i++;
+ }
+ return i;
+ }
+
+ /**
+ * Scans an object or an array, and every value nested in it, without recursion. When a sink
+ * is given, the container is an object and its direct members are added to the sink.
+ *
+ * @param text The JSON text.
+ * @param open The offset of the opening brace or bracket.
+ * @param sink The list to add the members of the object to, or {@code null} to only find
+ * the end.
+ * @return The offset after the closing brace or bracket.
+ * @throws IllegalArgumentException Thrown if the container or a value in it is malformed.
+ */
+ private static int endOfContainer(String text, int open, List<Member> sink) {
+ final Deque<Boolean> objects = new ArrayDeque<>();
+ String key = null;
+ int valueStart = 0;
+ int i = open;
+ boolean enter = true;
+ boolean elementExpected = false;
+ while (true) {
+ if (enter) {
+ enter = false;
+ objects.push(text.charAt(i) == OBJECT_OPEN);
+ i = skipWhitespace(text, i + 1);
+ elementExpected = !isCloser(text, i, objects.peek());
+ if (elementExpected) {
+ continue;
+ }
+ i++;
+ objects.pop();
+ if (objects.isEmpty()) {
+ return i;
+ }
+ } else if (elementExpected) {
+ if (objects.peek()) {
+ expect(text, i, QUOTE);
+ final int keyEnd = closingQuote(text, i);
+ if (sink != null && objects.size() == 1) {
+ key = unescape(text, i + 1, keyEnd);
+ }
+ final int colon = skipWhitespace(text, keyEnd + 1);
+ expect(text, colon, ':');
+ i = skipWhitespace(text, colon + 1);
+ }
+ if (sink != null && objects.size() == 1) {
+ valueStart = i;
+ }
+ if (i < text.length() && (text.charAt(i) == OBJECT_OPEN || text.charAt(i) == ARRAY_OPEN)) {
+ enter = true;
+ continue;
+ }
+ i = endOfScalar(text, i);
+ elementExpected = false;
+ } else {
+ if (sink != null && objects.size() == 1) {
+ sink.add(new Member(key, valueStart, i));
+ }
+ i = skipWhitespace(text, i);
+ if (isCloser(text, i, objects.peek())) {
+ i++;
+ objects.pop();
+ if (objects.isEmpty()) {
+ return i;
+ }
+ continue;
+ }
+ expect(text, i, ',');
+ i = skipWhitespace(text, i + 1);
+ elementExpected = true;
+ }
+ }
+ }
+
+ /**
+ * Tests whether the character at an offset closes the innermost container.
+ *
+ * @param text The JSON text.
+ * @param at The offset.
+ * @param object {@code true} if the innermost container is an object, {@code false} for an
+ * array.
+ * @return {@code true} if the offset carries the matching closing brace or bracket.
+ */
+ private static boolean isCloser(String text, int at, boolean object) {
+ return at < text.length() && text.charAt(at) == (object ? OBJECT_CLOSE : ARRAY_CLOSE);
+ }
+
+ /**
+ * Finds the end of the scalar value that starts at an offset: a string, a number, or one of
+ * the literals {@code true}, {@code false}, {@code null}, {@code NaN}, {@code Infinity}, and
+ * {@code -Infinity}.
+ *
+ * @param text The JSON text.
+ * @param from The offset of the first character of the value.
+ * @return The offset after the value.
+ * @throws IllegalArgumentException Thrown if no well-formed scalar starts at {@code from}.
+ */
+ private static int endOfScalar(String text, int from) {
+ if (from >= text.length()) {
+ throw malformed(text, from, "expected a value");
+ }
+ final char c = text.charAt(from);
+ if (c == QUOTE) {
+ return closingQuote(text, from) + 1;
+ }
+ for (String literal : LITERALS) {
+ if (text.startsWith(literal, from)) {
+ return from + literal.length();
+ }
+ }
+ if (c == '-' || StringUtil.isAsciiDigit(c)) {
+ return endOfNumber(text, from);
+ }
+ throw malformed(text, from, "expected a value");
+ }
+
+ /**
+ * Scans a number: an optional minus, an integer part without leading zeros, an optional
+ * fraction, and an optional exponent.
+ *
+ * @param text The JSON text.
+ * @param from The offset of the first character of the number.
+ * @return The offset after the number.
+ * @throws IllegalArgumentException Thrown if the number is malformed.
+ */
+ private static int endOfNumber(String text, int from) {
+ int i = from;
+ if (text.charAt(i) == '-') {
+ i++;
+ }
+ if (i < text.length() && text.charAt(i) == '0') {
+ i++;
+ } else {
+ final int digitsEnd = StringUtil.endOfAsciiDigits(text, i);
+ if (digitsEnd == i) {
+ throw malformed(text, i, "expected a digit");
+ }
+ i = digitsEnd;
+ }
+ if (i < text.length() && text.charAt(i) == '.') {
+ final int fractionEnd = StringUtil.endOfAsciiDigits(text, i + 1);
+ if (fractionEnd == i + 1) {
+ throw malformed(text, i + 1, "expected a digit");
+ }
+ i = fractionEnd;
+ }
+ if (i < text.length() && (text.charAt(i) == 'e' || text.charAt(i) == 'E')) {
+ i++;
+ if (i < text.length() && (text.charAt(i) == '+' || text.charAt(i) == '-')) {
+ i++;
+ }
+ final int exponentEnd = StringUtil.endOfAsciiDigits(text, i);
+ if (exponentEnd == i) {
+ throw malformed(text, i, "expected a digit");
+ }
+ i = exponentEnd;
+ }
+ return i;
+ }
+
+ /**
+ * Checks for an expected character.
+ *
+ * @param text The JSON text.
+ * @param at The offset.
+ * @param expected The character that must be there.
+ * @throws IllegalArgumentException Thrown if the offset is beyond the end or carries another
+ * character.
+ */
+ private static void expect(String text, int at, char expected) {
+ if (at >= text.length() || text.charAt(at) != expected) {
+ throw malformed(text, at, "expected '" + expected + "'");
+ }
+ }
+
+ /**
+ * Builds the exception for malformed text.
+ *
+ * @param text The JSON text.
+ * @param at The offset reading stopped at.
+ * @param reason What was expected there.
+ * @return The exception to throw.
+ */
+ private static IllegalArgumentException malformed(String text, int at, String reason) {
+ final String found = at >= text.length() ? "end of text"
+ : "'" + text.substring(at, Math.min(text.length(), at + MESSAGE_CONTEXT_LENGTH)) + "'";
+ return new IllegalArgumentException(
+ "Malformed JSON at offset " + at + ", " + reason + ", found " + found);
+ }
+
+ /**
+ * Tests for JSON whitespace.
+ *
+ * @param c The character.
+ * @return {@code true} for space, tab, line feed, or carriage return.
+ */
+ private static boolean isJsonWhitespace(char c) {
+ return c == ' ' || c == '\t' || c == '\n' || c == '\r';
+ }
+
+ /**
+ * Tests whether a range consists of hexadecimal digits only.
+ *
+ * @param text The JSON text.
+ * @param from The offset the range starts at (inclusive).
+ * @param to The offset the range ends at (exclusive).
+ * @return {@code true} if each character is a hexadecimal digit of either case.
+ */
+ private static boolean isHexDigits(String text, int from, int to) {
+ for (int i = from; i < to; i++) {
+ if (!HexFormat.isHexDigit(text.charAt(i))) {
+ return false;
+ }
+ }
+ return true;
+ }
+
+ /**
+ * Checks the text argument.
+ *
+ * @param text The JSON text.
+ * @throws IllegalArgumentException Thrown if it is {@code null}.
+ */
+ private static void requireText(String text) {
+ if (text == null) {
+ throw new IllegalArgumentException(TEXT_MUST_NOT_BE_NULL);
+ }
+ }
+
+ /**
+ * Checks a member against the text it must lie in.
+ *
+ * @param text The JSON text.
+ * @param member The member.
+ * @throws IllegalArgumentException Thrown if either is {@code null} or the member's value
+ * range reaches beyond the text.
+ */
+ private static void requireMember(String text, Member member) {
+ requireText(text);
+ if (member == null) {
+ throw new IllegalArgumentException("member must not be null");
+ }
+ if (member.valueEnd() > text.length() || member.valueStart() >= text.length()) {
+ throw new IllegalArgumentException("member must lie inside the text: " + member);
+ }
+ }
+}
diff --git a/opennlp-core/opennlp-ml/opennlp-dl/src/main/java/opennlp/dl/doccat/DocumentCategorizerConfig.java b/opennlp-core/opennlp-ml/opennlp-dl/src/main/java/opennlp/dl/doccat/DocumentCategorizerConfig.java
index 00805f16e..6425c91f9 100644
--- a/opennlp-core/opennlp-ml/opennlp-dl/src/main/java/opennlp/dl/doccat/DocumentCategorizerConfig.java
+++ b/opennlp-core/opennlp-ml/opennlp-dl/src/main/java/opennlp/dl/doccat/DocumentCategorizerConfig.java
@@ -18,41 +18,44 @@
package opennlp.dl.doccat;
import java.util.Collections;
-import java.util.HashMap;
import java.util.Map;
-import java.util.Objects;
-import java.util.regex.Matcher;
-import java.util.regex.Pattern;
+import opennlp.dl.JsonScan;
+
+/**
+ * The part of a model configuration that a {@link DocumentCategorizerDL} uses: the
+ * {@code id2label} map of output index to label.
+ *
+ * @param id2label The labels by output index.
+ */
public record DocumentCategorizerConfig(Map<String, String> id2label) {
- private static final Pattern ID_TO_LABEL_PATTERN =
- Pattern.compile("\"id2label\"\\s*:\\s*\\{(.*?)\\}", Pattern.DOTALL);
- private static final Pattern ENTRY_PATTERN =
- Pattern.compile("\"([^\"]+)\"\\s*:\\s*\"(.*?)\"");
+ private static final String ID_TO_LABEL_KEY = "id2label";
@Override
public Map<String, String> id2label() {
return Collections.unmodifiableMap(id2label);
}
+ /**
+ * Reads the top-level {@code id2label} member of a model configuration, the object that
+ * maps each output index to its label in the {@code config.json} files that accompany
+ * classification models. Keys and labels are decoded from their escapes, and a later entry
+ * for the same key overwrites an earlier one. An {@code id2label} member nested in another
+ * member is not the configuration's map.
+ *
+ * @param json The JSON text of the configuration. Blank text, with or without a leading
+ * byte order mark, is a configuration without labels. Must not be {@code null}.
+ * @return The configuration, with an empty map if the configuration has no top-level
+ * {@code id2label} member.
+ * @throws IllegalArgumentException Thrown if {@code json} is {@code null}, if the text is
+ * neither blank nor a single well-formed JSON object, if {@code id2label} is not an
+ * object, or if a label is not a string. The message names the offset or the key.
+ */
public static DocumentCategorizerConfig fromJson(String json) {
- Objects.requireNonNull(json, "json must not be null");
-
- final Map<String, String> id2label = new HashMap<>();
- final Matcher matcher = ID_TO_LABEL_PATTERN.matcher(json);
-
- if (matcher.find()) {
- final String id2labelContent = matcher.group(1);
- final Matcher entryMatcher = ENTRY_PATTERN.matcher(id2labelContent);
-
- while (entryMatcher.find()) {
- final String key = entryMatcher.group(1);
- final String value = entryMatcher.group(2);
- id2label.put(key, value);
- }
+ if (json == null) {
+ throw new IllegalArgumentException("json must not be null");
}
-
- return new DocumentCategorizerConfig(id2label);
+ return new DocumentCategorizerConfig(JsonScan.stringObject(json, ID_TO_LABEL_KEY));
}
}
diff --git a/opennlp-core/opennlp-ml/opennlp-dl/src/main/java/opennlp/dl/doccat/DocumentCategorizerDL.java b/opennlp-core/opennlp-ml/opennlp-dl/src/main/java/opennlp/dl/doccat/DocumentCategorizerDL.java
index 8b3020d45..208cb573e 100644
--- a/opennlp-core/opennlp-ml/opennlp-dl/src/main/java/opennlp/dl/doccat/DocumentCategorizerDL.java
+++ b/opennlp-core/opennlp-ml/opennlp-dl/src/main/java/opennlp/dl/doccat/DocumentCategorizerDL.java
@@ -43,6 +43,7 @@ import opennlp.dl.Tokens;
import opennlp.dl.doccat.scoring.ClassificationScoringStrategy;
import opennlp.tools.commons.ThreadSafe;
import opennlp.tools.doccat.DocumentCategorizer;
+import opennlp.tools.util.InvalidFormatException;
/**
@@ -117,6 +118,7 @@ public class DocumentCategorizerDL extends AbstractDL implements DocumentCategor
* @param inferenceOptions {@link InferenceOptions} to control the inference.
* @throws OrtException Thrown if the {@code model} cannot be loaded.
* @throws IOException Thrown if errors occurred loading the {@code model} or {@code vocabulary}.
+ * @throws InvalidFormatException Thrown if a JSON {@code vocabulary} is malformed.
*/
public DocumentCategorizerDL(File model, File vocabulary, Map<Integer, String> categories,
ClassificationScoringStrategy classificationScoringStrategy,
@@ -151,7 +153,11 @@ public class DocumentCategorizerDL extends AbstractDL implements DocumentCategor
* individual document part.
* @param inferenceOptions {@link InferenceOptions} to control the inference.
* @throws OrtException Thrown if the {@code model} cannot be loaded.
- * @throws IOException Thrown if errors occurred loading the {@code model} or {@code vocabulary}.
+ * @throws IOException Thrown if errors occurred loading the {@code model}, the
+ * {@code vocabulary}, or the {@code config}.
+ * @throws InvalidFormatException Thrown if a JSON {@code vocabulary} is malformed, or if
+ * {@code config} is not well-formed JSON, its {@code id2label} member does not map keys
+ * to strings, or a key is not an integer.
*/
public DocumentCategorizerDL(File model, File vocabulary, File config,
ClassificationScoringStrategy classificationScoringStrategy,
@@ -163,7 +169,7 @@ public class DocumentCategorizerDL extends AbstractDL implements DocumentCategor
inferenceOptions, config, classificationScoringStrategy)),
resolveLowerCase(inferenceOptions, LOWER_CASE_DEFAULT));
- this.categories = Map.copyOf(readCategoriesFromFile(config));
+ this.categories = Map.copyOf(readCategories(config));
this.classificationScoringStrategy = classificationScoringStrategy;
this.includeAttentionMask = inferenceOptions.isIncludeAttentionMask();
this.includeTokenTypeIds = inferenceOptions.isIncludeTokenTypeIds();
@@ -432,17 +438,43 @@ public class DocumentCategorizerDL extends AbstractDL implements DocumentCategor
.orElse(-1);
}
- private Map<Integer, String> readCategoriesFromFile(File config) throws IOException {
- final DocumentCategorizerConfig documentCategorizerConfig =
- DocumentCategorizerConfig.fromJson(Files.readString(config.toPath(), StandardCharsets.UTF_8));
-
+ /**
+ * Reads the categories of a model from the {@code id2label} member of its configuration file.
+ *
+ * @param config The {@code config.json} file.
+ * @return The categories by output index, empty if the configuration has no {@code id2label}.
+ * @throws IOException Thrown if the file cannot be read.
+ * @throws InvalidFormatException Thrown if the file is not well-formed JSON, its
+ * {@code id2label} member does not map keys to strings, or a key is not an integer. The
+ * message names the file.
+ */
+ static Map<Integer, String> readCategories(File config) throws IOException {
+ final String json = Files.readString(config.toPath(), StandardCharsets.UTF_8);
final Map<Integer, String> categories = new HashMap<>();
- for (final String key : documentCategorizerConfig.id2label().keySet()) {
- categories.put(Integer.valueOf(key), documentCategorizerConfig.id2label().get(key));
+ try {
+ for (Map.Entry<String, String> label : DocumentCategorizerConfig.fromJson(json).id2label().entrySet()) {
+ categories.put(parseIndex(label.getKey()), label.getValue());
+ }
+ } catch (IllegalArgumentException e) {
+ throw new InvalidFormatException(
+ "Configuration file " + config.getName() + ": " + e.getMessage(), e);
}
-
return categories;
+ }
+ /**
+ * Parses an {@code id2label} key as an output index.
+ *
+ * @param key The key to parse.
+ * @return The output index.
+ * @throws IllegalArgumentException Thrown if {@code key} is not an integer.
+ */
+ private static int parseIndex(String key) {
+ try {
+ return Integer.parseInt(key);
+ } catch (NumberFormatException e) {
+ throw new IllegalArgumentException("id2label key must be an integer: " + key, e);
+ }
}
}
diff --git a/opennlp-core/opennlp-ml/opennlp-dl/src/main/java/opennlp/dl/namefinder/NameFinderDL.java b/opennlp-core/opennlp-ml/opennlp-dl/src/main/java/opennlp/dl/namefinder/NameFinderDL.java
index db55827b1..877b15c2d 100644
--- a/opennlp-core/opennlp-ml/opennlp-dl/src/main/java/opennlp/dl/namefinder/NameFinderDL.java
+++ b/opennlp-core/opennlp-ml/opennlp-dl/src/main/java/opennlp/dl/namefinder/NameFinderDL.java
@@ -128,6 +128,8 @@ public class NameFinderDL extends AbstractDL implements OffsetMappingNameFinder
*
* @throws OrtException Thrown if the {@code model} cannot be loaded.
* @throws IOException Thrown if errors occurred loading the {@code model} or {@code vocabulary}.
+ * @throws opennlp.tools.util.InvalidFormatException Thrown if a JSON {@code vocabulary}
+ * is malformed.
* @throws IllegalArgumentException Thrown if {@code inferenceOptions}, {@code ids2Labels}, or
* the sentence detector is {@code null}.
*/
@@ -151,6 +153,8 @@ public class NameFinderDL extends AbstractDL implements OffsetMappingNameFinder
*
* @throws OrtException Thrown if the {@code model} cannot be loaded.
* @throws IOException Thrown if errors occurred loading the {@code model} or {@code vocabulary}.
+ * @throws opennlp.tools.util.InvalidFormatException Thrown if a JSON {@code vocabulary}
+ * is malformed.
* @throws IllegalArgumentException Thrown if {@code inferenceOptions}, {@code ids2Labels}, or
* the sentence detector is {@code null}.
*/
diff --git a/opennlp-core/opennlp-ml/opennlp-dl/src/main/java/opennlp/dl/vectors/SentenceVectorsDL.java b/opennlp-core/opennlp-ml/opennlp-dl/src/main/java/opennlp/dl/vectors/SentenceVectorsDL.java
index 0fe148d33..7b1d14880 100644
--- a/opennlp-core/opennlp-ml/opennlp-dl/src/main/java/opennlp/dl/vectors/SentenceVectorsDL.java
+++ b/opennlp-core/opennlp-ml/opennlp-dl/src/main/java/opennlp/dl/vectors/SentenceVectorsDL.java
@@ -103,6 +103,8 @@ public class SentenceVectorsDL extends AbstractDL implements TextEmbedder {
*
* @throws OrtException Thrown if the {@code model} cannot be loaded.
* @throws IOException Thrown if errors occurred loading the {@code model} or {@code vocabulary}.
+ * @throws opennlp.tools.util.InvalidFormatException Thrown if a JSON {@code vocabulary}
+ * is malformed.
*/
public SentenceVectorsDL(final File model, final File vocabulary)
throws OrtException, IOException {
@@ -123,6 +125,8 @@ public class SentenceVectorsDL extends AbstractDL implements TextEmbedder {
*
* @throws OrtException Thrown if the {@code model} cannot be loaded.
* @throws IOException Thrown if errors occurred loading the {@code model} or {@code vocabulary}.
+ * @throws opennlp.tools.util.InvalidFormatException Thrown if a JSON {@code vocabulary}
+ * is malformed.
*/
public SentenceVectorsDL(final File model, final File vocabulary, final boolean lowerCase)
throws OrtException, IOException {
diff --git a/opennlp-core/opennlp-ml/opennlp-dl/src/test/java/opennlp/dl/JsonScanTest.java b/opennlp-core/opennlp-ml/opennlp-dl/src/test/java/opennlp/dl/JsonScanTest.java
new file mode 100644
index 000000000..7704f2053
--- /dev/null
+++ b/opennlp-core/opennlp-ml/opennlp-dl/src/test/java/opennlp/dl/JsonScanTest.java
@@ -0,0 +1,454 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package opennlp.dl;
+
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Stream;
+
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
+import org.junit.jupiter.params.provider.ValueSource;
+
+import opennlp.dl.JsonScan.Member;
+
+public class JsonScanTest {
+
+ private static final int DEEP = 100_000;
+
+ static Stream<Arguments> documents() {
+ return Stream.of(
+ Arguments.of("{}", List.of()),
+ Arguments.of(" \t\r\n{ \t\r\n} \t\r\n", List.of()),
+ Arguments.of("{\"a\":1}", List.of("a=1")),
+ Arguments.of("{\"a\": 1, \"b\": \"x\"}", List.of("a=1", "b=\"x\"")),
+ Arguments.of("{ \"a\"\n:\r\n\t1 ,\n\"b\" : 2 }", List.of("a=1", "b=2")),
+ Arguments.of("{\"a\":1,\"b\":2}", List.of("a=1", "b=2")),
+ Arguments.of("{\"a\" : 1 , \"b\" :\t2}", List.of("a=1", "b=2")),
+ // each value kind is skipped in full
+ Arguments.of("{\"o\": {\"n\": {\"m\": [1, {\"k\": \"}\"}]}}, \"a\": [[], {}, \"]\"],"
+ + " \"t\": true, \"f\": false, \"z\": null, \"n\": -1.5e+10, \"e\": 0}",
+ List.of("o={\"n\": {\"m\": [1, {\"k\": \"}\"}]}}", "a=[[], {}, \"]\"]", "t=true",
+ "f=false", "z=null", "n=-1.5e+10", "e=0")),
+ Arguments.of("{\"a\":{},\"b\":[],\"c\":{\"d\":{}},\"e\":[[]]}",
+ List.of("a={}", "b=[]", "c={\"d\":{}}", "e=[[]]")),
+ // numbers in the forms RFC 8259 allows
+ Arguments.of("{\"a\": 0, \"b\": -0, \"c\": 10, \"d\": -10.5, \"e\": 1e10, \"f\": 1E-10,"
+ + " \"g\": 1.5e+3, \"h\": 0.5}",
+ List.of("a=0", "b=-0", "c=10", "d=-10.5", "e=1e10", "f=1E-10", "g=1.5e+3", "h=0.5")),
+ // keys are unescaped, values are reported as written
+ Arguments.of("{\"a\\\"b\": \"c\\\"d\", \"\\u00e9\": \"\\u00e9\"}",
+ List.of("a\"b=\"c\\\"d\"", "\u00E9=\"\\u00e9\"")),
+ Arguments.of("{\"a\\\\\": 1, \"\\\\\\\"\": 2, \"\\u0120\": 3}",
+ List.of("a\\=1", "\\\"=2", "\u0120=3")),
+ // a control character inside a string is content
+ Arguments.of("{\"a\nb\": \"c\nd\"}", List.of("a\nb=\"c\nd\"")),
+ Arguments.of("{\"a\u0001b\": 1}", List.of("a\u0001b=1")),
+ // an empty key and duplicate keys are members like any other
+ Arguments.of("{\"\": 1, \"a\": 1, \"a\": 2}", List.of("=1", "a=1", "a=2")),
+ // non-ASCII and supplementary-plane text in keys and values
+ Arguments.of("{\"\uD83D\uDE00\": \"\uD801\uDC12\", \"\u00E9\": \"\u3000x\"}",
+ List.of("\uD83D\uDE00=\"\uD801\uDC12\"", "\u00E9=\"\u3000x\"")));
+ }
+
+ @ParameterizedTest
+ @MethodSource("documents")
+ void testDocument(String text, List<String> expected) {
+ Assertions.assertEquals(expected, render(text, JsonScan.document(text)));
+ }
+
+ private static List<String> render(String text, List<Member> members) {
+ return members.stream()
+ .map(m -> m.key() + "=" + text.substring(m.valueStart(), m.valueEnd()))
+ .toList();
+ }
+
+ @ParameterizedTest
+ @ValueSource(strings = {
+ "", " ", "[]", "\"a\"", "1", "null",
+ "{", "{\"a\"", "{\"a\":", "{\"a\":1", "{\"a\":1,", "{\"a\":1,}", "{,}", "{\"a\" 1}",
+ "{\"a\":1 \"b\":2}", "{a:1}", "{'a':1}", "{\"a\":1}}", "{\"a\":1} x", "{}{}", "{\"a\":1}]",
+ "{\"a\"::1}",
+ // strings
+ "{\"a\":\"x}", "{\"a\":\"x\\\"}", "{\"a\\", "{\"a\\q\":1}", "{\"\\u12\":1}",
+ "{\"\\u+123\":1}", "{\"\\u-123\":1}", "{\"\\u12G4\":1}", "{\"a\":\"\\x\"}",
+ // numbers
+ "{\"a\":01}", "{\"a\":-01}", "{\"a\":-}", "{\"a\":1.}", "{\"a\":.5}", "{\"a\":1e}",
+ "{\"a\":1e+}", "{\"a\":+1}", "{\"a\":0x1}", "{\"a\":\u0661}", "{\"a\":1.5.2}",
+ // literals and containers
+ "{\"a\":tru}", "{\"a\":True}", "{\"a\":nul}", "{\"a\":truex}", "{\"a\":[1,]}",
+ "{\"a\":[1 2]}", "{\"a\":[}", "{\"a\":{]}", "{\"a\":[1}}", "{\"a\":{\"b\":1]}",
+ // whitespace outside strings is only the four RFC 8259 characters
+ "{\"a\":\u00A01}", "{\"a\"\u3000:1}", "\u000B{}", "\f{}", "{\"a\":1\u0085}",
+ "{\"a\"\u2003:1}", "{\"a\":\t1\u000B}", "\u001C{}"})
+ void testDocumentRejectsMalformedText(String text) {
+ Assertions.assertThrows(IllegalArgumentException.class, () -> JsonScan.document(text));
+ }
+
+ @Test
+ void testDocumentRejectsNull() {
+ Assertions.assertThrows(IllegalArgumentException.class, () -> JsonScan.document(null));
+ }
+
+ @Test
+ void testMembersRejectsNull() {
+ Assertions.assertThrows(IllegalArgumentException.class, () -> JsonScan.members(null, 0));
+ }
+
+ @Test
+ void testMembersReadsANestedObject() {
+ final String text = "{\"x\": {\"a\": 1, \"b\": {\"c\": 2}}, \"y\": 3}";
+ final Member x = JsonScan.member(JsonScan.document(text), "x");
+ Assertions.assertTrue(JsonScan.isObject(text, x));
+ Assertions.assertEquals(List.of("a=1", "b={\"c\": 2}"),
+ render(text, JsonScan.members(text, x.valueStart())));
+ Assertions.assertThrows(IllegalArgumentException.class,
+ () -> JsonScan.members(text, x.valueStart() + 1));
+ }
+
+ @Test
+ void testMemberFindsTheLastWithAKey() {
+ final String text = "{\"a\": 1, \"b\": 2, \"a\": 3}";
+ final List<Member> members = JsonScan.document(text);
+ Assertions.assertEquals(3, JsonScan.nonNegativeIntValue(text, JsonScan.member(members, "a")));
+ Assertions.assertEquals(2, JsonScan.nonNegativeIntValue(text, JsonScan.member(members, "b")));
+ Assertions.assertNull(JsonScan.member(members, "c"));
+ Assertions.assertNull(JsonScan.member(List.of(), "a"));
+ }
+
+ static Stream<Arguments> stringValues() {
+ return Stream.of(
+ Arguments.of("{\"a\": \"\"}", ""),
+ Arguments.of("{\"a\": \"x y\"}", "x y"),
+ Arguments.of("{\"a\": \"say \\\"hi\\\"\"}", "say \"hi\""),
+ Arguments.of("{\"a\": \"\\\\ \\/ \\b \\f \\n \\r \\t\"}", "\\ / \b \f \n \r \t"),
+ Arguments.of("{\"a\": \"\\u0120\\u00E9\\u00e9\"}", "\u0120\u00E9\u00E9"),
+ // a surrogate pair written as two escapes, and one written as text
+ Arguments.of("{\"a\": \"\\uD83D\\uDE00\"}", "\uD83D\uDE00"),
+ Arguments.of("{\"a\": \"\\ud83d\\ude00\"}", "\uD83D\uDE00"),
+ Arguments.of("{\"a\": \"\uD83D\uDE00\"}", "\uD83D\uDE00"),
+ // an unpaired surrogate escape is decoded as written
+ Arguments.of("{\"a\": \"\\uD83D\"}", "\uD83D"),
+ Arguments.of("{\"a\": \"tab\there\"}", "tab\there"),
+ Arguments.of("{\"a\": \"li\nne\"}", "li\nne"),
+ Arguments.of("{\"a\": \"\u00A0\u3000\"}", "\u00A0\u3000"),
+ Arguments.of("{\"a\": \"x}y\"}", "x}y"),
+ Arguments.of("{\"a\": \"x\\\\\"}", "x\\"));
+ }
+
+ @ParameterizedTest
+ @MethodSource("stringValues")
+ void testStringValue(String text, String expected) {
+ Assertions.assertEquals(expected,
+ JsonScan.stringValue(text, JsonScan.document(text).get(0)));
+ }
+
+ @ParameterizedTest
+ @ValueSource(strings = {"{\"a\": 1}", "{\"a\": true}", "{\"a\": null}", "{\"a\": {}}",
+ "{\"a\": [\"x\"]}", "{\"a\": NaN}"})
+ void testStringValueRejectsOtherValues(String text) {
+ final Member a = JsonScan.document(text).get(0);
+ Assertions.assertThrows(IllegalArgumentException.class, () -> JsonScan.stringValue(text, a));
+ }
+
+ static Stream<Arguments> intValues() {
+ return Stream.of(
+ Arguments.of("{\"a\": 0}", 0),
+ Arguments.of("{\"a\": 7}", 7),
+ Arguments.of("{\"a\":42}", 42),
+ Arguments.of("{\"a\": 2147483647}", Integer.MAX_VALUE));
+ }
+
+ @ParameterizedTest
+ @MethodSource("intValues")
+ void testNonNegativeIntValue(String text, int expected) {
+ Assertions.assertEquals(expected,
+ JsonScan.nonNegativeIntValue(text, JsonScan.document(text).get(0)));
+ }
+
+ @ParameterizedTest
+ @ValueSource(strings = {"{\"a\": -1}", "{\"a\": -0}", "{\"a\": 1.5}", "{\"a\": 1.0}",
+ "{\"a\": 1e3}", "{\"a\": 2147483648}", "{\"a\": \"1\"}", "{\"a\": true}", "{\"a\": null}",
+ "{\"a\": [1]}", "{\"a\": NaN}", "{\"a\": Infinity}", "{\"a\": -Infinity}"})
+ void testNonNegativeIntValueRejectsOtherValues(String text) {
+ final Member a = JsonScan.document(text).get(0);
+ Assertions.assertThrows(IllegalArgumentException.class,
+ () -> JsonScan.nonNegativeIntValue(text, a));
+ }
+
+ static Stream<Arguments> stringObjects() {
+ return Stream.of(
+ Arguments.of("{\"m\": {\"0\": \"x\"}}", Map.of("0", "x")),
+ Arguments.of("{\"m\": {}}", Map.of()),
+ Arguments.of("{\"a\": 1, \"m\": {\"0\": \"x\", \"1\": \"y\"}, \"z\": [{}]}",
+ Map.of("0", "x", "1", "y")),
+ Arguments.of("{\"m\": {\"0\": \"x\"}, \"m\": {\"0\": \"y\"}}", Map.of("0", "y")),
+ Arguments.of("{\"m\": {\"0\": \"x\", \"0\": \"y\"}}", Map.of("0", "y")),
+ Arguments.of("{\"\\u006d\": {\"\\u0030\": \"\\u0078\"}}", Map.of("0", "x")),
+ // no such top-level member, or only a nested one
+ Arguments.of("{}", Map.of()),
+ Arguments.of("{\"n\": {\"0\": \"x\"}}", Map.of()),
+ Arguments.of("{\"n\": {\"m\": {\"0\": \"x\"}}}", Map.of()),
+ // blank text, with or without a byte order mark
+ Arguments.of("", Map.of()),
+ Arguments.of(" \r\n\t", Map.of()),
+ Arguments.of("\uFEFF", Map.of()),
+ Arguments.of("\uFEFF \n", Map.of()),
+ Arguments.of("\uFEFF{\"m\": {\"0\": \"x\"}}", Map.of("0", "x")));
+ }
+
+ @ParameterizedTest
+ @MethodSource("stringObjects")
+ void testStringObject(String text, Map<String, String> expected) {
+ Assertions.assertEquals(expected, JsonScan.stringObject(text, "m"));
+ }
+
+ @ParameterizedTest
+ @ValueSource(strings = {"{\"m\": \"x\"}", "{\"m\": [\"x\"]}", "{\"m\": null}", "{\"m\": 1}",
+ "{\"m\": {\"0\": 1}}", "{\"m\": {\"0\": null}}", "{\"m\": {\"0\": {}}}",
+ "{\"m\": {\"0\": \"x\"}", "[]", "x", "\u00A0", "\u001C", "\uFEFF\uFEFF"})
+ void testStringObjectRejectsOtherText(String text) {
+ Assertions.assertThrows(IllegalArgumentException.class, () -> JsonScan.stringObject(text, "m"));
+ }
+
+ @Test
+ void testStringObjectRejectsNullArguments() {
+ Assertions.assertThrows(IllegalArgumentException.class, () -> JsonScan.stringObject(null, "m"));
+ Assertions.assertThrows(IllegalArgumentException.class, () -> JsonScan.stringObject("{}", null));
+ }
+
+ static Stream<Arguments> documentsWithAByteOrderMark() {
+ return Stream.of(
+ Arguments.of("\uFEFF{\"a\":1}", List.of("a=1")),
+ Arguments.of("\uFEFF \r\n{ \"a\" : 1 }\r\n", List.of("a=1")),
+ Arguments.of("\uFEFF{}", List.of()));
+ }
+
+ @ParameterizedTest
+ @MethodSource("documentsWithAByteOrderMark")
+ void testDocumentSkipsALeadingByteOrderMark(String text, List<String> expected) {
+ Assertions.assertEquals(expected, render(text, JsonScan.document(text)));
+ }
+
+ static Stream<Arguments> byteOrderMarksElsewhere() {
+ return Stream.of(
+ Arguments.of("\uFEFF\uFEFF{}", 1),
+ Arguments.of(" \uFEFF{}", 1),
+ Arguments.of("{\uFEFF}", 1),
+ Arguments.of("{}\uFEFF", 2),
+ Arguments.of("{\"a\":\uFEFF1}", 5),
+ Arguments.of("\uFEFF", 1));
+ }
+
+ @ParameterizedTest
+ @MethodSource("byteOrderMarksElsewhere")
+ void testDocumentRejectsAByteOrderMarkElsewhere(String text, int offset) {
+ IllegalArgumentException e = Assertions.assertThrows(IllegalArgumentException.class,
+ () -> JsonScan.document(text));
+ Assertions.assertTrue(e.getMessage().contains("offset " + offset + ","), e.getMessage());
+ }
+
+ static Stream<Arguments> malformedOffsets() {
+ return Stream.of(
+ // empty text, or whitespace only
+ Arguments.of("", 0, "expected '{'"),
+ Arguments.of(" ", 3, "expected '{'"),
+ Arguments.of("\u000B{}", 0, "expected '{'"),
+ // cut off after each token of a member
+ Arguments.of("{", 1, "expected '\"'"),
+ Arguments.of("{\"a", 1, "unterminated string"),
+ Arguments.of("{\"a\\", 1, "unterminated string"),
+ Arguments.of("{\"a\"", 4, "expected ':'"),
+ Arguments.of("{\"a\":", 5, "expected a value"),
+ Arguments.of("{\"a\": ", 6, "expected a value"),
+ Arguments.of("{\"a\":1", 6, "expected ','"),
+ Arguments.of("{\"a\":1,", 7, "expected '\"'"),
+ Arguments.of("{\"a\":1,}", 7, "expected '\"'"),
+ Arguments.of("{\"a\":\"x", 5, "unterminated string"),
+ Arguments.of("{\"a\":\"x\\\"", 5, "unterminated string"),
+ Arguments.of("{\"a\":\"x\\", 5, "unterminated string"),
+ Arguments.of("{\"a\":[1", 7, "expected ','"),
+ Arguments.of("{\"a\":[1,", 8, "expected a value"),
+ Arguments.of("{\"a\":[", 6, "expected a value"),
+ Arguments.of("{\"a\":[}", 6, "expected a value"),
+ Arguments.of("{\"a\":{]}", 6, "expected '\"'"),
+ Arguments.of("{\"a\":[1}", 7, "expected ','"),
+ Arguments.of("{\"a\":{\"b\":1]}", 11, "expected ','"),
+ Arguments.of("{\"a\":{\"b\":1}", 12, "expected ','"),
+ Arguments.of("{\"a\":{\"b\":", 10, "expected a value"),
+ Arguments.of("{\"a\":tr", 5, "expected a value"),
+ Arguments.of("{\"a\":-", 6, "expected a digit"),
+ Arguments.of("{\"a\":1.", 7, "expected a digit"),
+ Arguments.of("{\"a\":1e", 7, "expected a digit"),
+ Arguments.of("{\"a\":1e+", 8, "expected a digit"),
+ Arguments.of("{\"a\":-01}", 7, "expected ','"),
+ // content after the object
+ Arguments.of("{\"a\":1}x", 7, "content after the object"),
+ Arguments.of("{\"a\":1} \n{}", 9, "content after the object"),
+ Arguments.of("{}}", 2, "content after the object"),
+ Arguments.of("{}\u00A0", 2, "content after the object"),
+ // wrong separators and whitespace that is not JSON whitespace
+ Arguments.of("{\"a\" 1}", 5, "expected ':'"),
+ Arguments.of("{\"a\"::1}", 5, "expected a value"),
+ Arguments.of("{\"a\":1 \"b\":2}", 7, "expected ','"),
+ Arguments.of("{\"a\":\u00A01}", 5, "expected a value"),
+ Arguments.of("{\"a\"\u3000:1}", 4, "expected ':'"),
+ Arguments.of("{\"a\"\u000B:1}", 4, "expected ':'"),
+ Arguments.of("{\"a\"\u2003:1}", 4, "expected ':'"),
+ // escapes are checked where the backslash is
+ Arguments.of("{\"\\u12\":1}", 2, "four hexadecimal digits"),
+ Arguments.of("{\"a\\q\":1}", 3, "unknown escape \\q"));
+ }
+
+ @ParameterizedTest
+ @MethodSource("malformedOffsets")
+ void testMalformedMessageNamesTheOffsetAndReason(String text, int offset, String reason) {
+ IllegalArgumentException e = Assertions.assertThrows(IllegalArgumentException.class,
+ () -> JsonScan.document(text));
+ Assertions.assertTrue(e.getMessage().contains("offset " + offset + ","), e.getMessage());
+ Assertions.assertTrue(e.getMessage().contains(reason), e.getMessage());
+ }
+
+ @Test
+ void testMalformedMessageQuotesTheTextAtTheOffset() {
+ IllegalArgumentException e = Assertions.assertThrows(IllegalArgumentException.class,
+ () -> JsonScan.document("{\"a\": 1, \"b\" 2}"));
+ Assertions.assertTrue(e.getMessage().contains("found '2}'"), e.getMessage());
+ e = Assertions.assertThrows(IllegalArgumentException.class, () -> JsonScan.document("{\"a\": "));
+ Assertions.assertTrue(e.getMessage().contains("end of text"), e.getMessage());
+ }
+
+ static Stream<Arguments> lineEndings() {
+ return Stream.of(
+ Arguments.of("{\r\n\t\"a\"\r\n\t:\r\n\t1,\r\n\t\"b\": \"x\"\r\n}\r\n",
+ List.of("a=1", "b=\"x\"")),
+ Arguments.of("{\r\"a\":\r1\r}\r", List.of("a=1")),
+ Arguments.of("{\"a\":[\r\n1,\r\n2\r\n],\"o\":{\r\n}}",
+ List.of("a=[\r\n1,\r\n2\r\n]", "o={\r\n}")),
+ // a line break inside a string is content, so it stays in the key
+ Arguments.of("{\"a\r\nb\":1}", List.of("a\r\nb=1")),
+ // escaped quotes, backslashes, and solidus in keys
+ Arguments.of("{\"\\u005B\": 1, \"a\\\\\": 2, \"\\/\": 3, \"\\\\\\\"\": 4}",
+ List.of("[=1", "a\\=2", "/=3", "\\\"=4")));
+ }
+
+ @ParameterizedTest
+ @MethodSource("lineEndings")
+ void testLineEndingsAndEscapedKeys(String text, List<String> expected) {
+ Assertions.assertEquals(expected, render(text, JsonScan.document(text)));
+ }
+
+ static Stream<Arguments> deepDocuments() {
+ return Stream.of(
+ Arguments.of("{\"a\":" + "[".repeat(DEEP) + "]".repeat(DEEP) + "}"),
+ Arguments.of("{\"a\":" + "{\"a\":".repeat(DEEP) + "0" + "}".repeat(DEEP) + "}"),
+ Arguments.of("{\"a\":" + "[{\"k\":".repeat(DEEP) + "0" + "}]".repeat(DEEP) + ",\"b\":1}"));
+ }
+
+ @ParameterizedTest(name = "deep document {index}")
+ @MethodSource("deepDocuments")
+ void testDeeplyNestedValuesAreSkippedWithoutRecursion(String text) {
+ final List<Member> members = JsonScan.document(text);
+ Assertions.assertEquals("a", members.get(0).key());
+ Assertions.assertEquals(5, members.get(0).valueStart());
+ }
+
+ @ParameterizedTest(name = "deep document {index} cut off")
+ @MethodSource("deepDocuments")
+ void testDeeplyNestedValueCutOffIsRejectedWithoutRecursion(String text) {
+ final String cut = text.substring(0, text.length() - 2);
+ final IllegalArgumentException e = Assertions.assertThrows(IllegalArgumentException.class,
+ () -> JsonScan.document(cut));
+ Assertions.assertTrue(e.getMessage().contains("offset " + cut.length() + ","), e.getMessage());
+ }
+
+ static Stream<Arguments> badEscapesInsideValues() {
+ return Stream.of(
+ Arguments.of("{\"a\":\"\\q\"}", 6, "unknown escape \\q"),
+ Arguments.of("{\"a\":[\"\\q\"]}", 7, "unknown escape \\q"),
+ Arguments.of("{\"a\":{\"\\q\":1}}", 7, "unknown escape \\q"),
+ Arguments.of("{\"a\":[\"\\u12\"]}", 7, "four hexadecimal digits"),
+ Arguments.of("{\"a\":{\"b\":\"\\u12G4\"}}", 11, "four hexadecimal digits"),
+ Arguments.of("{\"a\":\"x\\", 5, "unterminated string"));
+ }
+
+ @ParameterizedTest
+ @MethodSource("badEscapesInsideValues")
+ void testDocumentRejectsABadEscapeInsideAnyString(String text, int offset, String reason) {
+ final IllegalArgumentException e = Assertions.assertThrows(IllegalArgumentException.class,
+ () -> JsonScan.document(text));
+ Assertions.assertTrue(e.getMessage().contains("offset " + offset + ","), e.getMessage());
+ Assertions.assertTrue(e.getMessage().contains(reason), e.getMessage());
+ }
+
+ static Stream<Arguments> nonFiniteNumbers() {
+ return Stream.of(
+ Arguments.of("{\"a\": NaN, \"b\": 1}", List.of("a=NaN", "b=1")),
+ Arguments.of("{\"a\": Infinity}", List.of("a=Infinity")),
+ Arguments.of("{\"a\": -Infinity}", List.of("a=-Infinity")),
+ Arguments.of("{\"a\": [NaN, -Infinity, Infinity], \"b\": {\"c\": NaN}}",
+ List.of("a=[NaN, -Infinity, Infinity]", "b={\"c\": NaN}")));
+ }
+
+ @ParameterizedTest
+ @MethodSource("nonFiniteNumbers")
+ void testDocumentSkipsTheNonFiniteNumbersPythonWrites(String text, List<String> expected) {
+ Assertions.assertEquals(expected, render(text, JsonScan.document(text)));
+ }
+
+ @ParameterizedTest
+ @ValueSource(strings = {"{\"a\": nan}", "{\"a\": NAN}", "{\"a\": -NaN}", "{\"a\": +Infinity}",
+ "{\"a\": inf}", "{\"a\": Infinit}", "{\"a\": NaNx}"})
+ void testDocumentRejectsOtherSpellingsOfNonFiniteNumbers(String text) {
+ Assertions.assertThrows(IllegalArgumentException.class, () -> JsonScan.document(text));
+ }
+
+ @ParameterizedTest
+ @ValueSource(ints = {-1, 2, 3, Integer.MAX_VALUE})
+ void testMembersRejectsAnOffsetOutsideTheText(int brace) {
+ Assertions.assertThrows(IllegalArgumentException.class, () -> JsonScan.members("{}", brace));
+ }
+
+ @Test
+ void testMemberRecordRejectsAnInvalidRange() {
+ Assertions.assertThrows(IllegalArgumentException.class, () -> new Member("k", 5, 3));
+ Assertions.assertThrows(IllegalArgumentException.class, () -> new Member("k", -1, 3));
+ Assertions.assertThrows(IllegalArgumentException.class, () -> new Member(null, 0, 1));
+ }
+
+ @Test
+ void testValueReadersRejectAMemberOutsideTheText() {
+ final Member outside = new Member("k", 5, 9);
+ Assertions.assertThrows(IllegalArgumentException.class, () -> JsonScan.isObject("{}", outside));
+ Assertions.assertThrows(IllegalArgumentException.class, () -> JsonScan.stringValue("{}", outside));
+ Assertions.assertThrows(IllegalArgumentException.class,
+ () -> JsonScan.nonNegativeIntValue("{}", outside));
+ }
+
+ @Test
+ void testMemberRejectsNullArguments() {
+ Assertions.assertThrows(IllegalArgumentException.class, () -> JsonScan.member(null, "k"));
+ Assertions.assertThrows(IllegalArgumentException.class, () -> JsonScan.member(List.of(), null));
+ Assertions.assertThrows(IllegalArgumentException.class, () -> JsonScan.stringValue("{}", null));
+ Assertions.assertThrows(IllegalArgumentException.class, () -> JsonScan.stringValue(null,
+ new Member("k", 0, 1)));
+ }
+}
diff --git a/opennlp-core/opennlp-ml/opennlp-dl/src/test/java/opennlp/dl/LoadVocabTest.java b/opennlp-core/opennlp-ml/opennlp-dl/src/test/java/opennlp/dl/LoadVocabTest.java
index 8b3961e78..90940ba4e 100644
--- a/opennlp-core/opennlp-ml/opennlp-dl/src/test/java/opennlp/dl/LoadVocabTest.java
+++ b/opennlp-core/opennlp-ml/opennlp-dl/src/test/java/opennlp/dl/LoadVocabTest.java
@@ -21,28 +21,51 @@ import java.io.File;
import java.io.IOException;
import java.io.InputStream;
import java.nio.file.Files;
+import java.nio.file.Path;
import java.nio.file.StandardCopyOption;
import java.util.Map;
import java.util.Objects;
+import java.util.stream.Stream;
import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
+import org.junit.jupiter.params.provider.ValueSource;
+
+import opennlp.tools.util.InvalidFormatException;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertNotNull;
import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
public class LoadVocabTest {
+ @TempDir
+ private Path tempDir;
+
private File getResource(String name) throws IOException {
try (InputStream is = Objects.requireNonNull(
getClass().getResourceAsStream("/opennlp/dl/" + name))) {
- final File tempFile = File.createTempFile("vocab-test-", "-" + name);
- tempFile.deleteOnExit();
- Files.copy(is, tempFile.toPath(), StandardCopyOption.REPLACE_EXISTING);
- return tempFile;
+ final Path file = tempDir.resolve(name);
+ Files.copy(is, file, StandardCopyOption.REPLACE_EXISTING);
+ return file.toFile();
}
}
+ /**
+ * Writes a vocabulary file into the per-test temporary directory.
+ *
+ * @param name The file name.
+ * @param content The file content, written as UTF-8.
+ * @return The written file.
+ */
+ private File vocabFile(String name, String content) throws IOException {
+ return Files.writeString(tempDir.resolve(name), content).toFile();
+ }
+
@Test
void testLoadPlainTextVocab() throws IOException {
final Map<String, Integer> vocab = AbstractDL.loadVocabFile(getResource("vocab-plain.txt"));
@@ -73,11 +96,7 @@ public class LoadVocabTest {
@Test
void testJsonVocabWithEscapedCharacters() throws IOException {
- final File tempFile = File.createTempFile("vocab-escaped", ".json");
- tempFile.deleteOnExit();
-
- Files.writeString(tempFile.toPath(),
- "{\"hello\\\"world\": 0, \"back\\\\slash\": 1}");
+ final File tempFile = vocabFile("vocab-escaped.json", "{\"hello\\\"world\": 0, \"back\\\\slash\": 1}");
final Map<String, Integer> vocab = AbstractDL.loadVocabFile(tempFile);
@@ -89,30 +108,18 @@ public class LoadVocabTest {
@Test
void testJsonVocabWithUnicodeEscapedCharacters() throws IOException {
- final File tempFile = File.createTempFile("vocab-unicode", ".json");
- tempFile.deleteOnExit();
-
- Files.writeString(tempFile.toPath(),
+ final File tempFile = vocabFile("vocab-unicode.json",
"{\"\\u0120token\": 0, \"line\\rbreak\": 1, \"form\\ffeed\": 2}");
final Map<String, Integer> vocab = AbstractDL.loadVocabFile(tempFile);
assertNotNull(vocab);
assertEquals(3, vocab.size());
- assertEquals(0, vocab.get("Ġtoken"));
+ assertEquals(0, vocab.get("\u0120token"));
assertEquals(1, vocab.get("line\rbreak"));
assertEquals(2, vocab.get("form\ffeed"));
}
- @Test
- void testJsonVocabRejectsInvalidEscapedCharacters() throws IOException {
- final File tempFile = File.createTempFile("vocab-invalid-escape", ".json");
- tempFile.deleteOnExit();
-
- Files.writeString(tempFile.toPath(), "{\"bad\\xescape\": 0}");
-
- assertThrows(IllegalArgumentException.class, () -> AbstractDL.loadVocabFile(tempFile));
- }
@Test
void testJsonAndPlainTextVocabProduceSameResult() throws IOException {
@@ -121,4 +128,178 @@ public class LoadVocabTest {
assertEquals(plainVocab, jsonVocab);
}
+
+ static Stream<Arguments> jsonVocabs() {
+ return Stream.of(
+ Arguments.of("{}", Map.of()),
+ Arguments.of("{\"a\": 1, \"b\": 2}", Map.of("a", 1, "b", 2)),
+ Arguments.of(" \t\r\n{\"a\"\n:\r\n 3\t}\n", Map.of("a", 3)),
+ Arguments.of("{\"a\":0}", Map.of("a", 0)),
+ Arguments.of("{\"model\":0,\"vocab\":1}", Map.of("model", 0, "vocab", 1)),
+ Arguments.of("{\"a\": 2147483647}", Map.of("a", Integer.MAX_VALUE)),
+ Arguments.of("{\"a\\\"b\": 1}", Map.of("a\"b", 1)),
+ Arguments.of("{\"a\\\\\": 1}", Map.of("a\\", 1)),
+ Arguments.of("{\"\\u0120x\": 7, \"\\u00e9\": 8}", Map.of("\u0120x", 7, "\u00E9", 8)),
+ Arguments.of("{\"\uD83D\uDE00\": 1}", Map.of("\uD83D\uDE00", 1)),
+ // a raw line break inside a token is content
+ Arguments.of("{\"a\nb\": 1}", Map.of("a\nb", 1)),
+ Arguments.of("{\"\": 1}", Map.of("", 1)),
+ // a later entry for the same token, written or escaped, overwrites the earlier one
+ Arguments.of("{\"a\": 1, \"a\": 2}", Map.of("a", 2)),
+ Arguments.of("{\"a\": 1, \"\\u0061\": 2}", Map.of("a", 2)));
+ }
+
+ @ParameterizedTest
+ @MethodSource("jsonVocabs")
+ void testLoadJsonVocab(String json, Map<String, Integer> expected) {
+ assertEquals(expected, AbstractDL.loadJsonVocab(json));
+ }
+
+ @ParameterizedTest
+ @ValueSource(strings = {
+ // not one object
+ "", " ", "[]", "\"a\"", "{\"a\": 1}{}", "{\"a\": 1} x",
+ // malformed structure
+ "{\"a\": 1", "{\"a\" 1}", "{\"a\": 1 \"b\": 2}", "{\"a\": 1,}", "{a: 1}", "\"a\":1\"b\":2",
+ // whitespace outside strings is only the four RFC 8259 characters
+ "{\"a\":\u00A05}", "{\"a\":\t4\u000B}",
+ // values that are not non-negative integers
+ "{\"a\": 1.5}", "{\"a\": -1}", "{\"a\": 12abc}", "{\"a\": \u0661}", "{\"a\": \"1\"}",
+ "{\"a\": 99999999999}", "{\"a\": {\"b\": 1}}", "{\"a\": [1]}", "{\"a\": null}",
+ // invalid escapes in a token
+ "{\"a\\q\": 1}", "{\"\\\uD83D\uDE00\": 2}", "{\"\\u12\": 3}", "{\"\\u+123\": 3}",
+ "{\"a\\\nb\": 1}"})
+ void testLoadJsonVocabRejectsMalformedText(String json) {
+ assertThrows(IllegalArgumentException.class, () -> AbstractDL.loadJsonVocab(json));
+ }
+
+ @Test
+ void testLoadJsonVocabRejectsNull() {
+ assertThrows(IllegalArgumentException.class, () -> AbstractDL.loadJsonVocab(null));
+ }
+
+ @Test
+ void testLoadJsonVocabMessageNamesTheToken() {
+ final IllegalArgumentException e = assertThrows(IllegalArgumentException.class,
+ () -> AbstractDL.loadJsonVocab("{\"ok\": 1, \"bad\": -1}"));
+ assertTrue(e.getMessage().contains("\"bad\""), e.getMessage());
+ }
+
+ @ParameterizedTest
+ @ValueSource(strings = {
+ "{\"version\":\"1.0\",\"model\":{\"type\":\"WordPiece\",\"vocab\":{\"a\":0}}}",
+ "{\"version\":\"1.0\",\"model\":{\"type\":\"BPE\",\"vocab\":{\"a\":0}}}",
+ "{\"version\":\"1.0\",\"model\":{\"type\":\"Unigram\",\"vocab\":[[\"a\",0.0]]}}",
+ "{\"model\":{\"vocab\":{\"a\":0}}}"})
+ void testTokenizerFileIsRejectedWithTheExpectedVocabularyLayout(String json) throws IOException {
+ final File tempFile = vocabFile("tokenizer-unsupported.json", json);
+
+ final InvalidFormatException e = assertThrows(InvalidFormatException.class,
+ () -> AbstractDL.loadVocabFile(tempFile));
+ assertTrue(e.getMessage().contains(tempFile.getName()), e.getMessage());
+ assertTrue(e.getMessage().contains(
+ "Expected one object mapping tokens to integer ids, as in vocab.json"), e.getMessage());
+ }
+
+ @Test
+ void testLoadJsonVocabSkipsALeadingByteOrderMark() {
+ assertEquals(Map.of("a", 1), AbstractDL.loadJsonVocab("\uFEFF{\"a\": 1}"));
+ }
+
+ @Test
+ void testJsonVocabFileWithAByteOrderMarkIsReadAsJson() throws IOException {
+ final File tempFile = vocabFile("vocab-bom.json", "\uFEFF{\"a\": 0, \"b\": 1}\n");
+
+ assertEquals(Map.of("a", 0, "b", 1), AbstractDL.loadVocabFile(tempFile));
+ }
+
+ @Test
+ void testPlainTextVocabFileWithAByteOrderMarkKeepsTheFirstToken() throws IOException {
+ final File tempFile = vocabFile("vocab-bom.txt", "\uFEFF[CLS]\n[SEP]\n");
+
+ assertEquals(Map.of("[CLS]", 0, "[SEP]", 1), AbstractDL.loadVocabFile(tempFile));
+ }
+
+ @ParameterizedTest
+ @ValueSource(strings = {"2147483648", "4294967296", "9223372036854775808",
+ "12345678901234567890"})
+ void testLoadJsonVocabNamesTheTokenOfAnIdThatDoesNotFit(String id) {
+ final String json = "{\"big\": " + id + "}";
+ final IllegalArgumentException e = assertThrows(IllegalArgumentException.class,
+ () -> AbstractDL.loadJsonVocab(json), json);
+ assertTrue(e.getMessage().contains("\"big\""), e.getMessage());
+ assertTrue(e.getMessage().contains("does not fit into an int"), e.getMessage());
+ }
+
+ @ParameterizedTest
+ @ValueSource(strings = {"{\"a\": 00}", "{\"a\": 0123}"})
+ void testLoadJsonVocabRejectsLeadingZeros(String json) {
+ final IllegalArgumentException e = assertThrows(IllegalArgumentException.class,
+ () -> AbstractDL.loadJsonVocab(json));
+ assertTrue(e.getMessage().contains("offset "), e.getMessage());
+ }
+
+ private static final String JSON_VOCAB = "{\"[PAD]\":0,\"hello\":1,\"\\u0120x\":2}";
+
+ static Stream<Arguments> jsonVocabPrefixes() {
+ final String text = JSON_VOCAB;
+ return Stream.iterate(0, n -> n + 1).limit(text.length())
+ .map(n -> Arguments.of(n, text.substring(0, n)));
+ }
+
+ @ParameterizedTest(name = "cut at {0}")
+ @MethodSource("jsonVocabPrefixes")
+ void testLoadJsonVocabRejectsATruncatedObject(int length, String prefix) {
+ final IllegalArgumentException e = assertThrows(IllegalArgumentException.class,
+ () -> AbstractDL.loadJsonVocab(prefix));
+ assertTrue(e.getMessage().contains("offset "), e.getMessage());
+ }
+
+ @ParameterizedTest
+ @ValueSource(strings = {"{}", "x", "\uFEFF", ",", "\"vocab\"", "{\"a\":1}"})
+ void testLoadJsonVocabRejectsContentAfterTheObject(String trailing) {
+ final String json = JSON_VOCAB + trailing;
+ final IllegalArgumentException e = assertThrows(IllegalArgumentException.class,
+ () -> AbstractDL.loadJsonVocab(json));
+ assertTrue(e.getMessage().contains("offset " + JSON_VOCAB.length() + ","), e.getMessage());
+ assertTrue(e.getMessage().contains("content after the object"), e.getMessage());
+ }
+
+ @Test
+ void testJsonVocabFileWithWindowsLineEndings() throws IOException {
+ final File tempFile = vocabFile("vocab-crlf.json", "{\r\n \"a\": 0,\r\n \"b\": 1\r\n}\r\n");
+
+ assertEquals(Map.of("a", 0, "b", 1), AbstractDL.loadVocabFile(tempFile));
+ }
+
+ @Test
+ void testPlainTextVocabFileWithWindowsLineEndings() throws IOException {
+ final File tempFile = vocabFile("vocab-crlf.txt", "[CLS]\r\n[SEP]\r\nhello\r\n");
+
+ assertEquals(Map.of("[CLS]", 0, "[SEP]", 1, "hello", 2), AbstractDL.loadVocabFile(tempFile));
+ }
+
+ @Test
+ void testMalformedJsonVocabFileIsReportedAsAnInvalidFormat() throws IOException {
+ final File tempFile = vocabFile("vocab-malformed.json", "{\"a\": 1, \"b\": }");
+
+ final InvalidFormatException e =
+ assertThrows(InvalidFormatException.class, () -> AbstractDL.loadVocabFile(tempFile));
+ assertTrue(e.getMessage().contains(tempFile.getName()), e.getMessage());
+ assertTrue(e.getMessage().contains("offset "), e.getMessage());
+ }
+
+ @Test
+ void testJsonVocabFileWithAnInvalidEscapeIsReportedAsAnInvalidFormat() throws IOException {
+ final File tempFile = vocabFile("vocab-invalid-escape.json", "{\"bad\\xescape\": 0}");
+
+ assertThrows(InvalidFormatException.class, () -> AbstractDL.loadVocabFile(tempFile));
+ }
+
+ @Test
+ void testLoadJsonVocabRejectsANonFiniteId() {
+ final IllegalArgumentException e = assertThrows(IllegalArgumentException.class,
+ () -> AbstractDL.loadJsonVocab("{\"ok\": 1, \"bad\": NaN}"));
+ assertTrue(e.getMessage().contains("\"bad\""), e.getMessage());
+ }
}
diff --git a/opennlp-core/opennlp-ml/opennlp-dl/src/test/java/opennlp/dl/doccat/DocumentCategorizerConfigTest.java b/opennlp-core/opennlp-ml/opennlp-dl/src/test/java/opennlp/dl/doccat/DocumentCategorizerConfigTest.java
index d381b2226..0c309836e 100644
--- a/opennlp-core/opennlp-ml/opennlp-dl/src/test/java/opennlp/dl/doccat/DocumentCategorizerConfigTest.java
+++ b/opennlp-core/opennlp-ml/opennlp-dl/src/test/java/opennlp/dl/doccat/DocumentCategorizerConfigTest.java
@@ -17,15 +17,87 @@
package opennlp.dl.doccat;
import java.util.Map;
+import java.util.stream.Stream;
import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
+import org.junit.jupiter.params.provider.ValueSource;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
public class DocumentCategorizerConfigTest {
+ static Stream<Arguments> id2labels() {
+ return Stream.of(
+ Arguments.of("{\"id2label\": {\"0\": \"x\"}}", Map.of("0", "x")),
+ Arguments.of("{\"id2label\":{}}", Map.of()),
+ Arguments.of("{\"id2label\" : {\n\"0\" : \"neg\" ,\n\"1\"\t:\r\n\"pos\"\n}}",
+ Map.of("0", "neg", "1", "pos")),
+ // no id2label member, regardless of the other members
+ Arguments.of("{\"vocab_size\": 5}", Map.of()),
+ Arguments.of("{\"label2id\": {\"x\": 0}, \"my_id2label\": {\"0\": \"x\"}}", Map.of()),
+ Arguments.of("{\"note\": \"the \\\"id2label\\\" map\"}", Map.of()),
+ // only the top-level member counts
+ Arguments.of("{\"other\": {\"id2label\": {\"0\": \"x\"}}}", Map.of()),
+ // a later member with the same key overwrites the earlier one
+ Arguments.of("{\"id2label\": {\"0\": \"x\"}, \"id2label\": {\"0\": \"y\"}}",
+ Map.of("0", "y")),
+ Arguments.of("{\"id2label\": {\"0\": \"x\", \"0\": \"y\"}}", Map.of("0", "y")),
+ // escapes in keys and labels are decoded
+ Arguments.of("{\"id2label\": {\"0\": \"say \\\"hi\\\"\", \"1\": \"ok\"}}",
+ Map.of("0", "say \"hi\"", "1", "ok")),
+ Arguments.of("{\"id2label\": {\"a\\\"b\": \"c\\\\d\"}}", Map.of("a\"b", "c\\d")),
+ Arguments.of("{\"id2label\": {\"0\": \"\\u00e9\\uD83D\\uDE00\"}}", Map.of("0", "\u00E9\uD83D\uDE00")),
+ // a raw line break or tab inside a label is content
+ Arguments.of("{\"id2label\": {\"0\": \"li\nne\", \"1\": \"tab\there\"}}",
+ Map.of("0", "li\nne", "1", "tab\there")),
+ Arguments.of("{\"id2label\": {\"k\ney\": \"v\"}}", Map.of("k\ney", "v")),
+ // a brace inside a label does not end the object
+ Arguments.of("{\"id2label\": {\"0\": \"x\", \"1\": \"y}\", \"2\": \"z\"}}",
+ Map.of("0", "x", "1", "y}", "2", "z")),
+ Arguments.of("{\"id2label\": {\"\": \"x\", \"0\":\"\"}}", Map.of("", "x", "0", "")),
+ Arguments.of("{\"id2label\": {\"\uD83D\uDE00\": \"\uD801\uDC12\", \"\u00E9\": \"\u3000x\"}}",
+ Map.of("\uD83D\uDE00", "\uD801\uDC12", "\u00E9", "\u3000x")));
+ }
+
+ @ParameterizedTest
+ @MethodSource("id2labels")
+ public void testId2LabelsFromJson(String json, Map<String, String> expected) {
+ assertEquals(expected, DocumentCategorizerConfig.fromJson(json).id2label());
+ }
+
+ @ParameterizedTest
+ @ValueSource(strings = {
+ // id2label is not an object, or a label is not a string
+ "{\"id2label\": \"nope\"}", "{\"id2label\": [\"0\", \"x\"]}", "{\"id2label\": null}",
+ "{\"id2label\": {\"0\": 5}}", "{\"id2label\": {\"0\": {\"n\": \"y\"}}}",
+ "{\"id2label\": {\"0\": null}}",
+ // malformed text at any position of the configuration
+ "{\"id2label\": {\"0\": \"x\"", "{\"id2label\": {\"0\": \"x\" \"1\": \"y\"}}",
+ "{\"id2label\"id2label\": {\"0\": \"x\"}}", "{\"id2label\":\u00A0{\"0\": \"x\"}}",
+ "{\"id2label\": {\"0\": \"x\\q\"}}", "{\"hidden_size\": 768,}", "[]", "x"})
+ public void testId2LabelsFromJsonRejectsMalformedText(String json) {
+ assertThrows(IllegalArgumentException.class, () -> DocumentCategorizerConfig.fromJson(json));
+ }
+
+ @Test
+ public void testId2LabelsFromJsonNullThrows() {
+ assertThrows(IllegalArgumentException.class, () -> DocumentCategorizerConfig.fromJson(null));
+ }
+
+ @Test
+ public void testId2LabelsFromJsonMessageNamesTheKey() {
+ final IllegalArgumentException e = assertThrows(IllegalArgumentException.class,
+ () -> DocumentCategorizerConfig.fromJson("{\"id2label\": {\"0\": \"x\", \"1\": 2}}"));
+ assertTrue(e.getMessage().contains("\"1\""), e.getMessage());
+ }
+
@Test
public void testId2LabelsFromJsonPrettyValid() {
final String json = """
@@ -193,4 +265,107 @@ public class DocumentCategorizerConfigTest {
assertEquals("4 stars", map.get("a3"));
assertEquals("5 stars", map.get("a4"));
}
+
+ @Test
+ public void testId2LabelsFromJsonSkipsALeadingByteOrderMark() {
+ assertEquals(Map.of("0", "x"),
+ DocumentCategorizerConfig.fromJson("\uFEFF{\"id2label\": {\"0\": \"x\"}}").id2label());
+ }
+
+ @ParameterizedTest
+ @ValueSource(strings = {"\uFEFF", "\uFEFF \r\n\t", "\uFEFF\n", "", " \n"})
+ public void testId2LabelsFromJsonBlankTextAfterAByteOrderMarkHasNoLabels(String json) {
+ assertEquals(Map.of(), DocumentCategorizerConfig.fromJson(json).id2label());
+ }
+
+ @ParameterizedTest
+ @ValueSource(strings = {"\uFEFF\uFEFF", " \uFEFF", "\uFEFF \uFEFF", "\uFEFF{}\uFEFF",
+ "{\uFEFF}", "\uFEFF{\"id2label\":\uFEFF{}}"})
+ public void testId2LabelsFromJsonRejectsAByteOrderMarkElsewhere(String json) {
+ final IllegalArgumentException e = assertThrows(IllegalArgumentException.class,
+ () -> DocumentCategorizerConfig.fromJson(json));
+ assertTrue(e.getMessage().contains("offset "), e.getMessage());
+ }
+
+ private static final String CONFIG_JSON =
+ "{\"hidden_size\": 768, \"id2label\": {\"0\": \"neg\", \"1\": \"pos\"}, \"pad_token_id\": 0}";
+
+ static Stream<Arguments> configLayouts() {
+ return Stream.of(
+ Arguments.of(CONFIG_JSON, Map.of("0", "neg", "1", "pos")),
+ Arguments.of("{\r\n \"id2label\": {\r\n \"0\": \"neg\",\r\n \"1\": \"pos\"\r\n }\r\n}"
+ + "\r\n", Map.of("0", "neg", "1", "pos")),
+ Arguments.of("{\r\"id2label\":\r{\"0\":\r\"neg\"}\r}", Map.of("0", "neg")),
+ // the member name may be written with escapes
+ Arguments.of("{\"id2l\\u0061bel\": {\"0\": \"x\"}}", Map.of("0", "x")),
+ // nested members of any depth before and after id2label are skipped
+ Arguments.of("{\"a\": {\"b\": [[{\"c\": {\"id2label\": {\"9\": \"no\"}}}]]},"
+ + " \"id2label\": {\"0\": \"x\"}, \"d\": [{}, [], \"}\"]}", Map.of("0", "x")),
+ // labels with line breaks written as escapes or as text
+ Arguments.of("{\"id2label\": {\"0\": \"a\\r\\nb\", \"1\": \"c\r\nd\"}}",
+ Map.of("0", "a\r\nb", "1", "c\r\nd")));
+ }
+
+ @ParameterizedTest
+ @MethodSource("configLayouts")
+ public void testId2LabelsFromJsonLayouts(String json, Map<String, String> expected) {
+ assertEquals(expected, DocumentCategorizerConfig.fromJson(json).id2label());
+ }
+
+ static Stream<Arguments> configPrefixes() {
+ return Stream.iterate(1, n -> n + 1).limit(CONFIG_JSON.length() - 1)
+ .map(n -> Arguments.of(n, CONFIG_JSON.substring(0, n)));
+ }
+
+ @ParameterizedTest(name = "cut at {0}")
+ @MethodSource("configPrefixes")
+ public void testId2LabelsFromJsonRejectsATruncatedConfig(int length, String prefix) {
+ final IllegalArgumentException e = assertThrows(IllegalArgumentException.class,
+ () -> DocumentCategorizerConfig.fromJson(prefix));
+ assertTrue(e.getMessage().contains("offset "), e.getMessage());
+ }
+
+ @ParameterizedTest
+ @ValueSource(strings = {" {}", "x", "\uFEFF", ",", " {\"id2label\": {\"0\": \"y\"}}"})
+ public void testId2LabelsFromJsonRejectsContentAfterTheConfig(String trailing) {
+ final String json = CONFIG_JSON + trailing;
+ final int offset = CONFIG_JSON.length() + (trailing.startsWith(" ") ? 1 : 0);
+ final IllegalArgumentException e = assertThrows(IllegalArgumentException.class,
+ () -> DocumentCategorizerConfig.fromJson(json));
+ assertTrue(e.getMessage().contains("offset " + offset + ","), e.getMessage());
+ assertTrue(e.getMessage().contains("content after the object"), e.getMessage());
+ }
+
+ static Stream<Arguments> pythonWrittenConfigs() {
+ return Stream.of(
+ // NaN and the infinities in unrelated members are skipped
+ Arguments.of("{\"x\": NaN, \"id2label\": {\"0\": \"y\"}}", Map.of("0", "y")),
+ Arguments.of("{\"id2label\": {\"0\": \"y\"}, \"r\": [Infinity, -Infinity]}", Map.of("0", "y")),
+ // labels written with ensure_ascii escapes, lowercase hex and surrogate pairs included
+ Arguments.of("{\"id2label\": {\"0\": \"n\\u00e9gatif\", \"1\": \"\\ud83d\\ude00\"}}",
+ Map.of("0", "n\u00E9gatif", "1", "\uD83D\uDE00")),
+ Arguments.of("{\"id2label\": {\"0\": \"n\u00E9gatif\", \"1\": \"say \\\"hi\\\"\", \"2\": \"x}y\","
+ + " \"3\": \"z\"}}", Map.of("0", "n\u00E9gatif", "1", "say \"hi\"", "2", "x}y", "3", "z")),
+ // id2label nested in another member is not the configuration's map
+ Arguments.of("{\"text_config\": {\"id2label\": {\"0\": \"y\"}}}", Map.of()));
+ }
+
+ @ParameterizedTest
+ @MethodSource("pythonWrittenConfigs")
+ public void testId2LabelsFromJsonPythonWrittenConfigs(String json, Map<String, String> expected) {
+ assertEquals(expected, DocumentCategorizerConfig.fromJson(json).id2label());
+ }
+
+ @ParameterizedTest
+ @ValueSource(strings = {
+ // a trailing comma and a comment line are not JSON
+ "{\"id2label\": {\"0\": \"y\"},}", "{\"id2label\": {\"0\": \"y\",}}",
+ "// labels\n{\"id2label\": {\"0\": \"y\"}}",
+ // NaN is never a label
+ "{\"id2label\": {\"0\": NaN}}",
+ // a control character alone is not blank text under any whitespace mode
+ "\u001C", "\u0085", "\u00A0", "\u2028", "\uFEFF\u001C"})
+ public void testId2LabelsFromJsonRejectsTextThatIsNotAnObject(String json) {
+ assertThrows(IllegalArgumentException.class, () -> DocumentCategorizerConfig.fromJson(json));
+ }
}
diff --git a/opennlp-core/opennlp-ml/opennlp-dl/src/test/java/opennlp/dl/doccat/DocumentCategorizerDLTest.java b/opennlp-core/opennlp-ml/opennlp-dl/src/test/java/opennlp/dl/doccat/DocumentCategorizerDLTest.java
index 90c9abcea..5c31b3b43 100644
--- a/opennlp-core/opennlp-ml/opennlp-dl/src/test/java/opennlp/dl/doccat/DocumentCategorizerDLTest.java
+++ b/opennlp-core/opennlp-ml/opennlp-dl/src/test/java/opennlp/dl/doccat/DocumentCategorizerDLTest.java
@@ -17,14 +17,22 @@
package opennlp.dl.doccat;
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
import java.util.HashMap;
import java.util.Map;
import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.ValueSource;
import opennlp.dl.InferenceOptions;
import opennlp.dl.doccat.scoring.AverageClassificationScoringStrategy;
import opennlp.tools.tokenize.WordpieceTokenizer;
+import opennlp.tools.util.InvalidFormatException;
import static org.junit.jupiter.api.Assertions.assertArrayEquals;
import static org.junit.jupiter.api.Assertions.assertEquals;
@@ -200,4 +208,29 @@ public class DocumentCategorizerDLTest {
() -> DocumentCategorizerDL.requireMatchingCategoryCount(new double[] {1.0}, 2));
assertTrue(e.getMessage().contains("do not match"), e.getMessage());
}
+
+
+ @TempDir
+ private Path tempDir;
+
+ private File configFile(String json) throws IOException {
+ return Files.writeString(tempDir.resolve("config.json"), json).toFile();
+ }
+
+ @Test
+ void testReadCategoriesReadsTheLabelsByIndex() throws IOException {
+ assertEquals(Map.of(0, "neg", 1, "pos"),
+ DocumentCategorizerDL.readCategories(configFile("{\"id2label\": {\"0\": \"neg\", \"1\": \"pos\"}}")));
+ assertEquals(Map.of(), DocumentCategorizerDL.readCategories(configFile("{\"hidden_size\": 768}")));
+ }
+
+ @ParameterizedTest
+ @ValueSource(strings = {"{\"id2label\": {\"0\": \"neg\"", "{\"id2label\": [\"neg\"]}",
+ "{\"id2label\": {\"0\": 1}}", "{\"id2label\": {\"a0\": \"neg\"}}", "x"})
+ void testReadCategoriesReportsAMalformedFileAsAnInvalidFormat(String json) throws IOException {
+ final File file = configFile(json);
+ final InvalidFormatException e =
+ assertThrows(InvalidFormatException.class, () -> DocumentCategorizerDL.readCategories(file));
+ assertTrue(e.getMessage().contains(file.getName()), e.getMessage());
+ }
}
diff --git a/opennlp-docs/src/docbkx/doccat.xml b/opennlp-docs/src/docbkx/doccat.xml
index c5da29224..b41729334 100644
--- a/opennlp-docs/src/docbkx/doccat.xml
+++ b/opennlp-docs/src/docbkx/doccat.xml
@@ -172,6 +172,19 @@ String category = myCategorizer.getBestCategory(outcomes);]]>
</programlisting>
For additional examples, refer to the <code>DocumentCategorizerDLEval</code> class.
</para>
+ <para>
+ The vocabulary file is either plain text with one token per line, as in
+ <code>vocab.txt</code>, or a JSON object mapping each token to an integer ID, as in
+ <code>vocab.json</code>. A Hugging Face <code>tokenizer.json</code> is not supported yet
+ (<ulink url="https://issues.apache.org/jira/browse/OPENNLP-1988">OPENNLP-1988</ulink>).
+ An unreadable vocabulary is rejected with an <code>InvalidFormatException</code>.
+ </para>
+ <para>
+ When a configuration file is given in place of the categories map, its top-level
+ <code>id2label</code> object maps each output index to a category label. A configuration
+ that is not well-formed JSON, or whose <code>id2label</code> is not an object of strings
+ keyed by integers, is rejected with an <code>InvalidFormatException</code>.
+ </para>
<para>
Like <code>NameFinderDL</code>, long input is split into overlapping chunks on the full
Unicode <code>White_Space</code> set rather than Java's <code>\s</code>, so text copied