You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@opennlp.apache.org by rz...@apache.org on 2026/10/08 09:31:08 UTC
(opennlp) branch main updated: OPENNLP-1988: Load WordPiece vocabularies from tokenizer.json (#1319)
This is an automated email from the ASF dual-hosted git repository.
rzo1 pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/opennlp.git
The following commit(s) were added to refs/heads/main by this push:
new bca31bb7c OPENNLP-1988: Load WordPiece vocabularies from tokenizer.json (#1319)
bca31bb7c is described below
commit bca31bb7c2847ce1532187ae71ee056e45077cf6
Author: Kristian Rickert <kr...@apache.org>
AuthorDate: Thu Oct 8 04:31:02 2026 -0500
OPENNLP-1988: Load WordPiece vocabularies from tokenizer.json (#1319)
Accept Hugging Face tokenizer.json files as WordPiece vocabularies in
opennlp-dl. Read model.vocab and include added_tokens with their declared
IDs. Reject token or ID conflicts, malformed entries, unsupported model
types and subword prefixes other than ##.
Reject a normalizer.lowercase setting that differs from the constructor's
lowerCase argument. Other tokenizer settings are not imported.
Error messages name the member and offset of a value of the wrong type.
Add tests for vocabulary loading, invalid input and a leading byte order
mark, using fixtures created for the tests. Document the vocabulary rules
in the document categorizer chapter and link to them from the name finder
and embedder sections.
Assisted-by: Claude Code 2.1.281
Reviewed-by: Kristian Rickert <kr...@apache.org>
Signed-off-by: Kristian Rickert <kr...@apache.org>
Co-authored-by: Claude <no...@anthropic.com>
---
.../src/main/java/opennlp/dl/AbstractDL.java | 264 +++++++++--
.../src/main/java/opennlp/dl/JsonScan.java | 143 ++++--
.../opennlp/dl/doccat/DocumentCategorizerDL.java | 10 +-
.../java/opennlp/dl/namefinder/NameFinderDL.java | 6 +-
.../java/opennlp/dl/vectors/SentenceVectorsDL.java | 6 +-
.../src/test/java/opennlp/dl/JsonScanTest.java | 57 +++
.../src/test/java/opennlp/dl/LoadVocabTest.java | 501 +++++++++++++++++++--
.../dl/vectors/SentenceVectorsDLEmbedderTest.java | 25 +
.../src/test/resources/opennlp/dl/tokenizer.json | 79 ++++
opennlp-docs/src/docbkx/doccat.xml | 11 +-
opennlp-docs/src/docbkx/machine-learning.xml | 4 +-
opennlp-docs/src/docbkx/namefinder.xml | 3 +
12 files changed, 973 insertions(+), 136 deletions(-)
diff --git a/opennlp-core/opennlp-ml/opennlp-dl/src/main/java/opennlp/dl/AbstractDL.java b/opennlp-core/opennlp-ml/opennlp-dl/src/main/java/opennlp/dl/AbstractDL.java
index 61f1286e1..44127cee2 100644
--- a/opennlp-core/opennlp-ml/opennlp-dl/src/main/java/opennlp/dl/AbstractDL.java
+++ b/opennlp-core/opennlp-ml/opennlp-dl/src/main/java/opennlp/dl/AbstractDL.java
@@ -56,6 +56,22 @@ public abstract class AbstractDL implements AutoCloseable {
public static final String ATTENTION_MASK = "attention_mask";
public static final String TOKEN_TYPE_IDS = "token_type_ids";
+ private static final String TOKENIZER_MODEL_KEY = "model";
+ private static final String TOKENIZER_TYPE_KEY = "type";
+ private static final String TOKENIZER_VOCAB_KEY = "vocab";
+ private static final String TOKENIZER_SUBWORD_PREFIX_KEY = "continuing_subword_prefix";
+ private static final String TOKENIZER_ADDED_TOKENS_KEY = "added_tokens";
+ private static final String TOKENIZER_ID_KEY = "id";
+ private static final String TOKENIZER_CONTENT_KEY = "content";
+ private static final String TOKENIZER_NORMALIZER_KEY = "normalizer";
+ private static final String TOKENIZER_LOWERCASE_KEY = "lowercase";
+ /** The only {@code model.type} of a {@code tokenizer.json} the WordPiece encoder can use. */
+ private static final String WORDPIECE_MODEL_TYPE = "WordPiece";
+ /** The continuing subword prefix the WordPiece encoder assumes. */
+ private static final String WORDPIECE_SUBWORD_PREFIX = "##";
+ /** The first non-whitespace character of a vocabulary file that is read as JSON. */
+ private static final String JSON_OBJECT_START = "{";
+
protected final OrtEnvironment env;
protected final OrtSession session;
protected final SubwordTokenizer tokenizer;
@@ -96,11 +112,14 @@ public abstract class AbstractDL implements AutoCloseable {
* @param sessionOptions The session options (e.g. CUDA execution provider); build with
* {@link #sessionOptions(InferenceOptions)} when honoring {@link InferenceOptions}.
* @param lowerCase {@code true} for uncased models (lower casing and accent stripping
- * during tokenization), {@code false} for cased models.
+ * during tokenization), {@code false} for cased models. A {@code tokenizer.json} that
+ * sets {@code normalizer.lowercase} must agree with it.
*
* @throws OrtException Thrown if the {@code model} cannot be loaded.
* @throws IOException Thrown if the {@code model} or {@code vocabulary} cannot be read.
- * @throws InvalidFormatException Thrown if a JSON {@code vocabulary} is malformed.
+ * @throws InvalidFormatException Thrown if a JSON {@code vocabulary} is malformed, has an
+ * unsupported layout, or sets {@code normalizer.lowercase} to the other value than
+ * {@code lowerCase}.
*/
protected AbstractDL(final File model, final File vocabulary,
final OrtSession.SessionOptions sessionOptions, final boolean lowerCase)
@@ -113,7 +132,9 @@ public abstract class AbstractDL implements AutoCloseable {
try (sessionOptions) {
final OrtSession createdSession = env.createSession(model.getPath(), sessionOptions);
try {
- this.vocab = Map.copyOf(loadVocabFile(vocabulary));
+ final Vocabulary loaded = readVocabFile(vocabulary);
+ requireLowerCase(vocabulary, loaded, lowerCase);
+ this.vocab = Map.copyOf(loaded.ids());
this.tokenizer = createWordpieceEncoder(vocab, lowerCase);
} catch (IOException | RuntimeException e) {
// Vocabulary/tokenizer init failed after the native session was created; close it
@@ -170,47 +191,57 @@ public abstract class AbstractDL implements AutoCloseable {
/**
* Loads a vocabulary {@link File} from disk. A file with an opening brace as its first
- * non-whitespace character is read as JSON: one object that maps each token to a non-negative
- * integer ID, as in {@code vocab.json}. Any other file is read as plain text with one token
- * per line, the line number being the ID, as in {@code vocab.txt}. An empty line in a plain
- * text file is skipped but still counts toward the IDs of the lines after it; a line that
- * holds only whitespace is a token. A byte order mark at the start of the file is not
- * content in either format. The JSON layout is that of
- * {@link #loadJsonVocab(String)}.
+ * non-whitespace character is read as JSON, in one of two layouts: one object that maps each
+ * token to a non-negative integer ID, as in {@code vocab.json}, or a Hugging Face
+ * {@code tokenizer.json} of a WordPiece model, whose {@code model.vocab} object supplies the
+ * tokens and whose {@code added_tokens} absent from {@code model.vocab} are added with their
+ * ids. Any other file is read as plain text with one token per line, the line number being
+ * the ID, as in {@code vocab.txt}. An empty line in a plain text file is skipped but still
+ * counts toward the IDs of the lines after it; a line that holds only whitespace is a token.
+ * A byte order mark at the start of the file is not content in either format.
*
* @param vocabFile The vocabulary file.
* @return A map of vocabulary words to IDs.
* @throws IOException Thrown if the vocabulary file cannot be opened or read.
- * @throws InvalidFormatException Thrown if a JSON vocabulary is malformed, has a layout that
- * is not supported, or contains a value that is not a non-negative integer. The message
- * names the file and the offset or the token.
+ * @throws InvalidFormatException Thrown if a JSON vocabulary is malformed, contains a value
+ * that is not a non-negative integer, or is a {@code tokenizer.json} of another model type
+ * than WordPiece or with an added token that conflicts with the vocabulary. The message
+ * names the file and the offset, the token, or the unsupported member.
*/
public Map<String, Integer> loadVocab(
final File vocabFile) throws IOException {
- return loadVocabFile(vocabFile);
+ return readVocabFile(vocabFile).ids();
+ }
+
+ /**
+ * A vocabulary read from a file or JSON text.
+ *
+ * @param ids The map of tokens to IDs.
+ * @param lowercase The {@code normalizer.lowercase} setting of a {@code tokenizer.json}, or
+ * {@code null} if the vocabulary does not carry that setting.
+ */
+ record Vocabulary(Map<String, Integer> ids, Boolean lowercase) {
}
/**
- * Loads a vocabulary file as {@link #loadVocab(File)} does.
+ * Reads a vocabulary file as {@link #loadVocab(File)} does, keeping the settings of a
+ * {@code tokenizer.json} that a component must agree with.
*
* @param vocabFile The vocabulary file.
- * @return A map of vocabulary words to IDs.
+ * @return The vocabulary.
* @throws IOException Thrown if the vocabulary file cannot be opened or read.
* @throws InvalidFormatException Thrown if a JSON vocabulary is malformed, has a layout that
* is not supported, or contains a value that is not a non-negative integer.
*/
- static Map<String, Integer> loadVocabFile(
- final File vocabFile) throws IOException {
-
+ Vocabulary readVocabFile(final File vocabFile) throws IOException {
final String read = Files.readString(Path.of(vocabFile.getPath()), StandardCharsets.UTF_8);
final String content = StringUtil.stripByteOrderMark(read);
final String trimmed = content.trim();
- // Detect JSON format by leading brace
- if (trimmed.startsWith("{")) {
+ if (trimmed.startsWith(JSON_OBJECT_START)) {
try {
- return loadJsonVocab(trimmed);
+ return readJsonVocab(trimmed);
} catch (IllegalArgumentException e) {
throw new InvalidFormatException(
"Vocabulary file " + vocabFile.getName() + ": " + e.getMessage(), e);
@@ -229,7 +260,28 @@ public abstract class AbstractDL implements AutoCloseable {
}
});
- return vocab;
+ return new Vocabulary(vocab, null);
+ }
+
+ /**
+ * Checks that the lower casing a component is configured with agrees with the
+ * {@code normalizer.lowercase} setting of its vocabulary file, so that the component encodes
+ * text as the model's own tokenizer does.
+ *
+ * @param vocabFile The vocabulary file, named in the message.
+ * @param vocabulary The vocabulary read from it.
+ * @param lowerCase {@code true} if the component lower cases text, {@code false} otherwise.
+ * @throws InvalidFormatException Thrown if the file sets {@code normalizer.lowercase} to the
+ * other value.
+ */
+ void requireLowerCase(final File vocabFile, final Vocabulary vocabulary,
+ final boolean lowerCase) throws InvalidFormatException {
+ final Boolean lowercase = vocabulary.lowercase();
+ if (lowercase != null && lowercase != lowerCase) {
+ throw new InvalidFormatException("Vocabulary file " + vocabFile.getName()
+ + " sets normalizer.lowercase to " + lowercase
+ + ", but the component is configured with lowerCase " + lowerCase);
+ }
}
/**
@@ -543,30 +595,168 @@ public abstract class AbstractDL implements AutoCloseable {
}
/**
- * Reads a JSON vocabulary as one object mapping tokens to non-negative integer IDs, as in
- * {@code vocab.json}. Keys are decoded from their escapes, and a later entry for the same
- * token overwrites an earlier one.
+ * Reads a JSON vocabulary in one of two layouts: a {@code vocab.json} object that maps each
+ * token to its ID, or a {@code tokenizer.json} of a WordPiece model, recognized by a
+ * {@code model} object with a string {@code type}, whose {@code model.vocab} object supplies
+ * the tokens and whose {@code added_tokens} entries absent from {@code model.vocab} are added
+ * with their ids. Keys are decoded from their escapes, and a later entry for the same token
+ * overwrites an earlier one.
*
* @param json The JSON text of the vocabulary.
- * @return A map of vocabulary tokens to IDs.
+ * @return The vocabulary, with the {@code normalizer.lowercase} setting of a
+ * {@code tokenizer.json}.
* @throws IllegalArgumentException Thrown if the text is not a single well-formed JSON object,
- * or if a value of the vocabulary object is not a
- * non-negative integer that fits into an {@code int}. The message names the offset or the
- * token.
+ * if a value of the vocabulary object is not a non-negative integer that fits into an
+ * {@code int}, if a {@code tokenizer.json} is not that of a WordPiece model with the
+ * {@code ##} continuing subword prefix and a {@code model.vocab} object, or if an added
+ * token is malformed or conflicts with another token or id. The message names the offset,
+ * the token, or the unsupported member.
*/
- static Map<String, Integer> loadJsonVocab(final String json) {
+ Vocabulary readJsonVocab(final String json) {
final List<JsonScan.Member> document = JsonScan.document(json);
+ final JsonScan.Member model = JsonScan.member(document, TOKENIZER_MODEL_KEY);
+ if (model != null && JsonScan.isObject(json, model)) {
+ final List<JsonScan.Member> modelMembers = JsonScan.members(json, model.valueStart());
+ final JsonScan.Member type = JsonScan.member(modelMembers, TOKENIZER_TYPE_KEY);
+ if (type != null && JsonScan.isString(json, type)) {
+ final Map<String, Integer> vocab =
+ tokenizerVocab(json, modelMembers, JsonScan.stringValue(json, type));
+ addTokens(json, document, vocab);
+ return new Vocabulary(vocab, lowercaseSetting(json, document));
+ }
+ }
final Map<String, Integer> vocab = new HashMap<>();
for (JsonScan.Member member : document) {
- try {
- vocab.put(member.key(), JsonScan.nonNegativeIntValue(json, member));
- } catch (IllegalArgumentException e) {
+ vocab.put(member.key(), JsonScan.nonNegativeIntValue(json, member));
+ }
+ return new Vocabulary(vocab, null);
+ }
+
+ /**
+ * Reads the vocabulary of the {@code model} object of a {@code tokenizer.json}.
+ *
+ * @param json The JSON text.
+ * @param model The members of the {@code model} object.
+ * @param type The value of {@code model.type}.
+ * @return A map of the tokens of {@code model.vocab} to their IDs.
+ * @throws IllegalArgumentException Thrown if {@code type} is not {@code WordPiece}, if
+ * {@code model.continuing_subword_prefix} is present and not {@code ##}, if
+ * {@code model.vocab} is missing or not an object, or if a value of it is not a
+ * non-negative integer that fits into an {@code int}.
+ */
+ private Map<String, Integer> tokenizerVocab(final String json,
+ final List<JsonScan.Member> model, final String type) {
+ if (!WORDPIECE_MODEL_TYPE.equals(type)) {
+ throw new IllegalArgumentException("Unsupported tokenizer.json: only WordPiece models"
+ + " are supported, found model.type \"" + type + "\"");
+ }
+ final JsonScan.Member prefix = JsonScan.member(model, TOKENIZER_SUBWORD_PREFIX_KEY);
+ if (prefix != null) {
+ final String found = JsonScan.isString(json, prefix) ? JsonScan.stringValue(json, prefix)
+ : json.substring(prefix.valueStart(), prefix.valueEnd());
+ if (!WORDPIECE_SUBWORD_PREFIX.equals(found)) {
+ throw new IllegalArgumentException("Unsupported tokenizer.json:"
+ + " model.continuing_subword_prefix must be \"##\", found " + found);
+ }
+ }
+ final JsonScan.Member vocab = JsonScan.member(model, TOKENIZER_VOCAB_KEY);
+ if (vocab == null) {
+ throw new IllegalArgumentException("Unsupported tokenizer.json: model.vocab is missing");
+ }
+ if (!JsonScan.isObject(json, vocab)) {
+ throw new IllegalArgumentException(
+ "Unsupported tokenizer.json: model.vocab must be an object that maps tokens to ids");
+ }
+ final Map<String, Integer> ids = new HashMap<>();
+ for (JsonScan.Member member : JsonScan.members(json, vocab.valueStart())) {
+ ids.put(member.key(), JsonScan.nonNegativeIntValue(json, member));
+ }
+ return ids;
+ }
+
+ /**
+ * Reads the {@code normalizer.lowercase} setting of a {@code tokenizer.json}.
+ *
+ * @param json The JSON text.
+ * @param document The members of the top-level object.
+ * @return The setting, or {@code null} if there is no {@code normalizer} object or it has no
+ * {@code lowercase} member.
+ * @throws IllegalArgumentException Thrown if the setting is neither {@code true} nor
+ * {@code false}.
+ */
+ private Boolean lowercaseSetting(final String json,
+ final List<JsonScan.Member> document) {
+ final JsonScan.Member normalizer = JsonScan.member(document, TOKENIZER_NORMALIZER_KEY);
+ if (normalizer == null || !JsonScan.isObject(json, normalizer)) {
+ return null;
+ }
+ final JsonScan.Member lowercase = JsonScan.member(
+ JsonScan.members(json, normalizer.valueStart()), TOKENIZER_LOWERCASE_KEY);
+ if (lowercase == null) {
+ return null;
+ }
+ return JsonScan.booleanValue(json, lowercase);
+ }
+
+ /**
+ * Adds the entries of the {@code added_tokens} list of a {@code tokenizer.json} to a
+ * vocabulary. An entry whose token the vocabulary already maps to the same id is left as it
+ * is.
+ *
+ * @param json The JSON text.
+ * @param document The members of the top-level object.
+ * @param vocab The vocabulary read from {@code model.vocab}, extended in place.
+ * @throws IllegalArgumentException Thrown if {@code added_tokens} is present and not a list,
+ * if an entry is not an object with a string {@code content} and a non-negative integer
+ * {@code id}, or if an entry conflicts with the vocabulary: its token has another id
+ * there, or its id belongs to another token.
+ */
+ private void addTokens(final String json, final List<JsonScan.Member> document,
+ final Map<String, Integer> vocab) {
+ final JsonScan.Member addedTokens = JsonScan.member(document, TOKENIZER_ADDED_TOKENS_KEY);
+ if (addedTokens == null) {
+ return;
+ }
+ if (!JsonScan.isArray(json, addedTokens)) {
+ throw new IllegalArgumentException("Unsupported tokenizer.json: added_tokens must be a list");
+ }
+ Map<Integer, String> tokensById = null;
+ for (JsonScan.Member entry : JsonScan.elements(json, addedTokens.valueStart())) {
+ final String where = "added_tokens[" + entry.key() + "]";
+ if (!JsonScan.isObject(json, entry)) {
throw new IllegalArgumentException(
- "Expected one object mapping tokens to integer ids, as in vocab.json: "
- + e.getMessage(), e);
+ "Unsupported tokenizer.json: " + where + " must be an object");
+ }
+ final List<JsonScan.Member> fields = JsonScan.members(json, entry.valueStart());
+ final JsonScan.Member content = JsonScan.member(fields, TOKENIZER_CONTENT_KEY);
+ final JsonScan.Member id = JsonScan.member(fields, TOKENIZER_ID_KEY);
+ if (content == null || id == null) {
+ throw new IllegalArgumentException("Unsupported tokenizer.json: " + where
+ + " must have \"content\" and \"id\"");
+ }
+ final String token = JsonScan.stringValue(json, content);
+ final int tokenId = JsonScan.nonNegativeIntValue(json, id);
+ final Integer known = vocab.get(token);
+ if (known != null) {
+ if (known != tokenId) {
+ throw new IllegalArgumentException("Unsupported tokenizer.json: added token \"" + token
+ + "\" has id " + tokenId + " but the vocabulary maps it to " + known);
+ }
+ continue;
+ }
+ if (tokensById == null) {
+ tokensById = new HashMap<>();
+ for (Map.Entry<String, Integer> e : vocab.entrySet()) {
+ tokensById.put(e.getValue(), e.getKey());
+ }
+ }
+ final String holder = tokensById.putIfAbsent(tokenId, token);
+ if (holder != null) {
+ throw new IllegalArgumentException("Unsupported tokenizer.json: added token \"" + token
+ + "\" has id " + tokenId + " but the vocabulary maps \"" + holder + "\" to it");
}
+ vocab.put(token, tokenId);
}
- return vocab;
}
/**
diff --git a/opennlp-core/opennlp-ml/opennlp-dl/src/main/java/opennlp/dl/JsonScan.java b/opennlp-core/opennlp-ml/opennlp-dl/src/main/java/opennlp/dl/JsonScan.java
index 60aa86e04..b817c1bef 100644
--- a/opennlp-core/opennlp-ml/opennlp-dl/src/main/java/opennlp/dl/JsonScan.java
+++ b/opennlp-core/opennlp-ml/opennlp-dl/src/main/java/opennlp/dl/JsonScan.java
@@ -74,9 +74,6 @@ public final class JsonScan {
/** The number of characters quoted in the message for malformed text. */
private static final int MESSAGE_CONTEXT_LENGTH = 20;
- private static final String TEXT_MUST_NOT_BE_NULL = "text must not be null";
- private static final String KEY_MUST_NOT_BE_NULL = "key must not be null";
-
private JsonScan() {
}
@@ -93,9 +90,7 @@ public final class JsonScan {
record Member(String key, int valueStart, int valueEnd) {
Member {
- if (key == null) {
- throw new IllegalArgumentException(KEY_MUST_NOT_BE_NULL);
- }
+ ParamChecks.requireNonNullArg(key, "key");
if (valueStart < 0 || valueEnd < valueStart) {
throw new IllegalArgumentException(
"value range must be non-negative and ordered: " + valueStart + ".." + valueEnd);
@@ -118,10 +113,8 @@ public final class JsonScan {
* is not an object whose values are all strings. The message names the offset or the key.
*/
public static Map<String, String> stringObject(String text, String key) {
- requireText(text);
- if (key == null) {
- throw new IllegalArgumentException(KEY_MUST_NOT_BE_NULL);
- }
+ ParamChecks.requireNonNullArg(text, "text");
+ ParamChecks.requireNonNullArg(key, "key");
final Map<String, String> entries = new HashMap<>();
if (isBlank(text)) {
return entries;
@@ -149,7 +142,7 @@ public final class JsonScan {
* consist of one object, or is malformed at any position.
*/
static List<Member> document(String text) {
- requireText(text);
+ ParamChecks.requireNonNullArg(text, "text");
final int start = skipWhitespace(text, afterByteOrderMark(text));
expect(text, start, OBJECT_OPEN);
final List<Member> members = new ArrayList<>();
@@ -171,7 +164,7 @@ public final class JsonScan {
* not the offset of an opening brace inside the text, or the object is malformed.
*/
static List<Member> members(String text, int brace) {
- requireText(text);
+ ParamChecks.requireNonNullArg(text, "text");
if (brace < 0 || brace >= text.length()) {
throw new IllegalArgumentException(
"brace must be an offset inside the text, not " + brace);
@@ -182,6 +175,29 @@ public final class JsonScan {
return Collections.unmodifiableList(members);
}
+ /**
+ * Reads the values of the array that starts at an offset. Each value is returned as a member
+ * whose key is the decimal index of the value in the array, so that a message about a value
+ * can name its position.
+ *
+ * @param text The JSON text. Must not be {@code null}.
+ * @param bracket The offset of the opening bracket.
+ * @return The values in document order, an empty list for an empty array.
+ * @throws IllegalArgumentException Thrown if {@code text} is {@code null}, {@code bracket} is
+ * not the offset of an opening bracket inside the text, or the array is malformed.
+ */
+ static List<Member> elements(String text, int bracket) {
+ ParamChecks.requireNonNullArg(text, "text");
+ if (bracket < 0 || bracket >= text.length()) {
+ throw new IllegalArgumentException(
+ "bracket must be an offset inside the text, not " + bracket);
+ }
+ expect(text, bracket, ARRAY_OPEN);
+ final List<Member> elements = new ArrayList<>();
+ endOfContainer(text, bracket, elements);
+ return Collections.unmodifiableList(elements);
+ }
+
/**
* Finds the member with a key in a list of members. A later member with the same key
* counts, as with a map that is filled in document order.
@@ -193,9 +209,7 @@ public final class JsonScan {
*/
static Member member(List<Member> members, String key) {
ParamChecks.requireNonNullArg(members, "members");
- if (key == null) {
- throw new IllegalArgumentException(KEY_MUST_NOT_BE_NULL);
- }
+ ParamChecks.requireNonNullArg(key, "key");
Member found = null;
for (Member m : members) {
if (m.key().equals(key)) {
@@ -219,6 +233,34 @@ public final class JsonScan {
return text.charAt(member.valueStart()) == OBJECT_OPEN;
}
+ /**
+ * Tests whether the value of a member is an array.
+ *
+ * @param text The JSON text. Must not be {@code null}.
+ * @param member The member, inside the text. Must not be {@code null}.
+ * @return {@code true} if the value starts with an opening bracket.
+ * @throws IllegalArgumentException Thrown if an argument is {@code null} or the member lies
+ * outside the text.
+ */
+ static boolean isArray(String text, Member member) {
+ requireMember(text, member);
+ return text.charAt(member.valueStart()) == ARRAY_OPEN;
+ }
+
+ /**
+ * Tests whether the value of a member is a string.
+ *
+ * @param text The JSON text. Must not be {@code null}.
+ * @param member The member, inside the text. Must not be {@code null}.
+ * @return {@code true} if the value starts with a quote.
+ * @throws IllegalArgumentException Thrown if an argument is {@code null} or the member lies
+ * outside the text.
+ */
+ static boolean isString(String text, Member member) {
+ requireMember(text, member);
+ return text.charAt(member.valueStart()) == QUOTE;
+ }
+
/**
* Reads the value of a member as a string.
*
@@ -231,12 +273,33 @@ public final class JsonScan {
static String stringValue(String text, Member member) {
requireMember(text, member);
if (text.charAt(member.valueStart()) != QUOTE) {
- throw new IllegalArgumentException("Value of \"" + member.key() + "\" must be a string: "
- + text.substring(member.valueStart(), member.valueEnd()));
+ throw badValue(text, member, "a string");
}
return unescape(text, member.valueStart() + 1, member.valueEnd() - 1);
}
+ /**
+ * Reads the value of a member as a boolean.
+ *
+ * @param text The JSON text. Must not be {@code null}.
+ * @param member The member, inside the text. Must not be {@code null}.
+ * @return {@code true} for the literal {@code true}, {@code false} for {@code false}.
+ * @throws IllegalArgumentException Thrown if an argument is {@code null}, the member lies
+ * outside the text, or the value is neither literal.
+ */
+ static boolean booleanValue(String text, Member member) {
+ requireMember(text, member);
+ final int start = member.valueStart();
+ final int end = member.valueEnd();
+ if (text.startsWith(TRUE, start) && end - start == TRUE.length()) {
+ return true;
+ }
+ if (text.startsWith(FALSE, start) && end - start == FALSE.length()) {
+ return false;
+ }
+ throw badValue(text, member, TRUE + " or " + FALSE);
+ }
+
/**
* Reads the value of a member as a non-negative integer: a run of ASCII digits with no sign,
* fraction, or exponent that fits into an {@code int}.
@@ -252,17 +315,31 @@ public final class JsonScan {
final int start = member.valueStart();
final int end = member.valueEnd();
if (StringUtil.endOfAsciiDigits(text, start) != end) {
- throw new IllegalArgumentException("Value of \"" + member.key()
- + "\" must be a non-negative integer: " + text.substring(start, end));
+ throw badValue(text, member, "a non-negative integer");
}
try {
return Integer.parseInt(text, start, end, 10);
} catch (NumberFormatException e) {
- throw new IllegalArgumentException("Value of \"" + member.key()
- + "\" does not fit into an int: " + text.substring(start, end), e);
+ throw new IllegalArgumentException("Value of \"" + member.key() + "\" at offset " + start
+ + " does not fit into an int: " + text.substring(start, end), e);
}
}
+ /**
+ * Builds the exception for a member whose value has the wrong type. The message names the
+ * key, the offset of the value, and the value, so the entry can be found in a large file.
+ *
+ * @param text The JSON text.
+ * @param member The member, inside the text.
+ * @param expected What the value must be.
+ * @return The exception to throw.
+ */
+ private static IllegalArgumentException badValue(String text, Member member, String expected) {
+ return new IllegalArgumentException("Value of \"" + member.key() + "\" at offset "
+ + member.valueStart() + " must be " + expected + ": "
+ + text.substring(member.valueStart(), member.valueEnd()));
+ }
+
/**
* Tests whether the text holds nothing but JSON whitespace after an optional byte order mark.
*
@@ -405,16 +482,18 @@ public final class JsonScan {
/**
* Scans an object or an array, and every value nested in it, without recursion. When a sink
- * is given, the container is an object and its direct members are added to the sink.
+ * is given, the direct members of the object, or the values of the array under their decimal
+ * index as the key, are added to the sink.
*
* @param text The JSON text.
* @param open The offset of the opening brace or bracket.
- * @param sink The list to add the members of the object to, or {@code null} to only find
- * the end.
+ * @param sink The list to add the members of the object or the values of the array to, or
+ * {@code null} to only find the end.
* @return The offset after the closing brace or bracket.
* @throws IllegalArgumentException Thrown if the container or a value in it is malformed.
*/
private static int endOfContainer(String text, int open, List<Member> sink) {
+ final boolean object = text.charAt(open) == OBJECT_OPEN;
final Deque<Boolean> objects = new ArrayDeque<>();
String key = null;
int valueStart = 0;
@@ -457,7 +536,7 @@ public final class JsonScan {
elementExpected = false;
} else {
if (sink != null && objects.size() == 1) {
- sink.add(new Member(key, valueStart, i));
+ sink.add(new Member(object ? key : Integer.toString(sink.size()), valueStart, i));
}
i = skipWhitespace(text, i);
if (isCloser(text, i, objects.peek())) {
@@ -618,18 +697,6 @@ public final class JsonScan {
return true;
}
- /**
- * Checks the text argument.
- *
- * @param text The JSON text.
- * @throws IllegalArgumentException Thrown if it is {@code null}.
- */
- private static void requireText(String text) {
- if (text == null) {
- throw new IllegalArgumentException(TEXT_MUST_NOT_BE_NULL);
- }
- }
-
/**
* Checks a member against the text it must lie in.
*
@@ -639,7 +706,7 @@ public final class JsonScan {
* range reaches beyond the text.
*/
private static void requireMember(String text, Member member) {
- requireText(text);
+ ParamChecks.requireNonNullArg(text, "text");
ParamChecks.requireNonNullArg(member, "member");
if (member.valueEnd() > text.length() || member.valueStart() >= text.length()) {
throw new IllegalArgumentException("member must lie inside the text: " + member);
diff --git a/opennlp-core/opennlp-ml/opennlp-dl/src/main/java/opennlp/dl/doccat/DocumentCategorizerDL.java b/opennlp-core/opennlp-ml/opennlp-dl/src/main/java/opennlp/dl/doccat/DocumentCategorizerDL.java
index cea40e01f..4082c122e 100644
--- a/opennlp-core/opennlp-ml/opennlp-dl/src/main/java/opennlp/dl/doccat/DocumentCategorizerDL.java
+++ b/opennlp-core/opennlp-ml/opennlp-dl/src/main/java/opennlp/dl/doccat/DocumentCategorizerDL.java
@@ -119,7 +119,8 @@ public class DocumentCategorizerDL extends AbstractDL implements DocumentCategor
* @param inferenceOptions {@link InferenceOptions} to control the inference.
* @throws OrtException Thrown if the {@code model} cannot be loaded.
* @throws IOException Thrown if errors occurred loading the {@code model} or {@code vocabulary}.
- * @throws InvalidFormatException Thrown if a JSON {@code vocabulary} is malformed.
+ * @throws InvalidFormatException Thrown if a JSON {@code vocabulary} is malformed, has an
+ * unsupported layout, or sets a lower casing that disagrees with {@code inferenceOptions}.
*/
public DocumentCategorizerDL(File model, File vocabulary, Map<Integer, String> categories,
ClassificationScoringStrategy classificationScoringStrategy,
@@ -156,9 +157,10 @@ public class DocumentCategorizerDL extends AbstractDL implements DocumentCategor
* @throws OrtException Thrown if the {@code model} cannot be loaded.
* @throws IOException Thrown if errors occurred loading the {@code model}, the
* {@code vocabulary}, or the {@code config}.
- * @throws InvalidFormatException Thrown if a JSON {@code vocabulary} is malformed, or if
- * {@code config} is not well-formed JSON, its {@code id2label} member does not map keys
- * to strings, or a key is not an integer.
+ * @throws InvalidFormatException Thrown if a JSON {@code vocabulary} is malformed, has an
+ * unsupported layout, or sets a lower casing that disagrees with {@code inferenceOptions},
+ * or if {@code config} is not well-formed JSON, its {@code id2label} member does not map
+ * keys to strings, or a key is not an integer.
*/
public DocumentCategorizerDL(File model, File vocabulary, File config,
ClassificationScoringStrategy classificationScoringStrategy,
diff --git a/opennlp-core/opennlp-ml/opennlp-dl/src/main/java/opennlp/dl/namefinder/NameFinderDL.java b/opennlp-core/opennlp-ml/opennlp-dl/src/main/java/opennlp/dl/namefinder/NameFinderDL.java
index 9f38271b9..b9b0baf33 100644
--- a/opennlp-core/opennlp-ml/opennlp-dl/src/main/java/opennlp/dl/namefinder/NameFinderDL.java
+++ b/opennlp-core/opennlp-ml/opennlp-dl/src/main/java/opennlp/dl/namefinder/NameFinderDL.java
@@ -127,7 +127,8 @@ public class NameFinderDL extends AbstractDL implements OffsetMappingNameFinder
* @throws OrtException Thrown if the {@code model} cannot be loaded.
* @throws IOException Thrown if errors occurred loading the {@code model} or {@code vocabulary}.
* @throws opennlp.tools.util.InvalidFormatException Thrown if a JSON {@code vocabulary}
- * is malformed.
+ * is malformed, has an unsupported layout, or sets a lower casing that disagrees with
+ * the lower casing the component is configured with.
* @throws IllegalArgumentException Thrown if {@code inferenceOptions}, {@code ids2Labels}, or
* the sentence detector is {@code null}.
*/
@@ -152,7 +153,8 @@ public class NameFinderDL extends AbstractDL implements OffsetMappingNameFinder
* @throws OrtException Thrown if the {@code model} cannot be loaded.
* @throws IOException Thrown if errors occurred loading the {@code model} or {@code vocabulary}.
* @throws opennlp.tools.util.InvalidFormatException Thrown if a JSON {@code vocabulary}
- * is malformed.
+ * is malformed, has an unsupported layout, or sets a lower casing that disagrees with
+ * the lower casing the component is configured with.
* @throws IllegalArgumentException Thrown if {@code inferenceOptions}, {@code ids2Labels}, or
* the sentence detector is {@code null}.
*/
diff --git a/opennlp-core/opennlp-ml/opennlp-dl/src/main/java/opennlp/dl/vectors/SentenceVectorsDL.java b/opennlp-core/opennlp-ml/opennlp-dl/src/main/java/opennlp/dl/vectors/SentenceVectorsDL.java
index 21f162e07..8f6c080ff 100644
--- a/opennlp-core/opennlp-ml/opennlp-dl/src/main/java/opennlp/dl/vectors/SentenceVectorsDL.java
+++ b/opennlp-core/opennlp-ml/opennlp-dl/src/main/java/opennlp/dl/vectors/SentenceVectorsDL.java
@@ -105,7 +105,8 @@ public class SentenceVectorsDL extends AbstractDL implements TextEmbedder {
* @throws OrtException Thrown if the {@code model} cannot be loaded.
* @throws IOException Thrown if errors occurred loading the {@code model} or {@code vocabulary}.
* @throws opennlp.tools.util.InvalidFormatException Thrown if a JSON {@code vocabulary}
- * is malformed.
+ * is malformed, has an unsupported layout, or sets a lower casing that disagrees with
+ * the lower casing the component is configured with.
*/
public SentenceVectorsDL(final File model, final File vocabulary)
throws OrtException, IOException {
@@ -127,7 +128,8 @@ public class SentenceVectorsDL extends AbstractDL implements TextEmbedder {
* @throws OrtException Thrown if the {@code model} cannot be loaded.
* @throws IOException Thrown if errors occurred loading the {@code model} or {@code vocabulary}.
* @throws opennlp.tools.util.InvalidFormatException Thrown if a JSON {@code vocabulary}
- * is malformed.
+ * is malformed, has an unsupported layout, or sets a lower casing that disagrees with
+ * the lower casing the component is configured with.
*/
public SentenceVectorsDL(final File model, final File vocabulary, final boolean lowerCase)
throws OrtException, IOException {
diff --git a/opennlp-core/opennlp-ml/opennlp-dl/src/test/java/opennlp/dl/JsonScanTest.java b/opennlp-core/opennlp-ml/opennlp-dl/src/test/java/opennlp/dl/JsonScanTest.java
index 7704f2053..3e30ff517 100644
--- a/opennlp-core/opennlp-ml/opennlp-dl/src/test/java/opennlp/dl/JsonScanTest.java
+++ b/opennlp-core/opennlp-ml/opennlp-dl/src/test/java/opennlp/dl/JsonScanTest.java
@@ -124,6 +124,59 @@ public class JsonScanTest {
() -> JsonScan.members(text, x.valueStart() + 1));
}
+ @Test
+ void testElementsReadsTheValuesOfAnArrayUnderTheirIndex() {
+ final String text = "{\"x\": [1, \"a\", {\"b\": [2]}, [], null], \"y\": [], \"z\": {}}";
+ final List<Member> members = JsonScan.document(text);
+ final Member x = JsonScan.member(members, "x");
+ Assertions.assertTrue(JsonScan.isArray(text, x));
+ Assertions.assertFalse(JsonScan.isArray(text, JsonScan.member(members, "z")));
+ Assertions.assertEquals(List.of("0=1", "1=\"a\"", "2={\"b\": [2]}", "3=[]", "4=null"),
+ render(text, JsonScan.elements(text, x.valueStart())));
+ Assertions.assertEquals(List.of(),
+ JsonScan.elements(text, JsonScan.member(members, "y").valueStart()));
+ Assertions.assertThrows(IllegalArgumentException.class,
+ () -> JsonScan.elements(text, JsonScan.member(members, "z").valueStart()));
+ }
+
+ @ParameterizedTest
+ @ValueSource(strings = {"[1,]", "[,1]", "[1 2]", "[1", "[\"a]", "[{]", "[1}"})
+ void testElementsRejectsAMalformedArray(String text) {
+ Assertions.assertThrows(IllegalArgumentException.class, () -> JsonScan.elements(text, 0));
+ }
+
+ @Test
+ void testElementsRejectsNullAndAnOffsetOutsideTheText() {
+ Assertions.assertThrows(IllegalArgumentException.class, () -> JsonScan.elements(null, 0));
+ Assertions.assertThrows(IllegalArgumentException.class, () -> JsonScan.elements("[]", -1));
+ Assertions.assertThrows(IllegalArgumentException.class, () -> JsonScan.elements("[]", 2));
+ }
+
+ @Test
+ void testBooleanValueReadsTheTwoLiterals() {
+ final String text = "{\"a\": true, \"b\":false, \"c\": [true], \"d\": \"true\", \"e\": null,"
+ + " \"f\": 1, \"g\": {}}";
+ final List<Member> members = JsonScan.document(text);
+ Assertions.assertTrue(JsonScan.booleanValue(text, JsonScan.member(members, "a")));
+ Assertions.assertFalse(JsonScan.booleanValue(text, JsonScan.member(members, "b")));
+ for (String key : List.of("c", "d", "e", "f", "g")) {
+ final IllegalArgumentException e = Assertions.assertThrows(IllegalArgumentException.class,
+ () -> JsonScan.booleanValue(text, JsonScan.member(members, key)), key);
+ Assertions.assertTrue(e.getMessage().contains("\"" + key + "\""), e.getMessage());
+ }
+ }
+
+ @Test
+ void testIsStringTellsStringValuesFromOthers() {
+ final String text = "{\"a\": \"x\", \"b\": \"\", \"c\": 1, \"d\": {}, \"e\": [\"x\"], \"f\": null}";
+ final List<Member> members = JsonScan.document(text);
+ Assertions.assertTrue(JsonScan.isString(text, JsonScan.member(members, "a")));
+ Assertions.assertTrue(JsonScan.isString(text, JsonScan.member(members, "b")));
+ for (String key : List.of("c", "d", "e", "f")) {
+ Assertions.assertFalse(JsonScan.isString(text, JsonScan.member(members, key)), key);
+ }
+ }
+
@Test
void testMemberFindsTheLastWithAKey() {
final String text = "{\"a\": 1, \"b\": 2, \"a\": 3}";
@@ -438,6 +491,10 @@ public class JsonScanTest {
void testValueReadersRejectAMemberOutsideTheText() {
final Member outside = new Member("k", 5, 9);
Assertions.assertThrows(IllegalArgumentException.class, () -> JsonScan.isObject("{}", outside));
+ Assertions.assertThrows(IllegalArgumentException.class, () -> JsonScan.isString("{}", outside));
+ Assertions.assertThrows(IllegalArgumentException.class, () -> JsonScan.isArray("{}", outside));
+ Assertions.assertThrows(IllegalArgumentException.class,
+ () -> JsonScan.booleanValue("{}", outside));
Assertions.assertThrows(IllegalArgumentException.class, () -> JsonScan.stringValue("{}", outside));
Assertions.assertThrows(IllegalArgumentException.class,
() -> JsonScan.nonNegativeIntValue("{}", outside));
diff --git a/opennlp-core/opennlp-ml/opennlp-dl/src/test/java/opennlp/dl/LoadVocabTest.java b/opennlp-core/opennlp-ml/opennlp-dl/src/test/java/opennlp/dl/LoadVocabTest.java
index 8ace1843d..f28e690e1 100644
--- a/opennlp-core/opennlp-ml/opennlp-dl/src/test/java/opennlp/dl/LoadVocabTest.java
+++ b/opennlp-core/opennlp-ml/opennlp-dl/src/test/java/opennlp/dl/LoadVocabTest.java
@@ -23,6 +23,7 @@ import java.io.InputStream;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.StandardCopyOption;
+import java.util.List;
import java.util.Map;
import java.util.Objects;
import java.util.stream.Stream;
@@ -31,11 +32,14 @@ import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.io.TempDir;
import org.junit.jupiter.params.ParameterizedTest;
import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.CsvSource;
import org.junit.jupiter.params.provider.MethodSource;
import org.junit.jupiter.params.provider.ValueSource;
+import opennlp.tools.tokenize.SubwordPiece;
import opennlp.tools.util.InvalidFormatException;
+import static org.junit.jupiter.api.Assertions.assertArrayEquals;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertNotNull;
import static org.junit.jupiter.api.Assertions.assertThrows;
@@ -43,6 +47,38 @@ import static org.junit.jupiter.api.Assertions.assertTrue;
public class LoadVocabTest {
+ /** A component without a model, to read vocabularies through. */
+ private static final AbstractDL DL =
+ new ModelFreeDL(Map.of("[CLS]", 0, "[SEP]", 1, "[UNK]", 2), true);
+
+ /** A flat vocab.json. */
+ private static final String JSON_VOCAB = "{\"[PAD]\":0,\"hello\":1,\"\\u0120x\":2}";
+
+ /** A tokenizer.json of a WordPiece model whose model.vocab holds {@link #TOKENIZER_VOCAB}. */
+ private static final String TOKENIZER_JSON = """
+ {
+ "version": "1.0",
+ "truncation": null,
+ "added_tokens": [
+ {"id": 0, "content": "[PAD]", "special": true},
+ {"id": 1, "content": "[UNK]", "special": true}
+ ],
+ "normalizer": {"type": "BertNormalizer", "lowercase": true},
+ "model": {
+ "type": "WordPiece",
+ "unk_token": "[UNK]",
+ "vocab_size": 5,
+ "vocab": {"[PAD]": 0, "[UNK]": 1, "hello": 2, "##ing": 3, "\\u0120x": 4}
+ }
+ }
+ """;
+
+ private static final Map<String, Integer> TOKENIZER_VOCAB =
+ Map.of("[PAD]", 0, "[UNK]", 1, "hello", 2, "##ing", 3, "\u0120x", 4);
+
+ private static final String UNIGRAM_TOKENIZER_JSON = "{\"version\": \"1.0\", \"model\": {\"type\": "
+ + "\"Unigram\", \"unk_id\": 0, \"vocab\": [[\"<unk>\", 0.0], [\"a\", -1.5]]}}";
+
@TempDir
private Path tempDir;
@@ -68,7 +104,7 @@ public class LoadVocabTest {
@Test
void testLoadPlainTextVocab() throws IOException {
- final Map<String, Integer> vocab = AbstractDL.loadVocabFile(getResource("vocab-plain.txt"));
+ final Map<String, Integer> vocab = DL.loadVocab(getResource("vocab-plain.txt"));
assertNotNull(vocab);
assertEquals(6, vocab.size());
@@ -82,7 +118,7 @@ public class LoadVocabTest {
@Test
void testLoadJsonVocab() throws IOException {
- final Map<String, Integer> vocab = AbstractDL.loadVocabFile(getResource("vocab.json"));
+ final Map<String, Integer> vocab = DL.loadVocab(getResource("vocab.json"));
assertNotNull(vocab);
assertEquals(6, vocab.size());
@@ -98,7 +134,7 @@ public class LoadVocabTest {
void testJsonVocabWithEscapedCharacters() throws IOException {
final File tempFile = vocabFile("vocab-escaped.json", "{\"hello\\\"world\": 0, \"back\\\\slash\": 1}");
- final Map<String, Integer> vocab = AbstractDL.loadVocabFile(tempFile);
+ final Map<String, Integer> vocab = DL.loadVocab(tempFile);
assertNotNull(vocab);
assertEquals(2, vocab.size());
@@ -111,7 +147,7 @@ public class LoadVocabTest {
final File tempFile = vocabFile("vocab-unicode.json",
"{\"\\u0120token\": 0, \"line\\rbreak\": 1, \"form\\ffeed\": 2}");
- final Map<String, Integer> vocab = AbstractDL.loadVocabFile(tempFile);
+ final Map<String, Integer> vocab = DL.loadVocab(tempFile);
assertNotNull(vocab);
assertEquals(3, vocab.size());
@@ -120,11 +156,100 @@ public class LoadVocabTest {
assertEquals(2, vocab.get("form\ffeed"));
}
+ @Test
+ void testLoadTokenizerJsonVocabFile() throws IOException {
+ final File file = getResource("tokenizer.json");
+
+ final Map<String, Integer> vocab = DL.loadVocab(file);
+
+ // the 24 entries of model.vocab, and the added token opennlp that model.vocab lacks
+ assertEquals(25, vocab.size());
+ assertEquals(0, vocab.get("[PAD]"));
+ assertEquals(1, vocab.get("[UNK]"));
+ assertEquals(2, vocab.get("[CLS]"));
+ assertEquals(3, vocab.get("[SEP]"));
+ assertEquals(6, vocab.get("hello"));
+ assertEquals(9, vocab.get("##ing"));
+ assertEquals(23, vocab.get("##c"));
+ assertEquals(24, vocab.get("opennlp"));
+ assertEquals(Boolean.TRUE, DL.readVocabFile(file).lowercase());
+ }
+
+ /**
+ * Sentences and the pieces and ids the fixture gives them, computed by hand from
+ * {@code tokenizer.json}: the text is lower cased and stripped of accents as the normalizer
+ * says, split at whitespace and punctuation, and each word is segmented into the longest
+ * pieces of the vocabulary, all but the first carrying the {@code ##} prefix; a word that
+ * cannot be segmented is {@code [UNK]}, and {@code [CLS]} and {@code [SEP]} frame the text.
+ */
+ static Stream<Arguments> tokenizerJsonEncodings() {
+ return Stream.of(
+ Arguments.of("Hello world!",
+ List.of("[CLS]", "hello", "world", "!", "[SEP]"),
+ new long[] {2, 6, 7, 19, 3}),
+ // un ##play ##able; the added token opennlp is longer than the piece open
+ Arguments.of("The unplayable OpenNLP plays, played.",
+ List.of("[CLS]", "the", "un", "##play", "##able", "opennlp", "play", "##s", ",",
+ "play", "##ed", ".", "[SEP]"),
+ new long[] {2, 5, 12, 13, 14, 24, 8, 10, 18, 8, 11, 17, 3}),
+ // c is only a continuation piece, so cab cannot start
+ Arguments.of("abc cab",
+ List.of("[CLS]", "a", "##b", "##c", "[UNK]", "[SEP]"),
+ new long[] {2, 20, 22, 23, 1, 3}),
+ // lowercase: true strips accents as well
+ Arguments.of("H\u00e9llo W\u00f6rld!",
+ List.of("[CLS]", "hello", "world", "!", "[SEP]"),
+ new long[] {2, 6, 7, 19, 3}),
+ Arguments.of("",
+ List.of("[CLS]", "[SEP]"),
+ new long[] {2, 3}));
+ }
+
+ @ParameterizedTest
+ @MethodSource("tokenizerJsonEncodings")
+ void testTokenizerJsonVocabularyEncodesAsComputedByHand(String text, List<String> pieces,
+ long[] ids) throws IOException {
+ final Map<String, Integer> vocab = DL.loadVocab(getResource("tokenizer.json"));
+ final List<SubwordPiece> encoded = AbstractDL.createWordpieceEncoder(vocab, true).encode(text);
+
+ assertEquals(pieces, encoded.stream().map(SubwordPiece::piece).toList());
+ assertArrayEquals(ids, encoded.stream().mapToLong(SubwordPiece::id).toArray());
+ }
+
+ static Stream<Arguments> unsupportedTokenizerFiles() {
+ return Stream.of(
+ Arguments.of("tokenizer-unigram.json", UNIGRAM_TOKENIZER_JSON, "\"Unigram\""),
+ Arguments.of("tokenizer-bpe.json", "{\"model\": {\"type\": \"BPE\", \"vocab\": {\"a\": 0},"
+ + " \"merges\": []}}", "\"BPE\""),
+ Arguments.of("tokenizer-prefix.json", "{\"model\": {\"type\": \"WordPiece\","
+ + " \"continuing_subword_prefix\": \"@@\", \"vocab\": {\"a\": 0}}}", "continuing_subword_prefix"),
+ Arguments.of("tokenizer-conflict.json", "{\"added_tokens\": [{\"id\": 1, \"content\": \"a\"}],"
+ + " \"model\": {\"type\": \"WordPiece\", \"vocab\": {\"a\": 0}}}", "added token \"a\""));
+ }
+
+ @ParameterizedTest
+ @MethodSource("unsupportedTokenizerFiles")
+ void testUnsupportedTokenizerJsonFileIsRejectedNamingTheFile(String name, String json,
+ String reason) throws IOException {
+ final File tempFile = vocabFile(name, json);
+
+ final InvalidFormatException e = assertThrows(InvalidFormatException.class,
+ () -> DL.loadVocab(tempFile));
+ assertTrue(e.getMessage().contains(tempFile.getName()), e.getMessage());
+ assertTrue(e.getMessage().contains(reason), e.getMessage());
+ }
+
+ @Test
+ void testTokenizerJsonFileWithAByteOrderMarkIsReadAsJson() throws IOException {
+ final File tempFile = vocabFile("tokenizer-bom.json", "\uFEFF" + TOKENIZER_JSON);
+
+ assertEquals(TOKENIZER_VOCAB, DL.loadVocab(tempFile));
+ }
@Test
void testJsonAndPlainTextVocabProduceSameResult() throws IOException {
- final Map<String, Integer> plainVocab = AbstractDL.loadVocabFile(getResource("vocab-plain.txt"));
- final Map<String, Integer> jsonVocab = AbstractDL.loadVocabFile(getResource("vocab.json"));
+ final Map<String, Integer> plainVocab = DL.loadVocab(getResource("vocab-plain.txt"));
+ final Map<String, Integer> jsonVocab = DL.loadVocab(getResource("vocab.json"));
assertEquals(plainVocab, jsonVocab);
}
@@ -135,7 +260,6 @@ public class LoadVocabTest {
Arguments.of("{\"a\": 1, \"b\": 2}", Map.of("a", 1, "b", 2)),
Arguments.of(" \t\r\n{\"a\"\n:\r\n 3\t}\n", Map.of("a", 3)),
Arguments.of("{\"a\":0}", Map.of("a", 0)),
- Arguments.of("{\"model\":0,\"vocab\":1}", Map.of("model", 0, "vocab", 1)),
Arguments.of("{\"a\": 2147483647}", Map.of("a", Integer.MAX_VALUE)),
Arguments.of("{\"a\\\"b\": 1}", Map.of("a\"b", 1)),
Arguments.of("{\"a\\\\\": 1}", Map.of("a\\", 1)),
@@ -146,13 +270,15 @@ public class LoadVocabTest {
Arguments.of("{\"\": 1}", Map.of("", 1)),
// a later entry for the same token, written or escaped, overwrites the earlier one
Arguments.of("{\"a\": 1, \"a\": 2}", Map.of("a", 2)),
- Arguments.of("{\"a\": 1, \"\\u0061\": 2}", Map.of("a", 2)));
+ Arguments.of("{\"a\": 1, \"\\u0061\": 2}", Map.of("a", 2)),
+ // tokens named model and vocab with integer ids are a flat vocabulary
+ Arguments.of("{\"model\":0,\"vocab\":1}", Map.of("model", 0, "vocab", 1)));
}
@ParameterizedTest
@MethodSource("jsonVocabs")
void testLoadJsonVocab(String json, Map<String, Integer> expected) {
- assertEquals(expected, AbstractDL.loadJsonVocab(json));
+ assertEquals(expected, DL.readJsonVocab(json).ids());
}
@ParameterizedTest
@@ -170,54 +296,267 @@ public class LoadVocabTest {
"{\"a\\q\": 1}", "{\"\\\uD83D\uDE00\": 2}", "{\"\\u12\": 3}", "{\"\\u+123\": 3}",
"{\"a\\\nb\": 1}"})
void testLoadJsonVocabRejectsMalformedText(String json) {
- assertThrows(IllegalArgumentException.class, () -> AbstractDL.loadJsonVocab(json));
+ assertThrows(IllegalArgumentException.class, () -> DL.readJsonVocab(json).ids());
}
@Test
void testLoadJsonVocabRejectsNull() {
- assertThrows(IllegalArgumentException.class, () -> AbstractDL.loadJsonVocab(null));
+ assertThrows(IllegalArgumentException.class, () -> DL.readJsonVocab(null).ids());
}
@Test
void testLoadJsonVocabMessageNamesTheToken() {
final IllegalArgumentException e = assertThrows(IllegalArgumentException.class,
- () -> AbstractDL.loadJsonVocab("{\"ok\": 1, \"bad\": -1}"));
+ () -> DL.readJsonVocab("{\"ok\": 1, \"bad\": -1}").ids());
assertTrue(e.getMessage().contains("\"bad\""), e.getMessage());
}
+ @ParameterizedTest
+ @CsvSource(delimiter = '|', value = {
+ // an int-valued top-level member makes the top-level object the vocabulary
+ "{\"a\": 1, \"model\": {\"vocab\": {\"b\": 2}}} | \"model\"",
+ // model missing or not an object
+ "{\"model\": \"x\"} | \"model\"", "{\"vocab\": {\"a\": 1}} | \"vocab\"",
+ // model.type missing or not a string: the file is not a tokenizer.json
+ "{\"model\": {\"vocab\": {\"a\": 0}}} | \"model\"",
+ "{\"a\": -1, \"model\": {\"vocab\": {\"b\": 2}}} | \"a\"",
+ "{\"model\": {\"type\": 5, \"vocab\": {\"a\": 0}}} | \"model\"",
+ // a config.json or a tokenizer_config.json handed over in place of the vocabulary
+ "{\"architectures\": [\"BertModel\"], \"hidden_size\": 768} | \"architectures\"",
+ "{\"do_lower_case\": true, \"tokenizer_class\": \"BertTokenizer\"} | \"do_lower_case\""})
+ void testLoadJsonVocabRejectsOtherLayoutsNamingTheFirstMemberThatIsNoId(String json,
+ String named) {
+ final IllegalArgumentException e = assertThrows(IllegalArgumentException.class,
+ () -> DL.readJsonVocab(json).ids());
+ assertTrue(e.getMessage().contains(named), e.getMessage());
+ }
+
+ @Test
+ void testLoadJsonVocabNamesTheOffsetOfAMalformedAddedToken() {
+ final String json = "{\"added_tokens\": [{\"id\": 0, \"content\": \"a\"},"
+ + " {\"id\": \"7\", \"content\": \"b\"}],"
+ + " \"model\": {\"type\": \"WordPiece\", \"vocab\": {\"a\": 0}}}";
+ final IllegalArgumentException e = assertThrows(IllegalArgumentException.class,
+ () -> DL.readJsonVocab(json).ids());
+ assertTrue(e.getMessage().contains("\"id\" at offset " + json.indexOf("\"7\"")),
+ e.getMessage());
+ }
+
+ @ParameterizedTest
+ @CsvSource(delimiter = '|', value = {
+ // model.vocab missing or not an object
+ "{\"model\": {\"type\": \"WordPiece\"}} | model.vocab",
+ "{\"model\": {\"type\": \"WordPiece\", \"vocab\": [\"a\"]}} | model.vocab",
+ "{\"model\": {\"type\": \"WordPiece\", \"vocab\": \"a\"}} | model.vocab",
+ // the same strict rules apply inside model.vocab, and the message names the token
+ "{\"model\": {\"type\": \"WordPiece\", \"vocab\": {\"a\": \"1\"}}} | \"a\"",
+ "{\"model\": {\"type\": \"WordPiece\", \"vocab\": {\"a\": -1}}} | \"a\""})
+ void testLoadJsonVocabRejectsAWordPieceModelWithoutAUsableVocab(String json, String named) {
+ final IllegalArgumentException e = assertThrows(IllegalArgumentException.class,
+ () -> DL.readJsonVocab(json).ids());
+ assertTrue(e.getMessage().contains(named), e.getMessage());
+ }
+
+ static Stream<Arguments> addedTokens() {
+ final String model = "\"model\": {\"type\": \"WordPiece\", \"vocab\": {\"a\": 0, \"b\": 2}}";
+ return Stream.of(
+ // a token absent from model.vocab is added with its id, before or after the model
+ Arguments.of("{\"added_tokens\": [{\"id\": 5, \"content\": \"[NEW]\", \"special\": true}], "
+ + model + "}", Map.of("a", 0, "b", 2, "[NEW]", 5)),
+ Arguments.of("{" + model + ", \"added_tokens\": [{\"content\": \"c\", \"id\": 1}]}",
+ Map.of("a", 0, "b", 2, "c", 1)),
+ // a token listed with the id model.vocab gives it is an entry once
+ Arguments.of("{\"added_tokens\": [{\"id\": 0, \"content\": \"a\"}, {\"id\": 2, \"content\": \"b\"}],"
+ + model + "}", Map.of("a", 0, "b", 2)),
+ // the same added token twice with one id, and content with escapes
+ Arguments.of("{\"added_tokens\": [{\"id\": 3, \"content\": \"\\u0120x\"},"
+ + " {\"id\": 3, \"content\": \"\\u0120x\"}], " + model + "}",
+ Map.of("a", 0, "b", 2, "\u0120x", 3)),
+ // an empty list adds nothing
+ Arguments.of("{\"added_tokens\": [], " + model + "}", Map.of("a", 0, "b", 2)),
+ // a later added_tokens list replaces an earlier one, as a later member does
+ Arguments.of("{\"added_tokens\": [{\"id\": 9, \"content\": \"x\"}], " + model
+ + ", \"added_tokens\": [{\"id\": 8, \"content\": \"y\"}]}", Map.of("a", 0, "b", 2, "y", 8)));
+ }
+
+ @ParameterizedTest
+ @MethodSource("addedTokens")
+ void testLoadJsonVocabAddsTheAddedTokensAbsentFromTheModelVocab(String json,
+ Map<String, Integer> expected) {
+ assertEquals(expected, DL.readJsonVocab(json).ids());
+ }
+
+ @Test
+ void testLoadJsonVocabRejectsAnAddedTokenWhoseIdDiffersFromTheModelVocab() {
+ final IllegalArgumentException e = assertThrows(IllegalArgumentException.class,
+ () -> DL.readJsonVocab("{\"added_tokens\": [{\"id\": 7, \"content\": \"b\"}],"
+ + " \"model\": {\"type\": \"WordPiece\", \"vocab\": {\"a\": 0, \"b\": 2}}}").ids());
+ assertTrue(e.getMessage().contains("\"b\""), e.getMessage());
+ assertTrue(e.getMessage().contains("7"), e.getMessage());
+ assertTrue(e.getMessage().contains("2"), e.getMessage());
+ }
+
@ParameterizedTest
@ValueSource(strings = {
- "{\"version\":\"1.0\",\"model\":{\"type\":\"WordPiece\",\"vocab\":{\"a\":0}}}",
- "{\"version\":\"1.0\",\"model\":{\"type\":\"BPE\",\"vocab\":{\"a\":0}}}",
- "{\"version\":\"1.0\",\"model\":{\"type\":\"Unigram\",\"vocab\":[[\"a\",0.0]]}}",
- "{\"model\":{\"vocab\":{\"a\":0}}}"})
- void testTokenizerFileIsRejectedWithTheExpectedVocabularyLayout(String json) throws IOException {
- final File tempFile = vocabFile("tokenizer-unsupported.json", json);
+ // the id is that of a token of model.vocab
+ "{\"added_tokens\": [{\"id\": 2, \"content\": \"[NEW]\"}],"
+ + " \"model\": {\"type\": \"WordPiece\", \"vocab\": {\"a\": 0, \"b\": 2}}}",
+ // the id is that of an earlier added token
+ "{\"added_tokens\": [{\"id\": 5, \"content\": \"[NEW]\"}, {\"id\": 5, \"content\": \"b\"}],"
+ + " \"model\": {\"type\": \"WordPiece\", \"vocab\": {\"a\": 0}}}"})
+ void testLoadJsonVocabRejectsAnAddedTokenWhoseIdAnotherTokenHas(String json) {
+ final IllegalArgumentException e = assertThrows(IllegalArgumentException.class,
+ () -> DL.readJsonVocab(json).ids());
+ assertTrue(e.getMessage().contains("\"[NEW]\""), e.getMessage());
+ assertTrue(e.getMessage().contains("\"b\""), e.getMessage());
+ }
+
+ @ParameterizedTest
+ @CsvSource(delimiter = '|', value = {
+ // not a list
+ "{\"added_tokens\": null, MODEL} | added_tokens",
+ "{\"added_tokens\": {\"id\": 1, \"content\": \"c\"}, MODEL} | added_tokens",
+ // an entry that is not an object
+ "{\"added_tokens\": [\"c\"], MODEL} | added_tokens[0]",
+ "{\"added_tokens\": [[1, \"c\"]], MODEL} | added_tokens[0]",
+ // id or content missing
+ "{\"added_tokens\": [{\"content\": \"c\"}], MODEL} | added_tokens[0]",
+ "{\"added_tokens\": [{\"id\": 1}], MODEL} | added_tokens[0]",
+ // id or content of another type: the message names the member
+ "{\"added_tokens\": [{\"id\": \"1\", \"content\": \"c\"}], MODEL} | \"id\"",
+ "{\"added_tokens\": [{\"id\": -1, \"content\": \"c\"}], MODEL} | \"id\"",
+ "{\"added_tokens\": [{\"id\": 1.0, \"content\": \"c\"}], MODEL} | \"id\"",
+ "{\"added_tokens\": [{\"id\": 1, \"content\": null}], MODEL} | \"content\"",
+ "{\"added_tokens\": [{\"id\": 1, \"content\": [\"c\"]}], MODEL} | \"content\""})
+ void testLoadJsonVocabRejectsMalformedAddedTokens(String template, String named) {
+ final String json = template.replace("MODEL",
+ "\"model\": {\"type\": \"WordPiece\", \"vocab\": {\"a\": 0}}");
+ final IllegalArgumentException e = assertThrows(IllegalArgumentException.class,
+ () -> DL.readJsonVocab(json).ids());
+ assertTrue(e.getMessage().contains(named), e.getMessage());
+ }
+
+ static Stream<Arguments> lowercaseSettings() {
+ final String model = "\"model\": {\"type\": \"WordPiece\", \"vocab\": {\"a\": 0}}";
+ return Stream.of(
+ Arguments.of(TOKENIZER_JSON, Boolean.TRUE),
+ Arguments.of("{\"normalizer\": {\"type\": \"BertNormalizer\", \"lowercase\": false}, " + model
+ + "}", Boolean.FALSE),
+ Arguments.of("{" + model + ", \"normalizer\": {\"lowercase\": true}}", Boolean.TRUE),
+ // a later normalizer, and a later lowercase, wins
+ Arguments.of("{\"normalizer\": {\"lowercase\": true, \"lowercase\": false}, " + model
+ + ", \"normalizer\": {\"lowercase\": false, \"lowercase\": true}}", Boolean.TRUE),
+ // no normalizer, a normalizer without the setting, or one that is not an object
+ Arguments.of("{" + model + "}", null),
+ Arguments.of("{\"normalizer\": {\"type\": \"NFC\"}, " + model + "}", null),
+ Arguments.of("{\"normalizer\": null, " + model + "}", null),
+ // a vocab.json has no setting
+ Arguments.of("{\"a\": 0, \"normalizer\": 1}", null));
+ }
+
+ @ParameterizedTest
+ @MethodSource("lowercaseSettings")
+ void testReadJsonVocabKeepsTheLowercaseSetting(String json, Boolean expected) {
+ assertEquals(expected, DL.readJsonVocab(json).lowercase());
+ }
+
+ @ParameterizedTest
+ @ValueSource(strings = {"\"true\"", "1", "null", "{}", "[true]", "NaN"})
+ void testReadJsonVocabRejectsALowercaseSettingThatIsNotABoolean(String value) {
+ final String json = "{\"normalizer\": {\"lowercase\": " + value + "},"
+ + " \"model\": {\"type\": \"WordPiece\", \"vocab\": {\"a\": 0}}}";
+ final IllegalArgumentException e = assertThrows(IllegalArgumentException.class,
+ () -> DL.readJsonVocab(json));
+ assertTrue(e.getMessage().contains("\"lowercase\""), e.getMessage());
+ }
+
+ @Test
+ void testReadVocabFileKeepsTheLowercaseSettingOfATokenizerJson() throws IOException {
+ final File tempFile = vocabFile("tokenizer.json", TOKENIZER_JSON);
+
+ final AbstractDL.Vocabulary vocabulary = DL.readVocabFile(tempFile);
+
+ assertEquals(TOKENIZER_VOCAB, vocabulary.ids());
+ assertEquals(Boolean.TRUE, vocabulary.lowercase());
+ assertThrows(InvalidFormatException.class,
+ () -> DL.requireLowerCase(tempFile, vocabulary, false));
+ DL.requireLowerCase(tempFile, vocabulary, true);
+ }
+
+ @Test
+ void testRequireLowerCaseNamesTheFileAndBothSettings() throws IOException {
+ final File tempFile = vocabFile("tokenizer-cased.json", "{\"normalizer\": {\"lowercase\": false},"
+ + " \"model\": {\"type\": \"WordPiece\", \"vocab\": {\"a\": 0}}}");
+ final AbstractDL.Vocabulary vocabulary = DL.readVocabFile(tempFile);
final InvalidFormatException e = assertThrows(InvalidFormatException.class,
- () -> AbstractDL.loadVocabFile(tempFile));
+ () -> DL.requireLowerCase(tempFile, vocabulary, true));
assertTrue(e.getMessage().contains(tempFile.getName()), e.getMessage());
- assertTrue(e.getMessage().contains(
- "Expected one object mapping tokens to integer ids, as in vocab.json"), e.getMessage());
+ assertTrue(e.getMessage().contains("normalizer.lowercase"), e.getMessage());
+ assertTrue(e.getMessage().contains("false"), e.getMessage());
+ assertTrue(e.getMessage().contains("lowerCase true"), e.getMessage());
+ }
+
+ @ParameterizedTest
+ @ValueSource(booleans = {true, false})
+ void testRequireLowerCaseAcceptsAVocabularyWithoutTheSetting(boolean lowerCase)
+ throws IOException {
+ final File plain = vocabFile("vocab.txt", "[CLS]\n[SEP]\n");
+ DL.requireLowerCase(plain, DL.readVocabFile(plain), lowerCase);
+ final File flat = vocabFile("vocab.json", "{\"[CLS]\": 0}");
+ DL.requireLowerCase(flat, DL.readVocabFile(flat), lowerCase);
+ }
+
+ @ParameterizedTest
+ @ValueSource(strings = {"BPE", "Unigram", "WordLevel", "wordpiece", ""})
+ void testLoadJsonVocabRejectsOtherModelTypes(String type) {
+ final String json = "{\"model\": {\"type\": \"" + type + "\", \"vocab\": {\"a\": 0}}}";
+ final IllegalArgumentException e = assertThrows(IllegalArgumentException.class,
+ () -> DL.readJsonVocab(json).ids());
+ assertTrue(e.getMessage().contains("WordPiece"), e.getMessage());
+ assertTrue(e.getMessage().contains("\"" + type + "\""), e.getMessage());
+ }
+
+ @Test
+ void testLoadJsonVocabRejectsAByteLevelBpeTokenizer() {
+ // RoBERTa: without the type check this would load, and the <s> and </s> tokens would then
+ // select the RoBERTa branch of the WordPiece encoder over a byte-level vocabulary
+ final String json = "{\"model\": {\"type\": \"BPE\", \"vocab\": {\"<s>\": 0, \"</s>\": 2,"
+ + " \"<unk>\": 3, \"\u0120the\": 4}, \"merges\": [\"\u0120 t\"]}}";
+ final IllegalArgumentException e = assertThrows(IllegalArgumentException.class,
+ () -> DL.readJsonVocab(json).ids());
+ assertTrue(e.getMessage().contains("\"BPE\""), e.getMessage());
+ }
+
+ @ParameterizedTest
+ @ValueSource(strings = {"\"@@\"", "\"\"", "\"#\"", "null", "7"})
+ void testLoadJsonVocabRejectsAnotherContinuingSubwordPrefix(String prefix) {
+ final String json = "{\"model\": {\"type\": \"WordPiece\", \"continuing_subword_prefix\": "
+ + prefix + ", \"vocab\": {\"a\": 0, \"##b\": 1}}}";
+ final IllegalArgumentException e = assertThrows(IllegalArgumentException.class,
+ () -> DL.readJsonVocab(json).ids());
+ assertTrue(e.getMessage().contains("continuing_subword_prefix"), e.getMessage());
+ assertTrue(e.getMessage().contains("##"), e.getMessage());
}
@Test
void testLoadJsonVocabSkipsALeadingByteOrderMark() {
- assertEquals(Map.of("a", 1), AbstractDL.loadJsonVocab("\uFEFF{\"a\": 1}"));
+ assertEquals(Map.of("a", 1), DL.readJsonVocab("\uFEFF{\"a\": 1}").ids());
+ assertEquals(TOKENIZER_VOCAB, DL.readJsonVocab("\uFEFF" + TOKENIZER_JSON).ids());
}
@Test
void testJsonVocabFileWithAByteOrderMarkIsReadAsJson() throws IOException {
final File tempFile = vocabFile("vocab-bom.json", "\uFEFF{\"a\": 0, \"b\": 1}\n");
- assertEquals(Map.of("a", 0, "b", 1), AbstractDL.loadVocabFile(tempFile));
+ assertEquals(Map.of("a", 0, "b", 1), DL.loadVocab(tempFile));
}
@Test
void testPlainTextVocabFileWithAByteOrderMarkKeepsTheFirstToken() throws IOException {
final File tempFile = vocabFile("vocab-bom.txt", "\uFEFF[CLS]\n[SEP]\n");
- assertEquals(Map.of("[CLS]", 0, "[SEP]", 1), AbstractDL.loadVocabFile(tempFile));
+ assertEquals(Map.of("[CLS]", 0, "[SEP]", 1), DL.loadVocab(tempFile));
}
static Stream<Arguments> plainTextVocabWithEmptyLines() {
@@ -236,58 +575,112 @@ public class LoadVocabTest {
throws IOException {
final File tempFile = vocabFile("vocab-empty-lines.txt", content);
- assertEquals(expected, AbstractDL.loadVocabFile(tempFile));
+ assertEquals(expected, DL.loadVocab(tempFile));
}
@Test
void testPlainTextVocabFileWithATrailingEmptyLineBuildsAnEncoder() throws IOException {
final File tempFile = vocabFile("vocab-trailing.txt", "[PAD]\n[UNK]\n[CLS]\n[SEP]\nhello\n\n");
- assertNotNull(AbstractDL.createWordpieceEncoder(AbstractDL.loadVocabFile(tempFile), true));
+ assertNotNull(AbstractDL.createWordpieceEncoder(DL.loadVocab(tempFile), true));
+ }
+
+ static Stream<Arguments> tokenizerLayouts() {
+ return Stream.of(
+ // model.vocab is the vocabulary; vocab_size and the other members are not entries
+ Arguments.of(TOKENIZER_JSON, TOKENIZER_VOCAB),
+ Arguments.of(TOKENIZER_JSON.replace("\n", "\r\n"), TOKENIZER_VOCAB),
+ Arguments.of(TOKENIZER_JSON.replace("\n", "").replace(" ", ""), TOKENIZER_VOCAB),
+ // added_tokens listed in model.vocab with the same id are entries once
+ Arguments.of("{\"added_tokens\": [{\"id\": 2, \"content\": \"b\", \"special\": true}],"
+ + " \"model\": {\"type\": \"WordPiece\", \"vocab\": {\"a\": 0, \"b\": 2}}}",
+ Map.of("a", 0, "b", 2)),
+ // an empty vocab object is an empty vocabulary
+ Arguments.of("{\"model\": {\"type\": \"WordPiece\", \"vocab\": {}}}", Map.of()),
+ // the continuing subword prefix of the WordPiece encoder is accepted
+ Arguments.of("{\"model\": {\"type\": \"WordPiece\", \"continuing_subword_prefix\": \"##\","
+ + " \"max_input_chars_per_word\": 100, \"vocab\": {\"a\": 0, \"##b\": 1}}}",
+ Map.of("a", 0, "##b", 1)),
+ // other top-level members, integers included, are not entries
+ Arguments.of("{\"version\": \"1.0\", \"truncation\": {\"max_length\": 512},"
+ + " \"padding\": null, \"vocab_size\": 1,"
+ + " \"model\": {\"type\": \"WordPiece\", \"vocab\": {\"a\": 0}}}", Map.of("a", 0)),
+ // vocab objects elsewhere, at any depth, are not the vocabulary
+ Arguments.of("{\"normalizer\": {\"a\": {\"b\": {\"c\": [[[{\"vocab\": {\"x\": 9}}]]]}}},"
+ + " \"model\": {\"type\": \"WordPiece\", \"decoder\": {\"vocab\": {\"y\": 8}},"
+ + " \"merges\": [[\"a\", \"b\"]], \"vocab\": {\"z\": 1}}}", Map.of("z", 1)),
+ // a later model, a later vocab, or a later token wins
+ Arguments.of("{\"model\": {\"type\": \"WordPiece\", \"vocab\": {\"a\": 1}},"
+ + " \"model\": {\"type\": \"WordPiece\", \"vocab\": {\"a\": 2}}}", Map.of("a", 2)),
+ Arguments.of("{\"model\": {\"type\": \"WordPiece\", \"vocab\": {\"a\": 1}, \"vocab\": {\"b\": 3}}}",
+ Map.of("b", 3)),
+ Arguments.of("{\"model\": {\"type\": \"WordPiece\","
+ + " \"vocab\": {\"a\": 1, \"a\": 5, \"\\u0061\": 6}}}", Map.of("a", 6)),
+ // the member names and the type may be written with escapes
+ Arguments.of("{\"\\u006dodel\": {\"t\\u0079pe\": \"Word\\u0050iece\", \"voc\\u0061b\": {\"a\": 1}}}",
+ Map.of("a", 1)),
+ Arguments.of("{ \"model\"\t:\t{ \"type\" : \"WordPiece\" , \"vocab\"\r\n:\r\n{ \"a\" : 1 } } }",
+ Map.of("a", 1)),
+ Arguments.of("{\"model\": {\"type\": \"WordPiece\", \"vocab\": {\"a\": 2147483647}}}",
+ Map.of("a", Integer.MAX_VALUE)));
+ }
+
+ @ParameterizedTest
+ @MethodSource("tokenizerLayouts")
+ void testLoadJsonVocabTokenizerLayouts(String json, Map<String, Integer> expected) {
+ assertEquals(expected, DL.readJsonVocab(json).ids());
}
@ParameterizedTest
@ValueSource(strings = {"2147483648", "4294967296", "9223372036854775808",
"12345678901234567890"})
void testLoadJsonVocabNamesTheTokenOfAnIdThatDoesNotFit(String id) {
- final String json = "{\"big\": " + id + "}";
- final IllegalArgumentException e = assertThrows(IllegalArgumentException.class,
- () -> AbstractDL.loadJsonVocab(json), json);
- assertTrue(e.getMessage().contains("\"big\""), e.getMessage());
- assertTrue(e.getMessage().contains("does not fit into an int"), e.getMessage());
+ for (String json : new String[] {"{\"big\": " + id + "}",
+ "{\"model\": {\"type\": \"WordPiece\", \"vocab\": {\"big\": " + id + "}}}"}) {
+ final IllegalArgumentException e = assertThrows(IllegalArgumentException.class,
+ () -> DL.readJsonVocab(json).ids(), json);
+ assertTrue(e.getMessage().contains("\"big\""), e.getMessage());
+ assertTrue(e.getMessage().contains("does not fit into an int"), e.getMessage());
+ }
}
@ParameterizedTest
- @ValueSource(strings = {"{\"a\": 00}", "{\"a\": 0123}"})
+ @ValueSource(strings = {"{\"a\": 00}", "{\"a\": 0123}",
+ "{\"model\": {\"type\": \"WordPiece\", \"vocab\": {\"a\": 01}}}"})
void testLoadJsonVocabRejectsLeadingZeros(String json) {
final IllegalArgumentException e = assertThrows(IllegalArgumentException.class,
- () -> AbstractDL.loadJsonVocab(json));
+ () -> DL.readJsonVocab(json).ids());
assertTrue(e.getMessage().contains("offset "), e.getMessage());
}
- private static final String JSON_VOCAB = "{\"[PAD]\":0,\"hello\":1,\"\\u0120x\":2}";
-
static Stream<Arguments> jsonVocabPrefixes() {
- final String text = JSON_VOCAB;
- return Stream.iterate(0, n -> n + 1).limit(text.length())
- .map(n -> Arguments.of(n, text.substring(0, n)));
+ return Stream.of(JSON_VOCAB, TOKENIZER_JSON.strip()).flatMap(text ->
+ Stream.iterate(0, n -> n + 1).limit(text.length())
+ .map(n -> Arguments.of(n, text.substring(0, n))));
}
- @ParameterizedTest(name = "cut at {0}")
+ @ParameterizedTest(name = "cut at {0}: {1}")
@MethodSource("jsonVocabPrefixes")
void testLoadJsonVocabRejectsATruncatedObject(int length, String prefix) {
final IllegalArgumentException e = assertThrows(IllegalArgumentException.class,
- () -> AbstractDL.loadJsonVocab(prefix));
+ () -> DL.readJsonVocab(prefix).ids());
assertTrue(e.getMessage().contains("offset "), e.getMessage());
}
+ static Stream<Arguments> jsonVocabTrailers() {
+ return Stream.of(JSON_VOCAB, TOKENIZER_JSON).flatMap(text ->
+ Stream.of("{}", "x", "\uFEFF", ",", "\"vocab\"", "{\"a\":1}",
+ "{\"model\": {\"type\": \"WordPiece\", \"vocab\": {}}}")
+ .map(trailing -> Arguments.of(text, trailing)));
+ }
+
@ParameterizedTest
- @ValueSource(strings = {"{}", "x", "\uFEFF", ",", "\"vocab\"", "{\"a\":1}"})
- void testLoadJsonVocabRejectsContentAfterTheObject(String trailing) {
- final String json = JSON_VOCAB + trailing;
+ @MethodSource("jsonVocabTrailers")
+ void testLoadJsonVocabRejectsContentAfterTheObject(String text, String trailing) {
+ final String json = text + trailing;
final IllegalArgumentException e = assertThrows(IllegalArgumentException.class,
- () -> AbstractDL.loadJsonVocab(json));
- assertTrue(e.getMessage().contains("offset " + JSON_VOCAB.length() + ","), e.getMessage());
+ () -> DL.readJsonVocab(json).ids());
+ assertTrue(e.getMessage().contains("offset " + text.length() + ","), e.getMessage());
assertTrue(e.getMessage().contains("content after the object"), e.getMessage());
}
@@ -295,14 +688,14 @@ public class LoadVocabTest {
void testJsonVocabFileWithWindowsLineEndings() throws IOException {
final File tempFile = vocabFile("vocab-crlf.json", "{\r\n \"a\": 0,\r\n \"b\": 1\r\n}\r\n");
- assertEquals(Map.of("a", 0, "b", 1), AbstractDL.loadVocabFile(tempFile));
+ assertEquals(Map.of("a", 0, "b", 1), DL.loadVocab(tempFile));
}
@Test
void testPlainTextVocabFileWithWindowsLineEndings() throws IOException {
final File tempFile = vocabFile("vocab-crlf.txt", "[CLS]\r\n[SEP]\r\nhello\r\n");
- assertEquals(Map.of("[CLS]", 0, "[SEP]", 1, "hello", 2), AbstractDL.loadVocabFile(tempFile));
+ assertEquals(Map.of("[CLS]", 0, "[SEP]", 1, "hello", 2), DL.loadVocab(tempFile));
}
@Test
@@ -310,7 +703,7 @@ public class LoadVocabTest {
final File tempFile = vocabFile("vocab-malformed.json", "{\"a\": 1, \"b\": }");
final InvalidFormatException e =
- assertThrows(InvalidFormatException.class, () -> AbstractDL.loadVocabFile(tempFile));
+ assertThrows(InvalidFormatException.class, () -> DL.loadVocab(tempFile));
assertTrue(e.getMessage().contains(tempFile.getName()), e.getMessage());
assertTrue(e.getMessage().contains("offset "), e.getMessage());
}
@@ -319,13 +712,21 @@ public class LoadVocabTest {
void testJsonVocabFileWithAnInvalidEscapeIsReportedAsAnInvalidFormat() throws IOException {
final File tempFile = vocabFile("vocab-invalid-escape.json", "{\"bad\\xescape\": 0}");
- assertThrows(InvalidFormatException.class, () -> AbstractDL.loadVocabFile(tempFile));
+ assertThrows(InvalidFormatException.class, () -> DL.loadVocab(tempFile));
+ }
+
+ @Test
+ void testLoadJsonVocabNamesTheUnsupportedUnigramModelType() {
+ final IllegalArgumentException e = assertThrows(IllegalArgumentException.class,
+ () -> DL.readJsonVocab(UNIGRAM_TOKENIZER_JSON).ids());
+ assertTrue(e.getMessage().contains("WordPiece"), e.getMessage());
+ assertTrue(e.getMessage().contains("\"Unigram\""), e.getMessage());
}
@Test
void testLoadJsonVocabRejectsANonFiniteId() {
final IllegalArgumentException e = assertThrows(IllegalArgumentException.class,
- () -> AbstractDL.loadJsonVocab("{\"ok\": 1, \"bad\": NaN}"));
+ () -> DL.readJsonVocab("{\"ok\": 1, \"bad\": NaN}").ids());
assertTrue(e.getMessage().contains("\"bad\""), e.getMessage());
}
}
diff --git a/opennlp-core/opennlp-ml/opennlp-dl/src/test/java/opennlp/dl/vectors/SentenceVectorsDLEmbedderTest.java b/opennlp-core/opennlp-ml/opennlp-dl/src/test/java/opennlp/dl/vectors/SentenceVectorsDLEmbedderTest.java
index 8ed511dcf..8351f8930 100644
--- a/opennlp-core/opennlp-ml/opennlp-dl/src/test/java/opennlp/dl/vectors/SentenceVectorsDLEmbedderTest.java
+++ b/opennlp-core/opennlp-ml/opennlp-dl/src/test/java/opennlp/dl/vectors/SentenceVectorsDLEmbedderTest.java
@@ -33,6 +33,7 @@ import org.junit.jupiter.api.io.TempDir;
import opennlp.tools.embeddings.TextEmbedder;
import opennlp.tools.embeddings.TextEmbedderProvider;
+import opennlp.tools.util.InvalidFormatException;
import opennlp.tools.util.ext.ProviderSpec;
import opennlp.tools.util.ext.Providers;
@@ -45,6 +46,7 @@ import static org.junit.jupiter.api.Assertions.assertArrayEquals;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertNotSame;
import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
/**
* The {@link TextEmbedder} adapter driven through a real ONNX session. The bundled
@@ -100,6 +102,16 @@ class SentenceVectorsDLEmbedderTest {
return file.toFile();
}
+ // The vocabulary above as a tokenizer.json of an uncased WordPiece model.
+ private static File tokenizerJson(final Path dir, final boolean lowercase) throws IOException {
+ final Path file = dir.resolve("tokenizer.json");
+ Files.writeString(file, "{\"version\": \"1.0\", \"normalizer\": {\"type\": \"BertNormalizer\","
+ + " \"lowercase\": " + lowercase + "}, \"model\": {\"type\": \"WordPiece\","
+ + " \"vocab\": {\"[PAD]\": 0, \"[UNK]\": 2, \"[SEP]\": 3, \"hello\": 4, \"world\": 5,"
+ + " \"[CLS]\": 7}}}");
+ return file.toFile();
+ }
+
private static SentenceVectorsDL meanVectors(final Path dir, final int maxLength)
throws Exception {
return new SentenceVectorsDL(model(dir), vocab(dir), true, Pooling.MEAN, false, maxLength);
@@ -205,6 +217,19 @@ class SentenceVectorsDLEmbedderTest {
vocab(dir), true, null, false, SentenceVectorsDL.DEFAULT_MAX_LENGTH));
}
+ @Test
+ void testTokenizerJsonVocabularyMustAgreeWithLowerCase(@TempDir final Path dir)
+ throws Exception {
+ try (SentenceVectorsDL vectors = new SentenceVectorsDL(model(dir), tokenizerJson(dir, true),
+ true, Pooling.MEAN, false, SentenceVectorsDL.DEFAULT_MAX_LENGTH)) {
+ assertArrayEquals(HELLO_WORLD_MEAN, vectors.embed("Hello World"), DELTA);
+ }
+ final InvalidFormatException e = assertThrows(InvalidFormatException.class,
+ () -> new SentenceVectorsDL(model(dir), tokenizerJson(dir, false), true));
+ assertTrue(e.getMessage().contains("tokenizer.json"), e.getMessage());
+ assertTrue(e.getMessage().contains("normalizer.lowercase"), e.getMessage());
+ }
+
@Test
void testPooledOutputIsSelectedByName(@TempDir final Path dir) throws Exception {
final File pooledModel = model(dir, "tiny-pooled.onnx");
diff --git a/opennlp-core/opennlp-ml/opennlp-dl/src/test/resources/opennlp/dl/tokenizer.json b/opennlp-core/opennlp-ml/opennlp-dl/src/test/resources/opennlp/dl/tokenizer.json
new file mode 100644
index 000000000..913872d10
--- /dev/null
+++ b/opennlp-core/opennlp-ml/opennlp-dl/src/test/resources/opennlp/dl/tokenizer.json
@@ -0,0 +1,79 @@
+{
+ "version": "1.0",
+ "truncation": null,
+ "padding": null,
+ "added_tokens": [
+ {"id": 0, "content": "[PAD]", "single_word": false, "lstrip": false, "rstrip": false, "normalized": false, "special": true},
+ {"id": 1, "content": "[UNK]", "single_word": false, "lstrip": false, "rstrip": false, "normalized": false, "special": true},
+ {"id": 2, "content": "[CLS]", "single_word": false, "lstrip": false, "rstrip": false, "normalized": false, "special": true},
+ {"id": 3, "content": "[SEP]", "single_word": false, "lstrip": false, "rstrip": false, "normalized": false, "special": true},
+ {"id": 4, "content": "[MASK]", "single_word": false, "lstrip": false, "rstrip": false, "normalized": false, "special": true},
+ {"id": 24, "content": "opennlp", "single_word": true, "lstrip": false, "rstrip": false, "normalized": true, "special": false}
+ ],
+ "normalizer": {
+ "type": "BertNormalizer",
+ "clean_text": true,
+ "handle_chinese_chars": true,
+ "strip_accents": null,
+ "lowercase": true
+ },
+ "pre_tokenizer": {
+ "type": "BertPreTokenizer"
+ },
+ "post_processor": {
+ "type": "TemplateProcessing",
+ "single": [
+ {"SpecialToken": {"id": "[CLS]", "type_id": 0}},
+ {"Sequence": {"id": "A", "type_id": 0}},
+ {"SpecialToken": {"id": "[SEP]", "type_id": 0}}
+ ],
+ "pair": [
+ {"SpecialToken": {"id": "[CLS]", "type_id": 0}},
+ {"Sequence": {"id": "A", "type_id": 0}},
+ {"SpecialToken": {"id": "[SEP]", "type_id": 0}},
+ {"Sequence": {"id": "B", "type_id": 1}},
+ {"SpecialToken": {"id": "[SEP]", "type_id": 1}}
+ ],
+ "special_tokens": {
+ "[CLS]": {"id": "[CLS]", "ids": [2], "tokens": ["[CLS]"]},
+ "[SEP]": {"id": "[SEP]", "ids": [3], "tokens": ["[SEP]"]}
+ }
+ },
+ "decoder": {
+ "type": "WordPiece",
+ "prefix": "##",
+ "cleanup": true
+ },
+ "model": {
+ "type": "WordPiece",
+ "unk_token": "[UNK]",
+ "continuing_subword_prefix": "##",
+ "max_input_chars_per_word": 100,
+ "vocab": {
+ "[PAD]": 0,
+ "[UNK]": 1,
+ "[CLS]": 2,
+ "[SEP]": 3,
+ "[MASK]": 4,
+ "the": 5,
+ "hello": 6,
+ "world": 7,
+ "play": 8,
+ "##ing": 9,
+ "##s": 10,
+ "##ed": 11,
+ "un": 12,
+ "##play": 13,
+ "##able": 14,
+ "open": 15,
+ "##nlp": 16,
+ ".": 17,
+ ",": 18,
+ "!": 19,
+ "a": 20,
+ "##a": 21,
+ "##b": 22,
+ "##c": 23
+ }
+ }
+}
diff --git a/opennlp-docs/src/docbkx/doccat.xml b/opennlp-docs/src/docbkx/doccat.xml
index b41729334..0207df2bc 100644
--- a/opennlp-docs/src/docbkx/doccat.xml
+++ b/opennlp-docs/src/docbkx/doccat.xml
@@ -175,9 +175,16 @@ String category = myCategorizer.getBestCategory(outcomes);]]>
<para>
The vocabulary file is either plain text with one token per line, as in
<code>vocab.txt</code>, or a JSON object mapping each token to an integer ID, as in
- <code>vocab.json</code>. A Hugging Face <code>tokenizer.json</code> is not supported yet
- (<ulink url="https://issues.apache.org/jira/browse/OPENNLP-1988">OPENNLP-1988</ulink>).
+ <code>vocab.json</code>. A Hugging Face <code>tokenizer.json</code> of a WordPiece model
+ is also accepted: <code>model.vocab</code> supplies the tokens, and an entry of
+ <code>added_tokens</code> that <code>model.vocab</code> lacks is added with its ID. A
+ <code>tokenizer.json</code> of another model type, such as BPE or Unigram, is rejected,
+ and so is an added token whose ID or token conflicts with <code>model.vocab</code>. When
+ the file sets <code>normalizer.lowercase</code>, it must agree with the lower casing the
+ component is configured with; the other tokenizer settings of the file are not read.
An unreadable vocabulary is rejected with an <code>InvalidFormatException</code>.
+ The same vocabulary files are accepted by <code>NameFinderDL</code> and
+ <code>SentenceVectorsDL</code>.
</para>
<para>
When a configuration file is given in place of the categories map, its top-level
diff --git a/opennlp-docs/src/docbkx/machine-learning.xml b/opennlp-docs/src/docbkx/machine-learning.xml
index 3aa6889f3..06688b47f 100644
--- a/opennlp-docs/src/docbkx/machine-learning.xml
+++ b/opennlp-docs/src/docbkx/machine-learning.xml
@@ -167,7 +167,9 @@ try (TextEmbedder embedder = providers.select(spec).create(spec)) {
<para>
The <code>onnx</code> provider supports a local model file named <code>*.onnx</code>. It
requires the option <code>vocabulary</code>, resolved against the directory of the model if
- it is relative, and accepts <code>lowerCase=true</code> (default) or
+ it is relative, which names a <code>vocab.txt</code>, a <code>vocab.json</code>, or the
+ <code>tokenizer.json</code> of a WordPiece model, as described for the document categorizer
+ in <xref linkend="tools.doccat.api.onnx"/>, and accepts <code>lowerCase=true</code> (default) or
<code>lowerCase=false</code>, <code>pooling=mean</code> (default) or
<code>pooling=cls</code>, <code>normalize=true</code> (default) or
<code>normalize=false</code>, and <code>maxLength</code>, the maximum number of tokens
diff --git a/opennlp-docs/src/docbkx/namefinder.xml b/opennlp-docs/src/docbkx/namefinder.xml
index 4a4183e18..43bbeb58a 100644
--- a/opennlp-docs/src/docbkx/namefinder.xml
+++ b/opennlp-docs/src/docbkx/namefinder.xml
@@ -150,6 +150,9 @@ Span[] nameSpans = nameFinder.find(sentence);]]>
<para>
Using an ONNX model is similar, except we will utilize the <code>NameFinderDL</code> class instead.
You must provide the path to the model file and the vocabulary file to the name finder.
+ The vocabulary file is a <code>vocab.txt</code>, a <code>vocab.json</code>, or the
+ <code>tokenizer.json</code> of a WordPiece model, as described for the document categorizer in
+ <xref linkend="tools.doccat.api.onnx"/>.
(There is no need to load the model as an InputStream as in the previous example.) The name finder
requires a tokenized list of strings as input. The output will be an array of spans.
<programlisting language="java">