[PR] OPENNLP-1885: Test the shared deep-learning encoder and fix the token type javadoc (opennlp)

["krickert (via GitHub)" <gi...@apache.org>]

krickert opened a new pull request, #1288:
URL: https://github.com/apache/opennlp/pull/1288

   Follow-up to #1165 on the same issue, based on main and independent of the rest.
   
   #1165 routed `DocumentCategorizerDL`, `NameFinderDL` and `SentenceVectorsDL` through one `AbstractDL.encodeTokens`. The per-component tests that had checked the token and id arrays were removed along with the duplicated code they covered, leaving one encoding case in `CreateTokenizerTest`. This adds a test class for the shared method, plus a javadoc fix found while writing it.
   
   `SharedDlEncodingTest` checks all model input arrays (tokens, ids, mask, types) across 26 cases: empty input, ASCII and non-ASCII whitespace, known and unknown words, wordpiece continuations, a `StringBuilder` input, precomposed and decomposed accents under both case settings, a word-final Greek capital sigma under both case settings, the RoBERTa special tokens, and a RoBERTa token map that falls back to the BERT unknown token. Fixture ids include 0 and `Integer.MAX_VALUE`, so a sign or ordering slip fails the test. Further cases pin that repeated calls return independent arrays, and that 16 concurrent calls on a small thread pool each get a complete result. No model file is needed and the class runs in milliseconds.
   
   `ModelFreeDL` is the model-free subclass both test classes now share, replacing the private copy in `CreateTokenizerTest`.
   
   Javadoc fixes:
   
   - `Tokens.types` was documented as "The token types. (Typically all 1.)". The encoder writes all 0 there, which is the right single-segment value, so the text did not match the code.
   - `WordpieceEncoder` now links `SpecialCasing.txt` where it mentions the Unicode full case mapping, matching the `WordBreakProperty.txt` reference already there.
   
   No production behavior changes.
   
   Verification, offline, with checkstyle:
   
   ```
   ./mvnw -o -pl opennlp-core/opennlp-ml/opennlp-dl -am test \
     -Dtest='SharedDlEncodingTest,CreateTokenizerTest' -DfailIfNoSpecifiedTests=false
   ```
   
   `SharedDlEncodingTest` 26 tests, `CreateTokenizerTest` 12 tests, no failures, no checkstyle violations.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@opennlp.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] OPENNLP-1942: Test the shared deep-learning encoder and fix the token type javadoc (opennlp)

["rzo1 (via GitHub)" <gi...@apache.org>]

rzo1 merged PR #1288:
URL: https://github.com/apache/opennlp/pull/1288


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@opennlp.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] OPENNLP-1942: Test the shared deep-learning encoder and fix the token type javadoc (opennlp)

["krickert (via GitHub)" <gi...@apache.org>]

krickert commented on PR #1288:
URL: https://github.com/apache/opennlp/pull/1288#issuecomment-5724688753

   Thanks for the review. Pushed 3d5943c39, rebased on current main:
   
   - Both earlier commits are reworded to OPENNLP-1942; the review changes are a third commit on top.
   - The fix version on OPENNLP-1942 is 3.0.0 only.
   - Supplementary-plane and CJK cases are in, unknown and split. While adding them I also covered a known emoji token, an emoji inside a word, punctuation isolation, special token text in the input, control and format character removal, and the 100 code point word limit. The PR description lists them.
   
   On the other PRs: #1236 is titled OPENNLP-547 with OPENNLP-1888 commits, and #1238 has no ticket in its commit subjects. I will fix both when I next touch those branches.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@opennlp.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] OPENNLP-1885: Test the shared deep-learning encoder and fix the token type javadoc (opennlp)

["rzo1 (via GitHub)" <gi...@apache.org>]

rzo1 commented on PR #1288:
URL: https://github.com/apache/opennlp/pull/1288#issuecomment-5583956695

   This cannot be 1885 because it is already in the proposed M6 release candidate. It needs a new Jira.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@opennlp.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] OPENNLP-1942: Test the shared deep-learning encoder and fix the token type javadoc (opennlp)

["krickert (via GitHub)" <gi...@apache.org>]

krickert commented on code in PR #1288:
URL: https://github.com/apache/opennlp/pull/1288#discussion_r4043546249


##########
opennlp-core/opennlp-ml/opennlp-dl/src/test/java/opennlp/dl/SharedDlEncodingTest.java:
##########
@@ -0,0 +1,353 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package opennlp.dl;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.stream.Stream;
+
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
+
+import opennlp.tools.tokenize.WordpieceTokenizer;
+
+import static org.junit.jupiter.api.Assertions.assertArrayEquals;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertNotSame;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+/** Checks the complete token arrays created by the shared deep-learning encoder. */
+public class SharedDlEncodingTest {
+
+  private static final int CONCURRENT_CALLS = 16;
+  private static final int CONCURRENT_WORKERS = 4;
+  private static final long WAIT_SECONDS = 10;
+
+  private static final Map<String, Integer> BERT_TOKEN_IDS = Map.ofEntries(
+      Map.entry(WordpieceTokenizer.BERT_CLS_TOKEN, 101),
+      Map.entry(WordpieceTokenizer.BERT_SEP_TOKEN, Integer.MAX_VALUE),
+      Map.entry(WordpieceTokenizer.BERT_UNK_TOKEN, 900_001),
+      Map.entry("hello", 42),
+      Map.entry("world", 1_500_000_000),
+      Map.entry("play", 0),
+      Map.entry("##ing", 2_147_483_646),
+      Map.entry("cafe", 88),
+      Map.entry("Caf\u00E9", 89),
+      Map.entry("\u03C3\u03BF\u03C6\u03BF\u03C2", 90),
+      Map.entry("\u03A3\u039F\u03A6\u039F\u03A3", 91));
+
+  private static final Map<String, Integer> ROBERTA_TOKEN_IDS = Map.of(
+      WordpieceTokenizer.ROBERTA_CLS_TOKEN, Integer.MAX_VALUE,
+      WordpieceTokenizer.ROBERTA_SEP_TOKEN, 2,
+      WordpieceTokenizer.ROBERTA_UNK_TOKEN, 800_000_000,
+      "hello", 0,
+      "world", 500,
+      "play", 10,
+      "##ing", 1_500_000_000);
+
+  private static final Map<String, Integer> ROBERTA_BERT_UNKNOWN_TOKEN_IDS = Map.of(
+      WordpieceTokenizer.ROBERTA_CLS_TOKEN, 71,
+      WordpieceTokenizer.ROBERTA_SEP_TOKEN, 72,
+      WordpieceTokenizer.BERT_UNK_TOKEN, Integer.MAX_VALUE,
+      "hello", 73);
+
+  private static final EncodingExpectation BERT_HELLO_EXPECTATION = new EncodingExpectation(
+      BERT_TOKEN_IDS, true, "Hello",
+      new String[] {"[CLS]", "hello", "[SEP]"},
+      new long[] {101, 42, 2_147_483_647L},
+      new long[] {1, 1, 1},
+      new long[] {0, 0, 0});
+
+  private static final EncodingExpectation BERT_MIXED_EXPECTATION = new EncodingExpectation(
+      BERT_TOKEN_IDS, true, "Hello playing rabbit",
+      new String[] {"[CLS]", "hello", "play", "##ing", "[UNK]", "[SEP]"},
+      new long[] {101, 42, 0, 2_147_483_646L, 900_001, 2_147_483_647L},
+      new long[] {1, 1, 1, 1, 1, 1},
+      new long[] {0, 0, 0, 0, 0, 0});
+
+  private static final EncodingExpectation BERT_UNKNOWN_EXPECTATION = new EncodingExpectation(
+      BERT_TOKEN_IDS, true, "rabbit",
+      new String[] {"[CLS]", "[UNK]", "[SEP]"},
+      new long[] {101, 900_001, 2_147_483_647L},
+      new long[] {1, 1, 1},
+      new long[] {0, 0, 0});
+
+  private static final EncodingExpectation BERT_WORDPIECES_EXPECTATION =
+      new EncodingExpectation(
+          BERT_TOKEN_IDS, true, "Playing",
+          new String[] {"[CLS]", "play", "##ing", "[SEP]"},
+          new long[] {101, 0, 2_147_483_646L, 2_147_483_647L},
+          new long[] {1, 1, 1, 1},
+          new long[] {0, 0, 0, 0});
+
+  private static final List<EncodingExpectation> CONCURRENT_EXPECTATIONS = List.of(
+      BERT_HELLO_EXPECTATION,
+      BERT_UNKNOWN_EXPECTATION,
+      BERT_WORDPIECES_EXPECTATION);
+
+  /** Expected output for one model-free encoding case. */
+  private record EncodingExpectation(
+      Map<String, Integer> tokenIds,
+      boolean lowerCase,
+      CharSequence input,
+      String[] tokens,
+      long[] ids,
+      long[] mask,
+      long[] types) {
+  }
+
+  /**
+   * Confirms the token strings, ids, attention mask, and token types.
+   *
+   * @param name The case name.
+   * @param expected The fixture and expected arrays.
+   */
+  @ParameterizedTest(name = "{0}")
+  @MethodSource("encodingCases")
+  void testEncodingArrays(String name, EncodingExpectation expected) {
+    final Tokens actual = new ModelFreeDL(expected.tokenIds(), expected.lowerCase())
+        .encode(expected.input());
+
+    assertEncoding(expected, actual);
+  }
+
+  /** Confirms null text is rejected at the instance encoder boundary. */
+  @Test
+  void testRejectsNullText() {
+    final ModelFreeDL encoder = new ModelFreeDL(BERT_TOKEN_IDS, true);
+
+    final IllegalArgumentException exception = assertThrows(IllegalArgumentException.class,
+        () -> encoder.encode(null));
+    assertEquals("text must not be null", exception.getMessage());
+  }
+
+  /** Confirms that successive calls return independent arrays. */
+  @Test
+  void testRepeatedCallsReturnIndependentArrays() {
+    final ModelFreeDL encoder = new ModelFreeDL(BERT_TOKEN_IDS, true);
+    final Tokens first = encoder.encode("Hello");
+    final Tokens second = encoder.encode("Hello");
+
+    assertNotSame(first.tokens(), second.tokens());
+    assertNotSame(first.ids(), second.ids());
+    assertNotSame(first.mask(), second.mask());
+    assertNotSame(first.types(), second.types());
+
+    first.tokens()[1] = "changed";
+    first.ids()[1] = -1;
+    first.mask()[1] = 0;
+    first.types()[1] = 1;
+
+    assertEncoding(BERT_HELLO_EXPECTATION, second);
+    assertEncoding(BERT_HELLO_EXPECTATION, encoder.encode("Hello"));
+  }
+
+  /**
+   * Confirms bounded concurrent calls produce complete independent results.
+   *
+   * @throws InterruptedException Thrown if the test thread is interrupted.
+   * @throws ExecutionException Thrown if an encoding task fails.
+   * @throws TimeoutException Thrown if an encoding task exceeds the wait limit.
+   */
+  @Test
+  void testConcurrentCallsReturnIndependentArrays()
+      throws InterruptedException, ExecutionException, TimeoutException {
+    final ModelFreeDL encoder = new ModelFreeDL(BERT_TOKEN_IDS, true);
+    final ExecutorService executor = Executors.newFixedThreadPool(CONCURRENT_WORKERS);
+    final CountDownLatch start = new CountDownLatch(1);
+    final List<Future<Tokens>> futures = new ArrayList<>();
+    try {
+      for (int i = 0; i < CONCURRENT_CALLS; i++) {
+        final EncodingExpectation expected =
+            CONCURRENT_EXPECTATIONS.get(i % CONCURRENT_EXPECTATIONS.size());
+        futures.add(executor.submit(() -> {
+          start.await();
+          return encoder.encode(expected.input());
+        }));
+      }
+      start.countDown();
+
+      final List<Tokens> completed = new ArrayList<>();
+      for (int i = 0; i < futures.size(); i++) {
+        final Future<Tokens> future = futures.get(i);
+        final Tokens actual = future.get(WAIT_SECONDS, TimeUnit.SECONDS);
+        assertEncoding(CONCURRENT_EXPECTATIONS.get(i % CONCURRENT_EXPECTATIONS.size()), actual);
+        for (final Tokens prior : completed) {
+          assertNotSame(prior.tokens(), actual.tokens());
+          assertNotSame(prior.ids(), actual.ids());
+          assertNotSame(prior.mask(), actual.mask());
+          assertNotSame(prior.types(), actual.types());
+        }
+        completed.add(actual);
+      }
+    } finally {
+      executor.shutdownNow();
+      assertTrue(executor.awaitTermination(WAIT_SECONDS, TimeUnit.SECONDS));
+    }
+  }
+
+  /**
+   * Supplies model-free encoding fixtures.
+   *
+   * @return The named fixtures.
+   */
+  private static Stream<Arguments> encodingCases() {
+    return Stream.of(
+        Arguments.of("bert-empty", new EncodingExpectation(BERT_TOKEN_IDS, true, "",
+            new String[] {"[CLS]", "[SEP]"},
+            new long[] {101, 2_147_483_647L},
+            new long[] {1, 1},
+            new long[] {0, 0})),
+        Arguments.of("bert-ascii-whitespace", new EncodingExpectation(
+            BERT_TOKEN_IDS, true, " \t\n",
+            new String[] {"[CLS]", "[SEP]"},
+            new long[] {101, 2_147_483_647L},
+            new long[] {1, 1},
+            new long[] {0, 0})),
+        Arguments.of("bert-unicode-whitespace", new EncodingExpectation(
+            BERT_TOKEN_IDS, true, "\u00A0\u2028\u3000",
+            new String[] {"[CLS]", "[SEP]"},
+            new long[] {101, 2_147_483_647L},
+            new long[] {1, 1},
+            new long[] {0, 0})),
+        Arguments.of("bert-known", BERT_HELLO_EXPECTATION),
+        Arguments.of("bert-known-sequence", new EncodingExpectation(
+            BERT_TOKEN_IDS, true, "Hello WORLD",
+            new String[] {"[CLS]", "hello", "world", "[SEP]"},
+            new long[] {101, 42, 1_500_000_000L, 2_147_483_647L},
+            new long[] {1, 1, 1, 1},
+            new long[] {0, 0, 0, 0})),
+        Arguments.of("bert-unicode-separator", new EncodingExpectation(
+            BERT_TOKEN_IDS, true, "Hello\u00A0WORLD",
+            new String[] {"[CLS]", "hello", "world", "[SEP]"},
+            new long[] {101, 42, 1_500_000_000L, 2_147_483_647L},
+            new long[] {1, 1, 1, 1},
+            new long[] {0, 0, 0, 0})),
+        Arguments.of("bert-unknown", BERT_UNKNOWN_EXPECTATION),
+        Arguments.of("bert-known-unknown", new EncodingExpectation(
+            BERT_TOKEN_IDS, true, "Hello rabbit",
+            new String[] {"[CLS]", "hello", "[UNK]", "[SEP]"},
+            new long[] {101, 42, 900_001, 2_147_483_647L},
+            new long[] {1, 1, 1, 1},
+            new long[] {0, 0, 0, 0})),
+        Arguments.of("bert-wordpieces", BERT_WORDPIECES_EXPECTATION),
+        Arguments.of("bert-mixed", BERT_MIXED_EXPECTATION),
+        Arguments.of("bert-string-builder", new EncodingExpectation(
+            BERT_TOKEN_IDS, true, new StringBuilder("Hello"),
+            new String[] {"[CLS]", "hello", "[SEP]"},
+            new long[] {101, 42, 2_147_483_647L},
+            new long[] {1, 1, 1},
+            new long[] {0, 0, 0})),
+        Arguments.of("bert-cased-accent", new EncodingExpectation(
+            BERT_TOKEN_IDS, false, "Caf\u00E9",
+            new String[] {"[CLS]", "Caf\u00E9", "[SEP]"},
+            new long[] {101, 89, 2_147_483_647L},
+            new long[] {1, 1, 1},
+            new long[] {0, 0, 0})),
+        Arguments.of("bert-cased-miss", new EncodingExpectation(
+            BERT_TOKEN_IDS, false, "Hello",
+            new String[] {"[CLS]", "[UNK]", "[SEP]"},
+            new long[] {101, 900_001, 2_147_483_647L},
+            new long[] {1, 1, 1},
+            new long[] {0, 0, 0})),
+        Arguments.of("bert-precomposed-accent", new EncodingExpectation(
+            BERT_TOKEN_IDS, true, "CAF\u00C9",
+            new String[] {"[CLS]", "cafe", "[SEP]"},
+            new long[] {101, 88, 2_147_483_647L},
+            new long[] {1, 1, 1},
+            new long[] {0, 0, 0})),
+        Arguments.of("bert-decomposed-accent", new EncodingExpectation(
+            BERT_TOKEN_IDS, true, "Cafe\u0301",
+            new String[] {"[CLS]", "cafe", "[SEP]"},
+            new long[] {101, 88, 2_147_483_647L},
+            new long[] {1, 1, 1},
+            new long[] {0, 0, 0})),
+        Arguments.of("bert-final-sigma", new EncodingExpectation(
+            BERT_TOKEN_IDS, true, "\u03A3\u039F\u03A6\u039F\u03A3",
+            new String[] {"[CLS]", "\u03C3\u03BF\u03C6\u03BF\u03C2", "[SEP]"},
+            new long[] {101, 90, 2_147_483_647L},
+            new long[] {1, 1, 1},
+            new long[] {0, 0, 0})),
+        Arguments.of("bert-cased-sigma", new EncodingExpectation(
+            BERT_TOKEN_IDS, false, "\u03A3\u039F\u03A6\u039F\u03A3",
+            new String[] {"[CLS]", "\u03A3\u039F\u03A6\u039F\u03A3", "[SEP]"},
+            new long[] {101, 91, 2_147_483_647L},
+            new long[] {1, 1, 1},
+            new long[] {0, 0, 0})),
+        Arguments.of("roberta-empty", new EncodingExpectation(
+            ROBERTA_TOKEN_IDS, true, "",
+            new String[] {"<s>", "</s>"},
+            new long[] {2_147_483_647L, 2},
+            new long[] {1, 1},
+            new long[] {0, 0})),
+        Arguments.of("roberta-known", new EncodingExpectation(
+            ROBERTA_TOKEN_IDS, true, "Hello",
+            new String[] {"<s>", "hello", "</s>"},
+            new long[] {2_147_483_647L, 0, 2},
+            new long[] {1, 1, 1},
+            new long[] {0, 0, 0})),
+        Arguments.of("roberta-unknown", new EncodingExpectation(
+            ROBERTA_TOKEN_IDS, true, "rabbit",
+            new String[] {"<s>", "<unk>", "</s>"},
+            new long[] {2_147_483_647L, 800_000_000, 2},
+            new long[] {1, 1, 1},
+            new long[] {0, 0, 0})),
+        Arguments.of("roberta-mixed", new EncodingExpectation(
+            ROBERTA_TOKEN_IDS, true, "Hello rabbit WORLD",
+            new String[] {"<s>", "hello", "<unk>", "world", "</s>"},
+            new long[] {2_147_483_647L, 0, 800_000_000, 500, 2},
+            new long[] {1, 1, 1, 1, 1},
+            new long[] {0, 0, 0, 0, 0})),
+        Arguments.of("roberta-wordpieces", new EncodingExpectation(

Review Comment:
   Added the comment: the case covers wordpiece with RoBERTa special tokens, not byte-level BPE.



##########
opennlp-core/opennlp-ml/opennlp-dl/src/test/java/opennlp/dl/CreateTokenizerTest.java:
##########
@@ -151,7 +140,7 @@ void testDlEncodingPreservesVocabularyIds() {
         WordpieceTokenizer.BERT_UNK_TOKEN, 999,
         "hello", 42);
 
-    final Tokens tokens = new TestDL(vocab).encode("Hello");
+    final Tokens tokens = new ModelFreeDL(vocab, true).encode("Hello");

Review Comment:
   Removed here; CreateTokenizerTest covers creation only now.



##########
opennlp-core/opennlp-ml/opennlp-dl/src/main/java/opennlp/dl/Tokens.java:
##########
@@ -23,7 +23,7 @@
  * @param tokens The tokens themselves.
  * @param ids    The token IDs as retrieved from the vocabulary.
  * @param mask   The token mask. (Typically all 1.)

Review Comment:
   Done: "The attention mask, all 1."



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@opennlp.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] OPENNLP-1885: Test the shared deep-learning encoder and fix the token type javadoc (opennlp)

["krickert (via GitHub)" <gi...@apache.org>]

krickert commented on PR #1288:
URL: https://github.com/apache/opennlp/pull/1288#issuecomment-5639389766

   > This cannot be 1885 because it is already in the proposed M6 release candidate. It needs a new Jira.
   
   I'll get to this tonight


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@opennlp.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] OPENNLP-1942: Test the shared deep-learning encoder and fix the token type javadoc (opennlp)

["rzo1 (via GitHub)" <gi...@apache.org>]

rzo1 commented on code in PR #1288:
URL: https://github.com/apache/opennlp/pull/1288#discussion_r4034533996


##########
opennlp-core/opennlp-ml/opennlp-dl/src/test/java/opennlp/dl/SharedDlEncodingTest.java:
##########
@@ -0,0 +1,353 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package opennlp.dl;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.stream.Stream;
+
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
+
+import opennlp.tools.tokenize.WordpieceTokenizer;
+
+import static org.junit.jupiter.api.Assertions.assertArrayEquals;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertNotSame;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+/** Checks the complete token arrays created by the shared deep-learning encoder. */

Review Comment:
   "shared encoder" only makes sense with #1165 in mind. Please rename the class to `AbstractDLEncodeTokensTest` (or `EncodeTokensTest`) and refer to `AbstractDL#encodeTokens` in the Javadoc.



##########
opennlp-core/opennlp-ml/opennlp-dl/src/test/java/opennlp/dl/SharedDlEncodingTest.java:
##########
@@ -0,0 +1,353 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package opennlp.dl;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.stream.Stream;
+
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
+
+import opennlp.tools.tokenize.WordpieceTokenizer;
+
+import static org.junit.jupiter.api.Assertions.assertArrayEquals;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertNotSame;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+/** Checks the complete token arrays created by the shared deep-learning encoder. */
+public class SharedDlEncodingTest {
+
+  private static final int CONCURRENT_CALLS = 16;
+  private static final int CONCURRENT_WORKERS = 4;
+  private static final long WAIT_SECONDS = 10;
+
+  private static final Map<String, Integer> BERT_TOKEN_IDS = Map.ofEntries(
+      Map.entry(WordpieceTokenizer.BERT_CLS_TOKEN, 101),
+      Map.entry(WordpieceTokenizer.BERT_SEP_TOKEN, Integer.MAX_VALUE),
+      Map.entry(WordpieceTokenizer.BERT_UNK_TOKEN, 900_001),
+      Map.entry("hello", 42),
+      Map.entry("world", 1_500_000_000),
+      Map.entry("play", 0),
+      Map.entry("##ing", 2_147_483_646),
+      Map.entry("cafe", 88),
+      Map.entry("Caf\u00E9", 89),
+      Map.entry("\u03C3\u03BF\u03C6\u03BF\u03C2", 90),
+      Map.entry("\u03A3\u039F\u03A6\u039F\u03A3", 91));
+
+  private static final Map<String, Integer> ROBERTA_TOKEN_IDS = Map.of(
+      WordpieceTokenizer.ROBERTA_CLS_TOKEN, Integer.MAX_VALUE,
+      WordpieceTokenizer.ROBERTA_SEP_TOKEN, 2,
+      WordpieceTokenizer.ROBERTA_UNK_TOKEN, 800_000_000,
+      "hello", 0,
+      "world", 500,
+      "play", 10,
+      "##ing", 1_500_000_000);
+
+  private static final Map<String, Integer> ROBERTA_BERT_UNKNOWN_TOKEN_IDS = Map.of(
+      WordpieceTokenizer.ROBERTA_CLS_TOKEN, 71,
+      WordpieceTokenizer.ROBERTA_SEP_TOKEN, 72,
+      WordpieceTokenizer.BERT_UNK_TOKEN, Integer.MAX_VALUE,
+      "hello", 73);
+
+  private static final EncodingExpectation BERT_HELLO_EXPECTATION = new EncodingExpectation(
+      BERT_TOKEN_IDS, true, "Hello",
+      new String[] {"[CLS]", "hello", "[SEP]"},
+      new long[] {101, 42, 2_147_483_647L},

Review Comment:
   `2_147_483_647L` appears 23 times, and the ids in the expected arrays repeat the values from `BERT_TOKEN_IDS` / `ROBERTA_TOKEN_IDS` (101, 42, 900_001, ...). Please declare named constants (`BERT_CLS_ID`, `BERT_SEP_ID`, `BERT_UNK_ID`, ...) and use them in both the maps and the expectations.



##########
opennlp-core/opennlp-ml/opennlp-dl/src/test/java/opennlp/dl/ModelFreeDL.java:
##########
@@ -0,0 +1,54 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package opennlp.dl;
+
+import java.util.Map;
+
+/**
+ * An {@link AbstractDL} without an ONNX environment or session, so the shared encoder can be

Review Comment:
   Same here: "shared encoder" -> `AbstractDL#encodeTokens`.



##########
opennlp-core/opennlp-ml/opennlp-dl/src/test/java/opennlp/dl/SharedDlEncodingTest.java:
##########
@@ -0,0 +1,353 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package opennlp.dl;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.stream.Stream;
+
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
+
+import opennlp.tools.tokenize.WordpieceTokenizer;
+
+import static org.junit.jupiter.api.Assertions.assertArrayEquals;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertNotSame;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+/** Checks the complete token arrays created by the shared deep-learning encoder. */
+public class SharedDlEncodingTest {
+
+  private static final int CONCURRENT_CALLS = 16;
+  private static final int CONCURRENT_WORKERS = 4;
+  private static final long WAIT_SECONDS = 10;
+
+  private static final Map<String, Integer> BERT_TOKEN_IDS = Map.ofEntries(
+      Map.entry(WordpieceTokenizer.BERT_CLS_TOKEN, 101),
+      Map.entry(WordpieceTokenizer.BERT_SEP_TOKEN, Integer.MAX_VALUE),
+      Map.entry(WordpieceTokenizer.BERT_UNK_TOKEN, 900_001),
+      Map.entry("hello", 42),
+      Map.entry("world", 1_500_000_000),
+      Map.entry("play", 0),
+      Map.entry("##ing", 2_147_483_646),
+      Map.entry("cafe", 88),
+      Map.entry("Caf\u00E9", 89),
+      Map.entry("\u03C3\u03BF\u03C6\u03BF\u03C2", 90),
+      Map.entry("\u03A3\u039F\u03A6\u039F\u03A3", 91));
+
+  private static final Map<String, Integer> ROBERTA_TOKEN_IDS = Map.of(
+      WordpieceTokenizer.ROBERTA_CLS_TOKEN, Integer.MAX_VALUE,
+      WordpieceTokenizer.ROBERTA_SEP_TOKEN, 2,
+      WordpieceTokenizer.ROBERTA_UNK_TOKEN, 800_000_000,
+      "hello", 0,
+      "world", 500,
+      "play", 10,
+      "##ing", 1_500_000_000);
+
+  private static final Map<String, Integer> ROBERTA_BERT_UNKNOWN_TOKEN_IDS = Map.of(
+      WordpieceTokenizer.ROBERTA_CLS_TOKEN, 71,
+      WordpieceTokenizer.ROBERTA_SEP_TOKEN, 72,
+      WordpieceTokenizer.BERT_UNK_TOKEN, Integer.MAX_VALUE,
+      "hello", 73);
+
+  private static final EncodingExpectation BERT_HELLO_EXPECTATION = new EncodingExpectation(
+      BERT_TOKEN_IDS, true, "Hello",
+      new String[] {"[CLS]", "hello", "[SEP]"},
+      new long[] {101, 42, 2_147_483_647L},
+      new long[] {1, 1, 1},
+      new long[] {0, 0, 0});
+
+  private static final EncodingExpectation BERT_MIXED_EXPECTATION = new EncodingExpectation(
+      BERT_TOKEN_IDS, true, "Hello playing rabbit",
+      new String[] {"[CLS]", "hello", "play", "##ing", "[UNK]", "[SEP]"},
+      new long[] {101, 42, 0, 2_147_483_646L, 900_001, 2_147_483_647L},
+      new long[] {1, 1, 1, 1, 1, 1},
+      new long[] {0, 0, 0, 0, 0, 0});
+
+  private static final EncodingExpectation BERT_UNKNOWN_EXPECTATION = new EncodingExpectation(
+      BERT_TOKEN_IDS, true, "rabbit",
+      new String[] {"[CLS]", "[UNK]", "[SEP]"},
+      new long[] {101, 900_001, 2_147_483_647L},
+      new long[] {1, 1, 1},
+      new long[] {0, 0, 0});
+
+  private static final EncodingExpectation BERT_WORDPIECES_EXPECTATION =
+      new EncodingExpectation(
+          BERT_TOKEN_IDS, true, "Playing",
+          new String[] {"[CLS]", "play", "##ing", "[SEP]"},
+          new long[] {101, 0, 2_147_483_646L, 2_147_483_647L},
+          new long[] {1, 1, 1, 1},
+          new long[] {0, 0, 0, 0});
+
+  private static final List<EncodingExpectation> CONCURRENT_EXPECTATIONS = List.of(
+      BERT_HELLO_EXPECTATION,
+      BERT_UNKNOWN_EXPECTATION,
+      BERT_WORDPIECES_EXPECTATION);
+
+  /** Expected output for one model-free encoding case. */
+  private record EncodingExpectation(
+      Map<String, Integer> tokenIds,
+      boolean lowerCase,
+      CharSequence input,
+      String[] tokens,
+      long[] ids,
+      long[] mask,
+      long[] types) {
+  }
+
+  /**
+   * Confirms the token strings, ids, attention mask, and token types.
+   *
+   * @param name The case name.
+   * @param expected The fixture and expected arrays.
+   */
+  @ParameterizedTest(name = "{0}")
+  @MethodSource("encodingCases")
+  void testEncodingArrays(String name, EncodingExpectation expected) {
+    final Tokens actual = new ModelFreeDL(expected.tokenIds(), expected.lowerCase())
+        .encode(expected.input());
+
+    assertEncoding(expected, actual);
+  }
+
+  /** Confirms null text is rejected at the instance encoder boundary. */
+  @Test
+  void testRejectsNullText() {
+    final ModelFreeDL encoder = new ModelFreeDL(BERT_TOKEN_IDS, true);
+
+    final IllegalArgumentException exception = assertThrows(IllegalArgumentException.class,
+        () -> encoder.encode(null));
+    assertEquals("text must not be null", exception.getMessage());
+  }
+
+  /** Confirms that successive calls return independent arrays. */
+  @Test
+  void testRepeatedCallsReturnIndependentArrays() {
+    final ModelFreeDL encoder = new ModelFreeDL(BERT_TOKEN_IDS, true);
+    final Tokens first = encoder.encode("Hello");
+    final Tokens second = encoder.encode("Hello");
+
+    assertNotSame(first.tokens(), second.tokens());
+    assertNotSame(first.ids(), second.ids());
+    assertNotSame(first.mask(), second.mask());
+    assertNotSame(first.types(), second.types());
+
+    first.tokens()[1] = "changed";
+    first.ids()[1] = -1;
+    first.mask()[1] = 0;
+    first.types()[1] = 1;
+
+    assertEncoding(BERT_HELLO_EXPECTATION, second);
+    assertEncoding(BERT_HELLO_EXPECTATION, encoder.encode("Hello"));
+  }
+
+  /**
+   * Confirms bounded concurrent calls produce complete independent results.
+   *
+   * @throws InterruptedException Thrown if the test thread is interrupted.
+   * @throws ExecutionException Thrown if an encoding task fails.
+   * @throws TimeoutException Thrown if an encoding task exceeds the wait limit.
+   */
+  @Test
+  void testConcurrentCallsReturnIndependentArrays()

Review Comment:
   What does this test guard against? 16 tiny inputs on 4 threads will almost never race, so it cannot catch a thread-safety bug. `assertNotSame` across results adds nothing over `testRepeatedCallsReturnIndependentArrays`, and `assertTrue(awaitTermination)` in `finally` can hide the original failure. Please drop it or explain.



##########
opennlp-core/opennlp-ml/opennlp-dl/src/test/java/opennlp/dl/SharedDlEncodingTest.java:
##########
@@ -0,0 +1,353 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package opennlp.dl;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.stream.Stream;
+
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
+
+import opennlp.tools.tokenize.WordpieceTokenizer;
+
+import static org.junit.jupiter.api.Assertions.assertArrayEquals;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertNotSame;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+/** Checks the complete token arrays created by the shared deep-learning encoder. */
+public class SharedDlEncodingTest {
+
+  private static final int CONCURRENT_CALLS = 16;
+  private static final int CONCURRENT_WORKERS = 4;
+  private static final long WAIT_SECONDS = 10;
+
+  private static final Map<String, Integer> BERT_TOKEN_IDS = Map.ofEntries(
+      Map.entry(WordpieceTokenizer.BERT_CLS_TOKEN, 101),
+      Map.entry(WordpieceTokenizer.BERT_SEP_TOKEN, Integer.MAX_VALUE),
+      Map.entry(WordpieceTokenizer.BERT_UNK_TOKEN, 900_001),
+      Map.entry("hello", 42),
+      Map.entry("world", 1_500_000_000),
+      Map.entry("play", 0),
+      Map.entry("##ing", 2_147_483_646),
+      Map.entry("cafe", 88),
+      Map.entry("Caf\u00E9", 89),
+      Map.entry("\u03C3\u03BF\u03C6\u03BF\u03C2", 90),
+      Map.entry("\u03A3\u039F\u03A6\u039F\u03A3", 91));
+
+  private static final Map<String, Integer> ROBERTA_TOKEN_IDS = Map.of(
+      WordpieceTokenizer.ROBERTA_CLS_TOKEN, Integer.MAX_VALUE,
+      WordpieceTokenizer.ROBERTA_SEP_TOKEN, 2,
+      WordpieceTokenizer.ROBERTA_UNK_TOKEN, 800_000_000,
+      "hello", 0,
+      "world", 500,
+      "play", 10,
+      "##ing", 1_500_000_000);
+
+  private static final Map<String, Integer> ROBERTA_BERT_UNKNOWN_TOKEN_IDS = Map.of(
+      WordpieceTokenizer.ROBERTA_CLS_TOKEN, 71,
+      WordpieceTokenizer.ROBERTA_SEP_TOKEN, 72,
+      WordpieceTokenizer.BERT_UNK_TOKEN, Integer.MAX_VALUE,
+      "hello", 73);
+
+  private static final EncodingExpectation BERT_HELLO_EXPECTATION = new EncodingExpectation(
+      BERT_TOKEN_IDS, true, "Hello",
+      new String[] {"[CLS]", "hello", "[SEP]"},
+      new long[] {101, 42, 2_147_483_647L},
+      new long[] {1, 1, 1},
+      new long[] {0, 0, 0});
+
+  private static final EncodingExpectation BERT_MIXED_EXPECTATION = new EncodingExpectation(
+      BERT_TOKEN_IDS, true, "Hello playing rabbit",
+      new String[] {"[CLS]", "hello", "play", "##ing", "[UNK]", "[SEP]"},
+      new long[] {101, 42, 0, 2_147_483_646L, 900_001, 2_147_483_647L},
+      new long[] {1, 1, 1, 1, 1, 1},
+      new long[] {0, 0, 0, 0, 0, 0});
+
+  private static final EncodingExpectation BERT_UNKNOWN_EXPECTATION = new EncodingExpectation(
+      BERT_TOKEN_IDS, true, "rabbit",
+      new String[] {"[CLS]", "[UNK]", "[SEP]"},
+      new long[] {101, 900_001, 2_147_483_647L},
+      new long[] {1, 1, 1},
+      new long[] {0, 0, 0});
+
+  private static final EncodingExpectation BERT_WORDPIECES_EXPECTATION =
+      new EncodingExpectation(
+          BERT_TOKEN_IDS, true, "Playing",
+          new String[] {"[CLS]", "play", "##ing", "[SEP]"},
+          new long[] {101, 0, 2_147_483_646L, 2_147_483_647L},
+          new long[] {1, 1, 1, 1},
+          new long[] {0, 0, 0, 0});
+
+  private static final List<EncodingExpectation> CONCURRENT_EXPECTATIONS = List.of(
+      BERT_HELLO_EXPECTATION,
+      BERT_UNKNOWN_EXPECTATION,
+      BERT_WORDPIECES_EXPECTATION);
+
+  /** Expected output for one model-free encoding case. */
+  private record EncodingExpectation(
+      Map<String, Integer> tokenIds,
+      boolean lowerCase,
+      CharSequence input,
+      String[] tokens,
+      long[] ids,
+      long[] mask,
+      long[] types) {
+  }
+
+  /**
+   * Confirms the token strings, ids, attention mask, and token types.
+   *
+   * @param name The case name.
+   * @param expected The fixture and expected arrays.
+   */
+  @ParameterizedTest(name = "{0}")
+  @MethodSource("encodingCases")
+  void testEncodingArrays(String name, EncodingExpectation expected) {
+    final Tokens actual = new ModelFreeDL(expected.tokenIds(), expected.lowerCase())
+        .encode(expected.input());
+
+    assertEncoding(expected, actual);
+  }
+
+  /** Confirms null text is rejected at the instance encoder boundary. */
+  @Test
+  void testRejectsNullText() {
+    final ModelFreeDL encoder = new ModelFreeDL(BERT_TOKEN_IDS, true);
+
+    final IllegalArgumentException exception = assertThrows(IllegalArgumentException.class,
+        () -> encoder.encode(null));
+    assertEquals("text must not be null", exception.getMessage());
+  }
+
+  /** Confirms that successive calls return independent arrays. */
+  @Test
+  void testRepeatedCallsReturnIndependentArrays() {
+    final ModelFreeDL encoder = new ModelFreeDL(BERT_TOKEN_IDS, true);
+    final Tokens first = encoder.encode("Hello");
+    final Tokens second = encoder.encode("Hello");
+
+    assertNotSame(first.tokens(), second.tokens());
+    assertNotSame(first.ids(), second.ids());
+    assertNotSame(first.mask(), second.mask());
+    assertNotSame(first.types(), second.types());
+
+    first.tokens()[1] = "changed";
+    first.ids()[1] = -1;
+    first.mask()[1] = 0;
+    first.types()[1] = 1;
+
+    assertEncoding(BERT_HELLO_EXPECTATION, second);
+    assertEncoding(BERT_HELLO_EXPECTATION, encoder.encode("Hello"));
+  }
+
+  /**
+   * Confirms bounded concurrent calls produce complete independent results.
+   *
+   * @throws InterruptedException Thrown if the test thread is interrupted.
+   * @throws ExecutionException Thrown if an encoding task fails.
+   * @throws TimeoutException Thrown if an encoding task exceeds the wait limit.
+   */
+  @Test
+  void testConcurrentCallsReturnIndependentArrays()
+      throws InterruptedException, ExecutionException, TimeoutException {
+    final ModelFreeDL encoder = new ModelFreeDL(BERT_TOKEN_IDS, true);
+    final ExecutorService executor = Executors.newFixedThreadPool(CONCURRENT_WORKERS);
+    final CountDownLatch start = new CountDownLatch(1);
+    final List<Future<Tokens>> futures = new ArrayList<>();
+    try {
+      for (int i = 0; i < CONCURRENT_CALLS; i++) {
+        final EncodingExpectation expected =
+            CONCURRENT_EXPECTATIONS.get(i % CONCURRENT_EXPECTATIONS.size());
+        futures.add(executor.submit(() -> {
+          start.await();
+          return encoder.encode(expected.input());
+        }));
+      }
+      start.countDown();
+
+      final List<Tokens> completed = new ArrayList<>();
+      for (int i = 0; i < futures.size(); i++) {
+        final Future<Tokens> future = futures.get(i);
+        final Tokens actual = future.get(WAIT_SECONDS, TimeUnit.SECONDS);
+        assertEncoding(CONCURRENT_EXPECTATIONS.get(i % CONCURRENT_EXPECTATIONS.size()), actual);
+        for (final Tokens prior : completed) {
+          assertNotSame(prior.tokens(), actual.tokens());
+          assertNotSame(prior.ids(), actual.ids());
+          assertNotSame(prior.mask(), actual.mask());
+          assertNotSame(prior.types(), actual.types());
+        }
+        completed.add(actual);
+      }
+    } finally {
+      executor.shutdownNow();
+      assertTrue(executor.awaitTermination(WAIT_SECONDS, TimeUnit.SECONDS));
+    }
+  }
+
+  /**
+   * Supplies model-free encoding fixtures.
+   *
+   * @return The named fixtures.
+   */
+  private static Stream<Arguments> encodingCases() {
+    return Stream.of(
+        Arguments.of("bert-empty", new EncodingExpectation(BERT_TOKEN_IDS, true, "",
+            new String[] {"[CLS]", "[SEP]"},
+            new long[] {101, 2_147_483_647L},
+            new long[] {1, 1},
+            new long[] {0, 0})),
+        Arguments.of("bert-ascii-whitespace", new EncodingExpectation(
+            BERT_TOKEN_IDS, true, " \t\n",
+            new String[] {"[CLS]", "[SEP]"},
+            new long[] {101, 2_147_483_647L},
+            new long[] {1, 1},
+            new long[] {0, 0})),
+        Arguments.of("bert-unicode-whitespace", new EncodingExpectation(
+            BERT_TOKEN_IDS, true, "\u00A0\u2028\u3000",
+            new String[] {"[CLS]", "[SEP]"},
+            new long[] {101, 2_147_483_647L},
+            new long[] {1, 1},
+            new long[] {0, 0})),
+        Arguments.of("bert-known", BERT_HELLO_EXPECTATION),
+        Arguments.of("bert-known-sequence", new EncodingExpectation(
+            BERT_TOKEN_IDS, true, "Hello WORLD",
+            new String[] {"[CLS]", "hello", "world", "[SEP]"},
+            new long[] {101, 42, 1_500_000_000L, 2_147_483_647L},
+            new long[] {1, 1, 1, 1},
+            new long[] {0, 0, 0, 0})),
+        Arguments.of("bert-unicode-separator", new EncodingExpectation(
+            BERT_TOKEN_IDS, true, "Hello\u00A0WORLD",
+            new String[] {"[CLS]", "hello", "world", "[SEP]"},
+            new long[] {101, 42, 1_500_000_000L, 2_147_483_647L},
+            new long[] {1, 1, 1, 1},
+            new long[] {0, 0, 0, 0})),
+        Arguments.of("bert-unknown", BERT_UNKNOWN_EXPECTATION),
+        Arguments.of("bert-known-unknown", new EncodingExpectation(
+            BERT_TOKEN_IDS, true, "Hello rabbit",
+            new String[] {"[CLS]", "hello", "[UNK]", "[SEP]"},
+            new long[] {101, 42, 900_001, 2_147_483_647L},
+            new long[] {1, 1, 1, 1},
+            new long[] {0, 0, 0, 0})),
+        Arguments.of("bert-wordpieces", BERT_WORDPIECES_EXPECTATION),
+        Arguments.of("bert-mixed", BERT_MIXED_EXPECTATION),
+        Arguments.of("bert-string-builder", new EncodingExpectation(
+            BERT_TOKEN_IDS, true, new StringBuilder("Hello"),
+            new String[] {"[CLS]", "hello", "[SEP]"},
+            new long[] {101, 42, 2_147_483_647L},
+            new long[] {1, 1, 1},
+            new long[] {0, 0, 0})),
+        Arguments.of("bert-cased-accent", new EncodingExpectation(
+            BERT_TOKEN_IDS, false, "Caf\u00E9",
+            new String[] {"[CLS]", "Caf\u00E9", "[SEP]"},
+            new long[] {101, 89, 2_147_483_647L},
+            new long[] {1, 1, 1},
+            new long[] {0, 0, 0})),
+        Arguments.of("bert-cased-miss", new EncodingExpectation(
+            BERT_TOKEN_IDS, false, "Hello",
+            new String[] {"[CLS]", "[UNK]", "[SEP]"},
+            new long[] {101, 900_001, 2_147_483_647L},
+            new long[] {1, 1, 1},
+            new long[] {0, 0, 0})),
+        Arguments.of("bert-precomposed-accent", new EncodingExpectation(
+            BERT_TOKEN_IDS, true, "CAF\u00C9",
+            new String[] {"[CLS]", "cafe", "[SEP]"},
+            new long[] {101, 88, 2_147_483_647L},
+            new long[] {1, 1, 1},
+            new long[] {0, 0, 0})),
+        Arguments.of("bert-decomposed-accent", new EncodingExpectation(
+            BERT_TOKEN_IDS, true, "Cafe\u0301",
+            new String[] {"[CLS]", "cafe", "[SEP]"},
+            new long[] {101, 88, 2_147_483_647L},
+            new long[] {1, 1, 1},
+            new long[] {0, 0, 0})),
+        Arguments.of("bert-final-sigma", new EncodingExpectation(
+            BERT_TOKEN_IDS, true, "\u03A3\u039F\u03A6\u039F\u03A3",
+            new String[] {"[CLS]", "\u03C3\u03BF\u03C6\u03BF\u03C2", "[SEP]"},
+            new long[] {101, 90, 2_147_483_647L},
+            new long[] {1, 1, 1},
+            new long[] {0, 0, 0})),
+        Arguments.of("bert-cased-sigma", new EncodingExpectation(
+            BERT_TOKEN_IDS, false, "\u03A3\u039F\u03A6\u039F\u03A3",
+            new String[] {"[CLS]", "\u03A3\u039F\u03A6\u039F\u03A3", "[SEP]"},
+            new long[] {101, 91, 2_147_483_647L},
+            new long[] {1, 1, 1},
+            new long[] {0, 0, 0})),
+        Arguments.of("roberta-empty", new EncodingExpectation(
+            ROBERTA_TOKEN_IDS, true, "",
+            new String[] {"<s>", "</s>"},
+            new long[] {2_147_483_647L, 2},
+            new long[] {1, 1},
+            new long[] {0, 0})),
+        Arguments.of("roberta-known", new EncodingExpectation(
+            ROBERTA_TOKEN_IDS, true, "Hello",
+            new String[] {"<s>", "hello", "</s>"},
+            new long[] {2_147_483_647L, 0, 2},
+            new long[] {1, 1, 1},
+            new long[] {0, 0, 0})),
+        Arguments.of("roberta-unknown", new EncodingExpectation(
+            ROBERTA_TOKEN_IDS, true, "rabbit",
+            new String[] {"<s>", "<unk>", "</s>"},
+            new long[] {2_147_483_647L, 800_000_000, 2},
+            new long[] {1, 1, 1},
+            new long[] {0, 0, 0})),
+        Arguments.of("roberta-mixed", new EncodingExpectation(
+            ROBERTA_TOKEN_IDS, true, "Hello rabbit WORLD",
+            new String[] {"<s>", "hello", "<unk>", "world", "</s>"},
+            new long[] {2_147_483_647L, 0, 800_000_000, 500, 2},
+            new long[] {1, 1, 1, 1, 1},
+            new long[] {0, 0, 0, 0, 0})),
+        Arguments.of("roberta-wordpieces", new EncodingExpectation(

Review Comment:
   RoBERTa uses byte-level BPE, not `##` pieces. This case pins current behavior, not RoBERTa semantics. Please add a one-line comment saying that, or drop the case.



##########
opennlp-core/opennlp-ml/opennlp-dl/src/test/java/opennlp/dl/CreateTokenizerTest.java:
##########
@@ -151,7 +140,7 @@ void testDlEncodingPreservesVocabularyIds() {
         WordpieceTokenizer.BERT_UNK_TOKEN, 999,
         "hello", 42);
 
-    final Tokens tokens = new TestDL(vocab).encode("Hello");
+    final Tokens tokens = new ModelFreeDL(vocab, true).encode("Hello");

Review Comment:
   This duplicates the `bert-known` case in `SharedDlEncodingTest`. Please move it there or remove it, so encoding is tested in one place and this class covers creation only.



##########
opennlp-core/opennlp-ml/opennlp-dl/src/test/java/opennlp/dl/SharedDlEncodingTest.java:
##########
@@ -0,0 +1,353 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package opennlp.dl;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.stream.Stream;
+
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
+
+import opennlp.tools.tokenize.WordpieceTokenizer;
+
+import static org.junit.jupiter.api.Assertions.assertArrayEquals;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertNotSame;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+/** Checks the complete token arrays created by the shared deep-learning encoder. */
+public class SharedDlEncodingTest {
+
+  private static final int CONCURRENT_CALLS = 16;
+  private static final int CONCURRENT_WORKERS = 4;
+  private static final long WAIT_SECONDS = 10;
+
+  private static final Map<String, Integer> BERT_TOKEN_IDS = Map.ofEntries(
+      Map.entry(WordpieceTokenizer.BERT_CLS_TOKEN, 101),
+      Map.entry(WordpieceTokenizer.BERT_SEP_TOKEN, Integer.MAX_VALUE),
+      Map.entry(WordpieceTokenizer.BERT_UNK_TOKEN, 900_001),
+      Map.entry("hello", 42),
+      Map.entry("world", 1_500_000_000),
+      Map.entry("play", 0),
+      Map.entry("##ing", 2_147_483_646),
+      Map.entry("cafe", 88),
+      Map.entry("Caf\u00E9", 89),
+      Map.entry("\u03C3\u03BF\u03C6\u03BF\u03C2", 90),
+      Map.entry("\u03A3\u039F\u03A6\u039F\u03A3", 91));
+
+  private static final Map<String, Integer> ROBERTA_TOKEN_IDS = Map.of(
+      WordpieceTokenizer.ROBERTA_CLS_TOKEN, Integer.MAX_VALUE,
+      WordpieceTokenizer.ROBERTA_SEP_TOKEN, 2,
+      WordpieceTokenizer.ROBERTA_UNK_TOKEN, 800_000_000,
+      "hello", 0,
+      "world", 500,
+      "play", 10,
+      "##ing", 1_500_000_000);
+
+  private static final Map<String, Integer> ROBERTA_BERT_UNKNOWN_TOKEN_IDS = Map.of(
+      WordpieceTokenizer.ROBERTA_CLS_TOKEN, 71,
+      WordpieceTokenizer.ROBERTA_SEP_TOKEN, 72,
+      WordpieceTokenizer.BERT_UNK_TOKEN, Integer.MAX_VALUE,
+      "hello", 73);
+
+  private static final EncodingExpectation BERT_HELLO_EXPECTATION = new EncodingExpectation(
+      BERT_TOKEN_IDS, true, "Hello",
+      new String[] {"[CLS]", "hello", "[SEP]"},
+      new long[] {101, 42, 2_147_483_647L},
+      new long[] {1, 1, 1},
+      new long[] {0, 0, 0});
+
+  private static final EncodingExpectation BERT_MIXED_EXPECTATION = new EncodingExpectation(
+      BERT_TOKEN_IDS, true, "Hello playing rabbit",
+      new String[] {"[CLS]", "hello", "play", "##ing", "[UNK]", "[SEP]"},
+      new long[] {101, 42, 0, 2_147_483_646L, 900_001, 2_147_483_647L},
+      new long[] {1, 1, 1, 1, 1, 1},
+      new long[] {0, 0, 0, 0, 0, 0});
+
+  private static final EncodingExpectation BERT_UNKNOWN_EXPECTATION = new EncodingExpectation(
+      BERT_TOKEN_IDS, true, "rabbit",
+      new String[] {"[CLS]", "[UNK]", "[SEP]"},
+      new long[] {101, 900_001, 2_147_483_647L},
+      new long[] {1, 1, 1},
+      new long[] {0, 0, 0});
+
+  private static final EncodingExpectation BERT_WORDPIECES_EXPECTATION =
+      new EncodingExpectation(
+          BERT_TOKEN_IDS, true, "Playing",
+          new String[] {"[CLS]", "play", "##ing", "[SEP]"},
+          new long[] {101, 0, 2_147_483_646L, 2_147_483_647L},
+          new long[] {1, 1, 1, 1},
+          new long[] {0, 0, 0, 0});
+
+  private static final List<EncodingExpectation> CONCURRENT_EXPECTATIONS = List.of(
+      BERT_HELLO_EXPECTATION,
+      BERT_UNKNOWN_EXPECTATION,
+      BERT_WORDPIECES_EXPECTATION);
+
+  /** Expected output for one model-free encoding case. */
+  private record EncodingExpectation(
+      Map<String, Integer> tokenIds,
+      boolean lowerCase,
+      CharSequence input,
+      String[] tokens,
+      long[] ids,
+      long[] mask,
+      long[] types) {
+  }
+
+  /**
+   * Confirms the token strings, ids, attention mask, and token types.
+   *
+   * @param name The case name.
+   * @param expected The fixture and expected arrays.
+   */
+  @ParameterizedTest(name = "{0}")
+  @MethodSource("encodingCases")
+  void testEncodingArrays(String name, EncodingExpectation expected) {
+    final Tokens actual = new ModelFreeDL(expected.tokenIds(), expected.lowerCase())
+        .encode(expected.input());
+
+    assertEncoding(expected, actual);
+  }
+
+  /** Confirms null text is rejected at the instance encoder boundary. */
+  @Test
+  void testRejectsNullText() {
+    final ModelFreeDL encoder = new ModelFreeDL(BERT_TOKEN_IDS, true);
+
+    final IllegalArgumentException exception = assertThrows(IllegalArgumentException.class,
+        () -> encoder.encode(null));
+    assertEquals("text must not be null", exception.getMessage());
+  }
+
+  /** Confirms that successive calls return independent arrays. */
+  @Test
+  void testRepeatedCallsReturnIndependentArrays() {
+    final ModelFreeDL encoder = new ModelFreeDL(BERT_TOKEN_IDS, true);
+    final Tokens first = encoder.encode("Hello");
+    final Tokens second = encoder.encode("Hello");
+
+    assertNotSame(first.tokens(), second.tokens());
+    assertNotSame(first.ids(), second.ids());
+    assertNotSame(first.mask(), second.mask());
+    assertNotSame(first.types(), second.types());
+
+    first.tokens()[1] = "changed";
+    first.ids()[1] = -1;
+    first.mask()[1] = 0;
+    first.types()[1] = 1;
+
+    assertEncoding(BERT_HELLO_EXPECTATION, second);
+    assertEncoding(BERT_HELLO_EXPECTATION, encoder.encode("Hello"));
+  }
+
+  /**
+   * Confirms bounded concurrent calls produce complete independent results.
+   *
+   * @throws InterruptedException Thrown if the test thread is interrupted.
+   * @throws ExecutionException Thrown if an encoding task fails.
+   * @throws TimeoutException Thrown if an encoding task exceeds the wait limit.
+   */
+  @Test
+  void testConcurrentCallsReturnIndependentArrays()
+      throws InterruptedException, ExecutionException, TimeoutException {
+    final ModelFreeDL encoder = new ModelFreeDL(BERT_TOKEN_IDS, true);
+    final ExecutorService executor = Executors.newFixedThreadPool(CONCURRENT_WORKERS);
+    final CountDownLatch start = new CountDownLatch(1);
+    final List<Future<Tokens>> futures = new ArrayList<>();
+    try {
+      for (int i = 0; i < CONCURRENT_CALLS; i++) {
+        final EncodingExpectation expected =
+            CONCURRENT_EXPECTATIONS.get(i % CONCURRENT_EXPECTATIONS.size());
+        futures.add(executor.submit(() -> {
+          start.await();
+          return encoder.encode(expected.input());
+        }));
+      }
+      start.countDown();
+
+      final List<Tokens> completed = new ArrayList<>();
+      for (int i = 0; i < futures.size(); i++) {
+        final Future<Tokens> future = futures.get(i);
+        final Tokens actual = future.get(WAIT_SECONDS, TimeUnit.SECONDS);
+        assertEncoding(CONCURRENT_EXPECTATIONS.get(i % CONCURRENT_EXPECTATIONS.size()), actual);
+        for (final Tokens prior : completed) {
+          assertNotSame(prior.tokens(), actual.tokens());
+          assertNotSame(prior.ids(), actual.ids());
+          assertNotSame(prior.mask(), actual.mask());
+          assertNotSame(prior.types(), actual.types());
+        }
+        completed.add(actual);
+      }
+    } finally {
+      executor.shutdownNow();
+      assertTrue(executor.awaitTermination(WAIT_SECONDS, TimeUnit.SECONDS));
+    }
+  }
+
+  /**
+   * Supplies model-free encoding fixtures.
+   *
+   * @return The named fixtures.
+   */
+  private static Stream<Arguments> encodingCases() {
+    return Stream.of(
+        Arguments.of("bert-empty", new EncodingExpectation(BERT_TOKEN_IDS, true, "",
+            new String[] {"[CLS]", "[SEP]"},
+            new long[] {101, 2_147_483_647L},
+            new long[] {1, 1},
+            new long[] {0, 0})),
+        Arguments.of("bert-ascii-whitespace", new EncodingExpectation(
+            BERT_TOKEN_IDS, true, " \t\n",
+            new String[] {"[CLS]", "[SEP]"},
+            new long[] {101, 2_147_483_647L},
+            new long[] {1, 1},
+            new long[] {0, 0})),
+        Arguments.of("bert-unicode-whitespace", new EncodingExpectation(
+            BERT_TOKEN_IDS, true, "\u00A0\u2028\u3000",
+            new String[] {"[CLS]", "[SEP]"},
+            new long[] {101, 2_147_483_647L},
+            new long[] {1, 1},
+            new long[] {0, 0})),
+        Arguments.of("bert-known", BERT_HELLO_EXPECTATION),
+        Arguments.of("bert-known-sequence", new EncodingExpectation(
+            BERT_TOKEN_IDS, true, "Hello WORLD",
+            new String[] {"[CLS]", "hello", "world", "[SEP]"},
+            new long[] {101, 42, 1_500_000_000L, 2_147_483_647L},
+            new long[] {1, 1, 1, 1},
+            new long[] {0, 0, 0, 0})),
+        Arguments.of("bert-unicode-separator", new EncodingExpectation(
+            BERT_TOKEN_IDS, true, "Hello\u00A0WORLD",
+            new String[] {"[CLS]", "hello", "world", "[SEP]"},
+            new long[] {101, 42, 1_500_000_000L, 2_147_483_647L},
+            new long[] {1, 1, 1, 1},
+            new long[] {0, 0, 0, 0})),
+        Arguments.of("bert-unknown", BERT_UNKNOWN_EXPECTATION),
+        Arguments.of("bert-known-unknown", new EncodingExpectation(
+            BERT_TOKEN_IDS, true, "Hello rabbit",
+            new String[] {"[CLS]", "hello", "[UNK]", "[SEP]"},
+            new long[] {101, 42, 900_001, 2_147_483_647L},
+            new long[] {1, 1, 1, 1},
+            new long[] {0, 0, 0, 0})),
+        Arguments.of("bert-wordpieces", BERT_WORDPIECES_EXPECTATION),
+        Arguments.of("bert-mixed", BERT_MIXED_EXPECTATION),
+        Arguments.of("bert-string-builder", new EncodingExpectation(
+            BERT_TOKEN_IDS, true, new StringBuilder("Hello"),
+            new String[] {"[CLS]", "hello", "[SEP]"},
+            new long[] {101, 42, 2_147_483_647L},
+            new long[] {1, 1, 1},
+            new long[] {0, 0, 0})),
+        Arguments.of("bert-cased-accent", new EncodingExpectation(
+            BERT_TOKEN_IDS, false, "Caf\u00E9",
+            new String[] {"[CLS]", "Caf\u00E9", "[SEP]"},
+            new long[] {101, 89, 2_147_483_647L},
+            new long[] {1, 1, 1},
+            new long[] {0, 0, 0})),
+        Arguments.of("bert-cased-miss", new EncodingExpectation(
+            BERT_TOKEN_IDS, false, "Hello",
+            new String[] {"[CLS]", "[UNK]", "[SEP]"},
+            new long[] {101, 900_001, 2_147_483_647L},
+            new long[] {1, 1, 1},
+            new long[] {0, 0, 0})),
+        Arguments.of("bert-precomposed-accent", new EncodingExpectation(
+            BERT_TOKEN_IDS, true, "CAF\u00C9",
+            new String[] {"[CLS]", "cafe", "[SEP]"},
+            new long[] {101, 88, 2_147_483_647L},
+            new long[] {1, 1, 1},
+            new long[] {0, 0, 0})),
+        Arguments.of("bert-decomposed-accent", new EncodingExpectation(
+            BERT_TOKEN_IDS, true, "Cafe\u0301",
+            new String[] {"[CLS]", "cafe", "[SEP]"},
+            new long[] {101, 88, 2_147_483_647L},
+            new long[] {1, 1, 1},
+            new long[] {0, 0, 0})),
+        Arguments.of("bert-final-sigma", new EncodingExpectation(
+            BERT_TOKEN_IDS, true, "\u03A3\u039F\u03A6\u039F\u03A3",
+            new String[] {"[CLS]", "\u03C3\u03BF\u03C6\u03BF\u03C2", "[SEP]"},
+            new long[] {101, 90, 2_147_483_647L},
+            new long[] {1, 1, 1},
+            new long[] {0, 0, 0})),
+        Arguments.of("bert-cased-sigma", new EncodingExpectation(
+            BERT_TOKEN_IDS, false, "\u03A3\u039F\u03A6\u039F\u03A3",
+            new String[] {"[CLS]", "\u03A3\u039F\u03A6\u039F\u03A3", "[SEP]"},
+            new long[] {101, 91, 2_147_483_647L},
+            new long[] {1, 1, 1},
+            new long[] {0, 0, 0})),
+        Arguments.of("roberta-empty", new EncodingExpectation(
+            ROBERTA_TOKEN_IDS, true, "",
+            new String[] {"<s>", "</s>"},
+            new long[] {2_147_483_647L, 2},
+            new long[] {1, 1},
+            new long[] {0, 0})),
+        Arguments.of("roberta-known", new EncodingExpectation(
+            ROBERTA_TOKEN_IDS, true, "Hello",
+            new String[] {"<s>", "hello", "</s>"},
+            new long[] {2_147_483_647L, 0, 2},
+            new long[] {1, 1, 1},
+            new long[] {0, 0, 0})),
+        Arguments.of("roberta-unknown", new EncodingExpectation(
+            ROBERTA_TOKEN_IDS, true, "rabbit",
+            new String[] {"<s>", "<unk>", "</s>"},
+            new long[] {2_147_483_647L, 800_000_000, 2},
+            new long[] {1, 1, 1},
+            new long[] {0, 0, 0})),
+        Arguments.of("roberta-mixed", new EncodingExpectation(
+            ROBERTA_TOKEN_IDS, true, "Hello rabbit WORLD",
+            new String[] {"<s>", "hello", "<unk>", "world", "</s>"},
+            new long[] {2_147_483_647L, 0, 800_000_000, 500, 2},
+            new long[] {1, 1, 1, 1, 1},
+            new long[] {0, 0, 0, 0, 0})),
+        Arguments.of("roberta-wordpieces", new EncodingExpectation(
+            ROBERTA_TOKEN_IDS, true, "Playing",
+            new String[] {"<s>", "play", "##ing", "</s>"},
+            new long[] {2_147_483_647L, 10, 1_500_000_000L, 2},
+            new long[] {1, 1, 1, 1},
+            new long[] {0, 0, 0, 0})),
+        Arguments.of("roberta-bert-unknown", new EncodingExpectation(
+            ROBERTA_BERT_UNKNOWN_TOKEN_IDS, true, "Hello rabbit",
+            new String[] {"<s>", "hello", "[UNK]", "</s>"},
+            new long[] {71, 73, 2_147_483_647L, 72},
+            new long[] {1, 1, 1, 1},
+            new long[] {0, 0, 0, 0})));
+  }
+
+  /**
+   * Compares all model input arrays.
+   *
+   * @param expected The expected arrays.
+   * @param actual The encoded arrays.
+   */
+  private void assertEncoding(EncodingExpectation expected, Tokens actual) {

Review Comment:
   The encoder always writes mask all 1 and types all 0, but every case spells out both arrays. Please derive them from `tokens.length` here (or in the record) and drop them from the fixtures. That removes ~50 lines and leaves only what varies per case.



##########
opennlp-core/opennlp-ml/opennlp-dl/src/main/java/opennlp/dl/Tokens.java:
##########
@@ -23,7 +23,7 @@
  * @param tokens The tokens themselves.
  * @param ids    The token IDs as retrieved from the vocabulary.
  * @param mask   The token mask. (Typically all 1.)

Review Comment:
   Please align with `types`: "The attention mask, all 1."



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@opennlp.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] OPENNLP-1942: Test the shared deep-learning encoder and fix the token type javadoc (opennlp)

["krickert (via GitHub)" <gi...@apache.org>]

krickert commented on code in PR #1288:
URL: https://github.com/apache/opennlp/pull/1288#discussion_r4043545592


##########
opennlp-core/opennlp-ml/opennlp-dl/src/test/java/opennlp/dl/SharedDlEncodingTest.java:
##########
@@ -0,0 +1,353 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package opennlp.dl;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.stream.Stream;
+
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
+
+import opennlp.tools.tokenize.WordpieceTokenizer;
+
+import static org.junit.jupiter.api.Assertions.assertArrayEquals;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertNotSame;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+/** Checks the complete token arrays created by the shared deep-learning encoder. */

Review Comment:
   Renamed to AbstractDLEncodeTokensTest; the class Javadoc names AbstractDL#encodeTokens.



##########
opennlp-core/opennlp-ml/opennlp-dl/src/test/java/opennlp/dl/ModelFreeDL.java:
##########
@@ -0,0 +1,54 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package opennlp.dl;
+
+import java.util.Map;
+
+/**
+ * An {@link AbstractDL} without an ONNX environment or session, so the shared encoder can be

Review Comment:
   Done, the Javadoc names AbstractDL#encodeTokens now.



##########
opennlp-core/opennlp-ml/opennlp-dl/src/test/java/opennlp/dl/SharedDlEncodingTest.java:
##########
@@ -0,0 +1,353 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package opennlp.dl;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.stream.Stream;
+
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
+
+import opennlp.tools.tokenize.WordpieceTokenizer;
+
+import static org.junit.jupiter.api.Assertions.assertArrayEquals;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertNotSame;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+/** Checks the complete token arrays created by the shared deep-learning encoder. */
+public class SharedDlEncodingTest {
+
+  private static final int CONCURRENT_CALLS = 16;
+  private static final int CONCURRENT_WORKERS = 4;
+  private static final long WAIT_SECONDS = 10;
+
+  private static final Map<String, Integer> BERT_TOKEN_IDS = Map.ofEntries(
+      Map.entry(WordpieceTokenizer.BERT_CLS_TOKEN, 101),
+      Map.entry(WordpieceTokenizer.BERT_SEP_TOKEN, Integer.MAX_VALUE),
+      Map.entry(WordpieceTokenizer.BERT_UNK_TOKEN, 900_001),
+      Map.entry("hello", 42),
+      Map.entry("world", 1_500_000_000),
+      Map.entry("play", 0),
+      Map.entry("##ing", 2_147_483_646),
+      Map.entry("cafe", 88),
+      Map.entry("Caf\u00E9", 89),
+      Map.entry("\u03C3\u03BF\u03C6\u03BF\u03C2", 90),
+      Map.entry("\u03A3\u039F\u03A6\u039F\u03A3", 91));
+
+  private static final Map<String, Integer> ROBERTA_TOKEN_IDS = Map.of(
+      WordpieceTokenizer.ROBERTA_CLS_TOKEN, Integer.MAX_VALUE,
+      WordpieceTokenizer.ROBERTA_SEP_TOKEN, 2,
+      WordpieceTokenizer.ROBERTA_UNK_TOKEN, 800_000_000,
+      "hello", 0,
+      "world", 500,
+      "play", 10,
+      "##ing", 1_500_000_000);
+
+  private static final Map<String, Integer> ROBERTA_BERT_UNKNOWN_TOKEN_IDS = Map.of(
+      WordpieceTokenizer.ROBERTA_CLS_TOKEN, 71,
+      WordpieceTokenizer.ROBERTA_SEP_TOKEN, 72,
+      WordpieceTokenizer.BERT_UNK_TOKEN, Integer.MAX_VALUE,
+      "hello", 73);
+
+  private static final EncodingExpectation BERT_HELLO_EXPECTATION = new EncodingExpectation(
+      BERT_TOKEN_IDS, true, "Hello",
+      new String[] {"[CLS]", "hello", "[SEP]"},
+      new long[] {101, 42, 2_147_483_647L},

Review Comment:
   Done. The ids are named constants (BERT_CLS_ID, BERT_SEP_ID, BERT_UNK_ID, ...) used in both the token maps and the expected arrays.



##########
opennlp-core/opennlp-ml/opennlp-dl/src/test/java/opennlp/dl/SharedDlEncodingTest.java:
##########
@@ -0,0 +1,353 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package opennlp.dl;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.stream.Stream;
+
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
+
+import opennlp.tools.tokenize.WordpieceTokenizer;
+
+import static org.junit.jupiter.api.Assertions.assertArrayEquals;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertNotSame;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+/** Checks the complete token arrays created by the shared deep-learning encoder. */
+public class SharedDlEncodingTest {
+
+  private static final int CONCURRENT_CALLS = 16;
+  private static final int CONCURRENT_WORKERS = 4;
+  private static final long WAIT_SECONDS = 10;
+
+  private static final Map<String, Integer> BERT_TOKEN_IDS = Map.ofEntries(
+      Map.entry(WordpieceTokenizer.BERT_CLS_TOKEN, 101),
+      Map.entry(WordpieceTokenizer.BERT_SEP_TOKEN, Integer.MAX_VALUE),
+      Map.entry(WordpieceTokenizer.BERT_UNK_TOKEN, 900_001),
+      Map.entry("hello", 42),
+      Map.entry("world", 1_500_000_000),
+      Map.entry("play", 0),
+      Map.entry("##ing", 2_147_483_646),
+      Map.entry("cafe", 88),
+      Map.entry("Caf\u00E9", 89),
+      Map.entry("\u03C3\u03BF\u03C6\u03BF\u03C2", 90),
+      Map.entry("\u03A3\u039F\u03A6\u039F\u03A3", 91));
+
+  private static final Map<String, Integer> ROBERTA_TOKEN_IDS = Map.of(
+      WordpieceTokenizer.ROBERTA_CLS_TOKEN, Integer.MAX_VALUE,
+      WordpieceTokenizer.ROBERTA_SEP_TOKEN, 2,
+      WordpieceTokenizer.ROBERTA_UNK_TOKEN, 800_000_000,
+      "hello", 0,
+      "world", 500,
+      "play", 10,
+      "##ing", 1_500_000_000);
+
+  private static final Map<String, Integer> ROBERTA_BERT_UNKNOWN_TOKEN_IDS = Map.of(
+      WordpieceTokenizer.ROBERTA_CLS_TOKEN, 71,
+      WordpieceTokenizer.ROBERTA_SEP_TOKEN, 72,
+      WordpieceTokenizer.BERT_UNK_TOKEN, Integer.MAX_VALUE,
+      "hello", 73);
+
+  private static final EncodingExpectation BERT_HELLO_EXPECTATION = new EncodingExpectation(
+      BERT_TOKEN_IDS, true, "Hello",
+      new String[] {"[CLS]", "hello", "[SEP]"},
+      new long[] {101, 42, 2_147_483_647L},
+      new long[] {1, 1, 1},
+      new long[] {0, 0, 0});
+
+  private static final EncodingExpectation BERT_MIXED_EXPECTATION = new EncodingExpectation(
+      BERT_TOKEN_IDS, true, "Hello playing rabbit",
+      new String[] {"[CLS]", "hello", "play", "##ing", "[UNK]", "[SEP]"},
+      new long[] {101, 42, 0, 2_147_483_646L, 900_001, 2_147_483_647L},
+      new long[] {1, 1, 1, 1, 1, 1},
+      new long[] {0, 0, 0, 0, 0, 0});
+
+  private static final EncodingExpectation BERT_UNKNOWN_EXPECTATION = new EncodingExpectation(
+      BERT_TOKEN_IDS, true, "rabbit",
+      new String[] {"[CLS]", "[UNK]", "[SEP]"},
+      new long[] {101, 900_001, 2_147_483_647L},
+      new long[] {1, 1, 1},
+      new long[] {0, 0, 0});
+
+  private static final EncodingExpectation BERT_WORDPIECES_EXPECTATION =
+      new EncodingExpectation(
+          BERT_TOKEN_IDS, true, "Playing",
+          new String[] {"[CLS]", "play", "##ing", "[SEP]"},
+          new long[] {101, 0, 2_147_483_646L, 2_147_483_647L},
+          new long[] {1, 1, 1, 1},
+          new long[] {0, 0, 0, 0});
+
+  private static final List<EncodingExpectation> CONCURRENT_EXPECTATIONS = List.of(
+      BERT_HELLO_EXPECTATION,
+      BERT_UNKNOWN_EXPECTATION,
+      BERT_WORDPIECES_EXPECTATION);
+
+  /** Expected output for one model-free encoding case. */
+  private record EncodingExpectation(
+      Map<String, Integer> tokenIds,
+      boolean lowerCase,
+      CharSequence input,
+      String[] tokens,
+      long[] ids,
+      long[] mask,
+      long[] types) {
+  }
+
+  /**
+   * Confirms the token strings, ids, attention mask, and token types.
+   *
+   * @param name The case name.
+   * @param expected The fixture and expected arrays.
+   */
+  @ParameterizedTest(name = "{0}")
+  @MethodSource("encodingCases")
+  void testEncodingArrays(String name, EncodingExpectation expected) {
+    final Tokens actual = new ModelFreeDL(expected.tokenIds(), expected.lowerCase())
+        .encode(expected.input());
+
+    assertEncoding(expected, actual);
+  }
+
+  /** Confirms null text is rejected at the instance encoder boundary. */
+  @Test
+  void testRejectsNullText() {
+    final ModelFreeDL encoder = new ModelFreeDL(BERT_TOKEN_IDS, true);
+
+    final IllegalArgumentException exception = assertThrows(IllegalArgumentException.class,
+        () -> encoder.encode(null));
+    assertEquals("text must not be null", exception.getMessage());
+  }
+
+  /** Confirms that successive calls return independent arrays. */
+  @Test
+  void testRepeatedCallsReturnIndependentArrays() {
+    final ModelFreeDL encoder = new ModelFreeDL(BERT_TOKEN_IDS, true);
+    final Tokens first = encoder.encode("Hello");
+    final Tokens second = encoder.encode("Hello");
+
+    assertNotSame(first.tokens(), second.tokens());
+    assertNotSame(first.ids(), second.ids());
+    assertNotSame(first.mask(), second.mask());
+    assertNotSame(first.types(), second.types());
+
+    first.tokens()[1] = "changed";
+    first.ids()[1] = -1;
+    first.mask()[1] = 0;
+    first.types()[1] = 1;
+
+    assertEncoding(BERT_HELLO_EXPECTATION, second);
+    assertEncoding(BERT_HELLO_EXPECTATION, encoder.encode("Hello"));
+  }
+
+  /**
+   * Confirms bounded concurrent calls produce complete independent results.
+   *
+   * @throws InterruptedException Thrown if the test thread is interrupted.
+   * @throws ExecutionException Thrown if an encoding task fails.
+   * @throws TimeoutException Thrown if an encoding task exceeds the wait limit.
+   */
+  @Test
+  void testConcurrentCallsReturnIndependentArrays()
+      throws InterruptedException, ExecutionException, TimeoutException {
+    final ModelFreeDL encoder = new ModelFreeDL(BERT_TOKEN_IDS, true);
+    final ExecutorService executor = Executors.newFixedThreadPool(CONCURRENT_WORKERS);
+    final CountDownLatch start = new CountDownLatch(1);
+    final List<Future<Tokens>> futures = new ArrayList<>();
+    try {
+      for (int i = 0; i < CONCURRENT_CALLS; i++) {
+        final EncodingExpectation expected =
+            CONCURRENT_EXPECTATIONS.get(i % CONCURRENT_EXPECTATIONS.size());
+        futures.add(executor.submit(() -> {
+          start.await();
+          return encoder.encode(expected.input());
+        }));
+      }
+      start.countDown();
+
+      final List<Tokens> completed = new ArrayList<>();
+      for (int i = 0; i < futures.size(); i++) {
+        final Future<Tokens> future = futures.get(i);
+        final Tokens actual = future.get(WAIT_SECONDS, TimeUnit.SECONDS);
+        assertEncoding(CONCURRENT_EXPECTATIONS.get(i % CONCURRENT_EXPECTATIONS.size()), actual);
+        for (final Tokens prior : completed) {
+          assertNotSame(prior.tokens(), actual.tokens());
+          assertNotSame(prior.ids(), actual.ids());
+          assertNotSame(prior.mask(), actual.mask());
+          assertNotSame(prior.types(), actual.types());
+        }
+        completed.add(actual);
+      }
+    } finally {
+      executor.shutdownNow();
+      assertTrue(executor.awaitTermination(WAIT_SECONDS, TimeUnit.SECONDS));
+    }
+  }
+
+  /**
+   * Supplies model-free encoding fixtures.
+   *
+   * @return The named fixtures.
+   */
+  private static Stream<Arguments> encodingCases() {
+    return Stream.of(
+        Arguments.of("bert-empty", new EncodingExpectation(BERT_TOKEN_IDS, true, "",
+            new String[] {"[CLS]", "[SEP]"},
+            new long[] {101, 2_147_483_647L},
+            new long[] {1, 1},
+            new long[] {0, 0})),
+        Arguments.of("bert-ascii-whitespace", new EncodingExpectation(
+            BERT_TOKEN_IDS, true, " \t\n",
+            new String[] {"[CLS]", "[SEP]"},
+            new long[] {101, 2_147_483_647L},
+            new long[] {1, 1},
+            new long[] {0, 0})),
+        Arguments.of("bert-unicode-whitespace", new EncodingExpectation(
+            BERT_TOKEN_IDS, true, "\u00A0\u2028\u3000",
+            new String[] {"[CLS]", "[SEP]"},
+            new long[] {101, 2_147_483_647L},
+            new long[] {1, 1},
+            new long[] {0, 0})),
+        Arguments.of("bert-known", BERT_HELLO_EXPECTATION),
+        Arguments.of("bert-known-sequence", new EncodingExpectation(
+            BERT_TOKEN_IDS, true, "Hello WORLD",
+            new String[] {"[CLS]", "hello", "world", "[SEP]"},
+            new long[] {101, 42, 1_500_000_000L, 2_147_483_647L},
+            new long[] {1, 1, 1, 1},
+            new long[] {0, 0, 0, 0})),
+        Arguments.of("bert-unicode-separator", new EncodingExpectation(
+            BERT_TOKEN_IDS, true, "Hello\u00A0WORLD",
+            new String[] {"[CLS]", "hello", "world", "[SEP]"},
+            new long[] {101, 42, 1_500_000_000L, 2_147_483_647L},
+            new long[] {1, 1, 1, 1},
+            new long[] {0, 0, 0, 0})),
+        Arguments.of("bert-unknown", BERT_UNKNOWN_EXPECTATION),
+        Arguments.of("bert-known-unknown", new EncodingExpectation(
+            BERT_TOKEN_IDS, true, "Hello rabbit",
+            new String[] {"[CLS]", "hello", "[UNK]", "[SEP]"},
+            new long[] {101, 42, 900_001, 2_147_483_647L},
+            new long[] {1, 1, 1, 1},
+            new long[] {0, 0, 0, 0})),
+        Arguments.of("bert-wordpieces", BERT_WORDPIECES_EXPECTATION),
+        Arguments.of("bert-mixed", BERT_MIXED_EXPECTATION),
+        Arguments.of("bert-string-builder", new EncodingExpectation(
+            BERT_TOKEN_IDS, true, new StringBuilder("Hello"),
+            new String[] {"[CLS]", "hello", "[SEP]"},
+            new long[] {101, 42, 2_147_483_647L},
+            new long[] {1, 1, 1},
+            new long[] {0, 0, 0})),
+        Arguments.of("bert-cased-accent", new EncodingExpectation(
+            BERT_TOKEN_IDS, false, "Caf\u00E9",
+            new String[] {"[CLS]", "Caf\u00E9", "[SEP]"},
+            new long[] {101, 89, 2_147_483_647L},
+            new long[] {1, 1, 1},
+            new long[] {0, 0, 0})),
+        Arguments.of("bert-cased-miss", new EncodingExpectation(
+            BERT_TOKEN_IDS, false, "Hello",
+            new String[] {"[CLS]", "[UNK]", "[SEP]"},
+            new long[] {101, 900_001, 2_147_483_647L},
+            new long[] {1, 1, 1},
+            new long[] {0, 0, 0})),
+        Arguments.of("bert-precomposed-accent", new EncodingExpectation(
+            BERT_TOKEN_IDS, true, "CAF\u00C9",
+            new String[] {"[CLS]", "cafe", "[SEP]"},
+            new long[] {101, 88, 2_147_483_647L},
+            new long[] {1, 1, 1},
+            new long[] {0, 0, 0})),
+        Arguments.of("bert-decomposed-accent", new EncodingExpectation(
+            BERT_TOKEN_IDS, true, "Cafe\u0301",
+            new String[] {"[CLS]", "cafe", "[SEP]"},
+            new long[] {101, 88, 2_147_483_647L},
+            new long[] {1, 1, 1},
+            new long[] {0, 0, 0})),
+        Arguments.of("bert-final-sigma", new EncodingExpectation(
+            BERT_TOKEN_IDS, true, "\u03A3\u039F\u03A6\u039F\u03A3",
+            new String[] {"[CLS]", "\u03C3\u03BF\u03C6\u03BF\u03C2", "[SEP]"},
+            new long[] {101, 90, 2_147_483_647L},
+            new long[] {1, 1, 1},
+            new long[] {0, 0, 0})),
+        Arguments.of("bert-cased-sigma", new EncodingExpectation(
+            BERT_TOKEN_IDS, false, "\u03A3\u039F\u03A6\u039F\u03A3",
+            new String[] {"[CLS]", "\u03A3\u039F\u03A6\u039F\u03A3", "[SEP]"},
+            new long[] {101, 91, 2_147_483_647L},
+            new long[] {1, 1, 1},
+            new long[] {0, 0, 0})),
+        Arguments.of("roberta-empty", new EncodingExpectation(
+            ROBERTA_TOKEN_IDS, true, "",
+            new String[] {"<s>", "</s>"},
+            new long[] {2_147_483_647L, 2},
+            new long[] {1, 1},
+            new long[] {0, 0})),
+        Arguments.of("roberta-known", new EncodingExpectation(
+            ROBERTA_TOKEN_IDS, true, "Hello",
+            new String[] {"<s>", "hello", "</s>"},
+            new long[] {2_147_483_647L, 0, 2},
+            new long[] {1, 1, 1},
+            new long[] {0, 0, 0})),
+        Arguments.of("roberta-unknown", new EncodingExpectation(
+            ROBERTA_TOKEN_IDS, true, "rabbit",
+            new String[] {"<s>", "<unk>", "</s>"},
+            new long[] {2_147_483_647L, 800_000_000, 2},
+            new long[] {1, 1, 1},
+            new long[] {0, 0, 0})),
+        Arguments.of("roberta-mixed", new EncodingExpectation(
+            ROBERTA_TOKEN_IDS, true, "Hello rabbit WORLD",
+            new String[] {"<s>", "hello", "<unk>", "world", "</s>"},
+            new long[] {2_147_483_647L, 0, 800_000_000, 500, 2},
+            new long[] {1, 1, 1, 1, 1},
+            new long[] {0, 0, 0, 0, 0})),
+        Arguments.of("roberta-wordpieces", new EncodingExpectation(
+            ROBERTA_TOKEN_IDS, true, "Playing",
+            new String[] {"<s>", "play", "##ing", "</s>"},
+            new long[] {2_147_483_647L, 10, 1_500_000_000L, 2},
+            new long[] {1, 1, 1, 1},
+            new long[] {0, 0, 0, 0})),
+        Arguments.of("roberta-bert-unknown", new EncodingExpectation(
+            ROBERTA_BERT_UNKNOWN_TOKEN_IDS, true, "Hello rabbit",
+            new String[] {"<s>", "hello", "[UNK]", "</s>"},
+            new long[] {71, 73, 2_147_483_647L, 72},
+            new long[] {1, 1, 1, 1},
+            new long[] {0, 0, 0, 0})));
+  }
+
+  /**
+   * Compares all model input arrays.
+   *
+   * @param expected The expected arrays.
+   * @param actual The encoded arrays.
+   */
+  private void assertEncoding(EncodingExpectation expected, Tokens actual) {

Review Comment:
   Done. The record has only tokens and ids; assertEncoding builds the mask and segment arrays from the token count.



##########
opennlp-core/opennlp-ml/opennlp-dl/src/test/java/opennlp/dl/SharedDlEncodingTest.java:
##########
@@ -0,0 +1,353 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package opennlp.dl;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.stream.Stream;
+
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
+
+import opennlp.tools.tokenize.WordpieceTokenizer;
+
+import static org.junit.jupiter.api.Assertions.assertArrayEquals;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertNotSame;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+/** Checks the complete token arrays created by the shared deep-learning encoder. */
+public class SharedDlEncodingTest {
+
+  private static final int CONCURRENT_CALLS = 16;
+  private static final int CONCURRENT_WORKERS = 4;
+  private static final long WAIT_SECONDS = 10;
+
+  private static final Map<String, Integer> BERT_TOKEN_IDS = Map.ofEntries(
+      Map.entry(WordpieceTokenizer.BERT_CLS_TOKEN, 101),
+      Map.entry(WordpieceTokenizer.BERT_SEP_TOKEN, Integer.MAX_VALUE),
+      Map.entry(WordpieceTokenizer.BERT_UNK_TOKEN, 900_001),
+      Map.entry("hello", 42),
+      Map.entry("world", 1_500_000_000),
+      Map.entry("play", 0),
+      Map.entry("##ing", 2_147_483_646),
+      Map.entry("cafe", 88),
+      Map.entry("Caf\u00E9", 89),
+      Map.entry("\u03C3\u03BF\u03C6\u03BF\u03C2", 90),
+      Map.entry("\u03A3\u039F\u03A6\u039F\u03A3", 91));
+
+  private static final Map<String, Integer> ROBERTA_TOKEN_IDS = Map.of(
+      WordpieceTokenizer.ROBERTA_CLS_TOKEN, Integer.MAX_VALUE,
+      WordpieceTokenizer.ROBERTA_SEP_TOKEN, 2,
+      WordpieceTokenizer.ROBERTA_UNK_TOKEN, 800_000_000,
+      "hello", 0,
+      "world", 500,
+      "play", 10,
+      "##ing", 1_500_000_000);
+
+  private static final Map<String, Integer> ROBERTA_BERT_UNKNOWN_TOKEN_IDS = Map.of(
+      WordpieceTokenizer.ROBERTA_CLS_TOKEN, 71,
+      WordpieceTokenizer.ROBERTA_SEP_TOKEN, 72,
+      WordpieceTokenizer.BERT_UNK_TOKEN, Integer.MAX_VALUE,
+      "hello", 73);
+
+  private static final EncodingExpectation BERT_HELLO_EXPECTATION = new EncodingExpectation(
+      BERT_TOKEN_IDS, true, "Hello",
+      new String[] {"[CLS]", "hello", "[SEP]"},
+      new long[] {101, 42, 2_147_483_647L},
+      new long[] {1, 1, 1},
+      new long[] {0, 0, 0});
+
+  private static final EncodingExpectation BERT_MIXED_EXPECTATION = new EncodingExpectation(
+      BERT_TOKEN_IDS, true, "Hello playing rabbit",
+      new String[] {"[CLS]", "hello", "play", "##ing", "[UNK]", "[SEP]"},
+      new long[] {101, 42, 0, 2_147_483_646L, 900_001, 2_147_483_647L},
+      new long[] {1, 1, 1, 1, 1, 1},
+      new long[] {0, 0, 0, 0, 0, 0});
+
+  private static final EncodingExpectation BERT_UNKNOWN_EXPECTATION = new EncodingExpectation(
+      BERT_TOKEN_IDS, true, "rabbit",
+      new String[] {"[CLS]", "[UNK]", "[SEP]"},
+      new long[] {101, 900_001, 2_147_483_647L},
+      new long[] {1, 1, 1},
+      new long[] {0, 0, 0});
+
+  private static final EncodingExpectation BERT_WORDPIECES_EXPECTATION =
+      new EncodingExpectation(
+          BERT_TOKEN_IDS, true, "Playing",
+          new String[] {"[CLS]", "play", "##ing", "[SEP]"},
+          new long[] {101, 0, 2_147_483_646L, 2_147_483_647L},
+          new long[] {1, 1, 1, 1},
+          new long[] {0, 0, 0, 0});
+
+  private static final List<EncodingExpectation> CONCURRENT_EXPECTATIONS = List.of(
+      BERT_HELLO_EXPECTATION,
+      BERT_UNKNOWN_EXPECTATION,
+      BERT_WORDPIECES_EXPECTATION);
+
+  /** Expected output for one model-free encoding case. */
+  private record EncodingExpectation(
+      Map<String, Integer> tokenIds,
+      boolean lowerCase,
+      CharSequence input,
+      String[] tokens,
+      long[] ids,
+      long[] mask,
+      long[] types) {
+  }
+
+  /**
+   * Confirms the token strings, ids, attention mask, and token types.
+   *
+   * @param name The case name.
+   * @param expected The fixture and expected arrays.
+   */
+  @ParameterizedTest(name = "{0}")
+  @MethodSource("encodingCases")
+  void testEncodingArrays(String name, EncodingExpectation expected) {
+    final Tokens actual = new ModelFreeDL(expected.tokenIds(), expected.lowerCase())
+        .encode(expected.input());
+
+    assertEncoding(expected, actual);
+  }
+
+  /** Confirms null text is rejected at the instance encoder boundary. */
+  @Test
+  void testRejectsNullText() {
+    final ModelFreeDL encoder = new ModelFreeDL(BERT_TOKEN_IDS, true);
+
+    final IllegalArgumentException exception = assertThrows(IllegalArgumentException.class,
+        () -> encoder.encode(null));
+    assertEquals("text must not be null", exception.getMessage());
+  }
+
+  /** Confirms that successive calls return independent arrays. */
+  @Test
+  void testRepeatedCallsReturnIndependentArrays() {
+    final ModelFreeDL encoder = new ModelFreeDL(BERT_TOKEN_IDS, true);
+    final Tokens first = encoder.encode("Hello");
+    final Tokens second = encoder.encode("Hello");
+
+    assertNotSame(first.tokens(), second.tokens());
+    assertNotSame(first.ids(), second.ids());
+    assertNotSame(first.mask(), second.mask());
+    assertNotSame(first.types(), second.types());
+
+    first.tokens()[1] = "changed";
+    first.ids()[1] = -1;
+    first.mask()[1] = 0;
+    first.types()[1] = 1;
+
+    assertEncoding(BERT_HELLO_EXPECTATION, second);
+    assertEncoding(BERT_HELLO_EXPECTATION, encoder.encode("Hello"));
+  }
+
+  /**
+   * Confirms bounded concurrent calls produce complete independent results.
+   *
+   * @throws InterruptedException Thrown if the test thread is interrupted.
+   * @throws ExecutionException Thrown if an encoding task fails.
+   * @throws TimeoutException Thrown if an encoding task exceeds the wait limit.
+   */
+  @Test
+  void testConcurrentCallsReturnIndependentArrays()

Review Comment:
   Removed. The repeated-call test covers the independent arrays.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@opennlp.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org