You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@opennlp.apache.org by kr...@apache.org on 2026/07/28 11:31:50 UTC

(opennlp) branch sentencepiece updated (402bfc22a -> 39ba8ef8f)

This is an automated email from the ASF dual-hosted git repository.

krickert pushed a change to branch sentencepiece
in repository https://gitbox.apache.org/repos/asf/opennlp.git


    from 402bfc22a OPENNLP-1885: Address review: checked InvalidFormatException for malformed models, tag helpers, javadoc throws
     add 39ba8ef8f OPENNLP-1885: Address review: validation message style, shared test fixtures, thread safety wording

No new revisions were added by this update.

Summary of changes:
 .../java/opennlp/tools/tokenize/SubwordPiece.java  |  2 +-
 .../opennlp/tools/tokenize/SubwordTokenizer.java   |  3 +-
 .../opennlp/tools/tokenize/WordpieceEncoder.java   | 12 ++--
 .../opennlp/tools/tokenize/WordpieceTokenizer.java |  3 +-
 .../tools/tokenize/ReferenceBertPipeline.java      |  6 +-
 .../tools/tokenize/WordpieceEncoderTest.java       |  3 +
 opennlp-docs/src/docbkx/tokenizer.xml              |  5 +-
 .../opennlp/subword/sentencepiece/BpeEncoder.java  |  2 +-
 .../subword/sentencepiece/ModelProtoReader.java    |  2 +-
 .../sentencepiece/SentencePieceTokenizer.java      | 28 ++++-----
 .../subword/sentencepiece/UnigramEncoder.java      |  2 +-
 .../sentencepiece/SentencePieceAlignmentTest.java  |  2 +-
 .../sentencepiece/SentencePieceFixtures.java       | 67 ++++++++++++++++++++--
 .../SentencePieceModelValidationTest.java          | 37 +++++++++++-
 .../sentencepiece/SentencePieceParityTest.java     | 47 +++------------
 .../SentencePieceRealModelEvalTest.java            | 11 ++--
 .../SentencePieceTokenizerSerializationTest.java   | 19 +++---
 17 files changed, 154 insertions(+), 97 deletions(-)