You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@opennlp.apache.org by kr...@apache.org on 2026/07/21 11:41:23 UTC

(opennlp) branch sentencepiece updated (6d40bc071 -> e82aa10fb)

This is an automated email from the ASF dual-hosted git repository.

krickert pushed a change to branch sentencepiece
in repository https://gitbox.apache.org/repos/asf/opennlp.git


    from 6d40bc071 OPENNLP-1885: Guard tokenizer deserialization with an allow-listing ObjectInputFilter
     add 3405694f5 OPENNLP-1885: Cite the SentencePiece usage example test in the manual
     add e82aa10fb OPENNLP-1885: Align null contracts and annotations with the review conventions

No new revisions were added by this update.

Summary of changes:
 .../opennlp/tools/tokenize/WordpieceEncoder.java   | 21 ++++++-
 opennlp-docs/src/docbkx/tokenizer.xml              |  9 +--
 .../sentencepiece/SentencePieceTokenizer.java      |  2 +
 .../subword/sentencepiece/PieceTrieTest.java       |  3 +-
 .../SentencePieceUsageExampleTest.java             | 69 ++++++++++++++++++++++
 5 files changed, 96 insertions(+), 8 deletions(-)
 create mode 100644 opennlp-extensions/opennlp-subword/src/test/java/opennlp/subword/sentencepiece/SentencePieceUsageExampleTest.java