You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@opennlp.apache.org by kr...@apache.org on 2026/08/09 20:47:26 UTC
(opennlp) branch sentencepiece updated (241b70718 -> 45db4212d)
This is an automated email from the ASF dual-hosted git repository.
krickert pushed a change to branch sentencepiece
in repository https://gitbox.apache.org/repos/asf/opennlp.git
omit 241b70718 OPENNLP-1885: Deprecate BertTokenizer as a shim over WordpieceEncoder per review
omit 1c0af89a0 OPENNLP-1885: Address review: validation message style, shared test fixtures, thread safety wording
omit 2d14e49e8 OPENNLP-1885: Address review: checked InvalidFormatException for malformed models, tag helpers, javadoc throws
omit 0aad3dd6a OPENNLP-1885: Align null contracts and annotations with the review conventions
omit 6ad917e9f OPENNLP-1885: Cite the SentencePiece usage example test in the manual
omit a8597ba91 OPENNLP-1885: Guard tokenizer deserialization with an allow-listing ObjectInputFilter
omit 865f48550 OPENNLP-1885: Make the tokenizer graph serializable with computed UIDs, name the format constants, document every helper
omit 5dbb92a95 OPENNLP-1885: Document subword tokenization in the manual
omit 58bfc53b6 OPENNLP-1885: Trim residual commentary per review conventions
omit 92a3e992d OPENNLP-1885: Declare serialVersionUID on SentencePieceTokenizer
omit d86892712 OPENNLP-1885: Tighten javadoc to contracts and document helpers and overrides
omit 8d7496b0a OPENNLP-1885: Trim commentary and tighten javadoc per review conventions
omit 0a70edc48 OPENNLP-1885: Document the hand-rolled protobuf reader rationale and refactor trigger
omit b6488a814 OPENNLP-1885: Add WordpieceEncoder and fold the unreleased BertTokenizer into it
omit 6c8195b96 OPENNLP-1885: Move the subword contract into opennlp-api
omit d97054720 OPENNLP-1885: Speed up the encode path 2.3x, parity-checked at every step
omit 93fe6798a OPENNLP-1885: Add opennlp-subword: pure-Java SentencePiece inference with exact original-text spans
add 014c0182a OPENNLP-1899: Harden SymSpell model deserialization against oversized count fields (#1197)
add 757716610 OPENNLP-1907 - Bump io.github.classgraph:classgraph from 4.8.184 to 4.8.186 (#1201)
add b4226529a Bump actions/setup-java from 5.6.0 to 5.7.0 (#1199)
add ed51a6a6f OPENNLP-1908 - Bump onnxruntime.version from 1.27.0 to 1.28.0 (#1203)
add 4e77bb67b Minor: Regenerated NOTICE File for ed51a6a6f974fd0bf3be900458125fb57b011c04 (#1209)
add fc9824a97 OPENNLP-1900: Document and publish the OpenNLP security model (#1198)
add 6a71703c7 OPENNLP-1885: Add opennlp-subword: pure-Java SentencePiece inference with exact original-text spans
add fe4abddf7 OPENNLP-1885: Speed up the encode path 2.3x, parity-checked at every step
add db1c37149 OPENNLP-1885: Move the subword contract into opennlp-api
add f71db4d2e OPENNLP-1885: Add WordpieceEncoder and fold the unreleased BertTokenizer into it
add abbf3e6bc OPENNLP-1885: Document the hand-rolled protobuf reader rationale and refactor trigger
add 5c78b97f3 OPENNLP-1885: Trim commentary and tighten javadoc per review conventions
add dea3031f2 OPENNLP-1885: Tighten javadoc to contracts and document helpers and overrides
add 1f3883a67 OPENNLP-1885: Declare serialVersionUID on SentencePieceTokenizer
add 976326a3f OPENNLP-1885: Trim residual commentary per review conventions
add bf3cacff4 OPENNLP-1885: Document subword tokenization in the manual
add 81b15814b OPENNLP-1885: Make the tokenizer graph serializable with computed UIDs, name the format constants, document every helper
add e43010913 OPENNLP-1885: Guard tokenizer deserialization with an allow-listing ObjectInputFilter
add b843cf7ba OPENNLP-1885: Cite the SentencePiece usage example test in the manual
add f55fbec08 OPENNLP-1885: Align null contracts and annotations with the review conventions
add 8ebca7eaf OPENNLP-1885: Address review: checked InvalidFormatException for malformed models, tag helpers, javadoc throws
add 80029caec OPENNLP-1885: Address review: validation message style, shared test fixtures, thread safety wording
add 397c9ec87 OPENNLP-1885: Deprecate BertTokenizer as a shim over WordpieceEncoder per review
add 632d99613 OPENNLP-1885: Document fixture regeneration and Utf8Text span mapping
add 9670c34d5 OPENNLP-1885: Attribute the double-array literature in DoubleArrayTrie
add d8b04491a OPENNLP-1885: Expand fixtures README into a validation tutorial and link it from the manual
add dcc33bd68 OPENNLP-1885: Pin InvalidFormatException on a malformed precompiled character map
add f512cece4 OPENNLP-1885: Fix loader exception type and wire opennlp-subword into distr
add 45db4212d OPENNLP-1885: Reference the fixtures README by its in-tree path
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (241b70718)
\
N -- N -- N refs/heads/sentencepiece (45db4212d)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
.github/workflows/license.yml | 2 +-
.github/workflows/maven.yml | 8 +-
.github/workflows/publish-snapshots.yml | 2 +-
NOTICE | 6 +-
README.md | 9 +
SECURITY.md | 244 +++++++++++++++++++++
.../tools/ml/model/AbstractModelReader.java | 5 +-
.../WordpieceEncoderReferenceSequencesTest.java | 2 +-
opennlp-distr/pom.xml | 4 +
opennlp-distr/src/main/assembly/bin.xml | 7 +
opennlp-distr/src/main/readme/NOTICE | 6 +-
opennlp-docs/src/docbkx/tokenizer.xml | 9 +
.../dictionary/SymSpellModelSerializer.java | 11 +-
.../SymSpellModelSerializerLimitsTest.java | 109 +++++++++
.../opennlp/subword/sentencepiece/BpeEncoder.java | 11 +-
.../subword/sentencepiece/DoubleArrayTrie.java | 17 +-
.../subword/sentencepiece/ModelProtoReader.java | 3 -
.../sentencepiece/SentencePieceNormalizer.java | 15 +-
.../sentencepiece/SentencePieceTokenizer.java | 3 +-
.../opennlp/subword/sentencepiece/Utf8Text.java | 3 +-
.../SentencePieceModelValidationTest.java | 19 ++
.../SentencePieceUsageExampleTest.java | 2 +-
.../opennlp/subword/sentencepiece/README.md | 100 +++++++++
opennlp-extensions/pom.xml | 2 +-
pom.xml | 10 +-
25 files changed, 568 insertions(+), 41 deletions(-)
create mode 100644 SECURITY.md
create mode 100644 opennlp-extensions/opennlp-spellcheck/src/test/java/opennlp/spellcheck/dictionary/SymSpellModelSerializerLimitsTest.java
create mode 100644 opennlp-extensions/opennlp-subword/src/test/resources/opennlp/subword/sentencepiece/README.md