You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@opennlp.apache.org by rz...@apache.org on 2026/09/08 08:16:31 UTC
(opennlp) 01/01: OPENNLP-1885: Cite the subword tokenization literature in the API javadoc
This is an automated email from the ASF dual-hosted git repository.
rzo1 pushed a commit to branch OPENNLP-1885-subword-literature-refs
in repository https://gitbox.apache.org/repos/asf/opennlp.git
commit d2c665f6467b5ad011221c2318169b3c69844fe3
Author: Richard Zowalla <rz...@apache.org>
AuthorDate: Tue Sep 8 10:11:50 2026 +0200
OPENNLP-1885: Cite the subword tokenization literature in the API javadoc
Adds the primary references for the subword API: SentencePiece for the
piece/id/span encoding contract, BPE and the unigram LM for the subword
interface, and Schuster/Nakajima, Wu et al. and BERT for the wordpiece
encoder. Also corrects the attribution of the 100 code point word limit,
which is the Hugging Face default; google-research/bert uses 200.
---
.../java/opennlp/tools/tokenize/SubwordPiece.java | 12 ++++++++++++
.../opennlp/tools/tokenize/SubwordTokenizer.java | 18 ++++++++++++++++++
.../opennlp/tools/tokenize/WordpieceEncoder.java | 21 ++++++++++++++++++++-
3 files changed, 50 insertions(+), 1 deletion(-)
diff --git a/opennlp-api/src/main/java/opennlp/tools/tokenize/SubwordPiece.java b/opennlp-api/src/main/java/opennlp/tools/tokenize/SubwordPiece.java
index 09e3f98ea..dca3d56ae 100644
--- a/opennlp-api/src/main/java/opennlp/tools/tokenize/SubwordPiece.java
+++ b/opennlp-api/src/main/java/opennlp/tools/tokenize/SubwordPiece.java
@@ -26,10 +26,22 @@ package opennlp.tools.tokenize;
* adjacent source characters when normalization reorders characters. Pieces without source text,
* such as control symbols, report an empty span with {@code start == end}.</p>
*
+ * <p>The piece, id and span triple follows the representation established by SentencePiece
+ * (Kudo and Richardson, 2018), whose {@code SentencePieceText.SentencePiece} message carries
+ * {@code piece}, {@code id} and a {@code [begin, end)} range into the input, and gives control
+ * symbols an empty surface with {@code begin == end}. The {@code surface} field of that message
+ * has no counterpart here because it is {@code text.subSequence(start, end)}. The offsets are
+ * UTF-16 indices rather than the UTF-8 byte offsets used there.</p>
+ *
* @param piece The piece in the vocabulary's normalized form; must not be {@code null} or empty.
* @param id The non-negative vocabulary id of the piece.
* @param start The inclusive start offset in the original text.
* @param end The exclusive end offset in the original text; at least {@code start}.
+ *
+ * @see SubwordTokenizer
+ * @see <a href="https://aclanthology.org/D18-2012/">Taku Kudo, John Richardson (2018):
+ * SentencePiece: A simple and language independent subword tokenizer and detokenizer for
+ * Neural Text Processing. EMNLP 2018 (system demonstrations), pages 66-71</a>
* @since 3.0.0
*/
public record SubwordPiece(String piece, int id, int start, int end) {
diff --git a/opennlp-api/src/main/java/opennlp/tools/tokenize/SubwordTokenizer.java b/opennlp-api/src/main/java/opennlp/tools/tokenize/SubwordTokenizer.java
index e80f1bd81..056bb5942 100644
--- a/opennlp-api/src/main/java/opennlp/tools/tokenize/SubwordTokenizer.java
+++ b/opennlp-api/src/main/java/opennlp/tools/tokenize/SubwordTokenizer.java
@@ -26,11 +26,29 @@ import java.util.List;
* model's normalized form and need not equal the input. Offsets in each {@link SubwordPiece}
* refer to the original input text.</p>
*
+ * <p>Subword units were introduced to neural models to keep the vocabulary closed while still
+ * covering rare and unseen words: byte pair encoding (Sennrich et al., 2016), the WordPiece
+ * inventory (Schuster and Nakajima, 2012; Wu et al., 2016) and the unigram language model
+ * (Kudo, 2018). This interface abstracts over the segmentation method and describes only what
+ * the trained inventories have in common, following the tokenizer-independent encoding contract
+ * of SentencePiece (Kudo and Richardson, 2018): every unit is a vocabulary entry with an id and
+ * a range over the input.</p>
+ *
* <p>An implementation may include model control pieces with empty source spans. Their presence
* and placement are part of that tokenizer's contract, not this interface.</p>
*
* <p>Thread safety is implementation specific.</p>
*
+ * @see SubwordPiece
+ * @see <a href="https://aclanthology.org/P16-1162/">Rico Sennrich, Barry Haddow, Alexandra Birch
+ * (2016): Neural Machine Translation of Rare Words with Subword Units. ACL 2016,
+ * pages 1715-1725</a>
+ * @see <a href="https://aclanthology.org/P18-1007/">Taku Kudo (2018): Subword Regularization:
+ * Improving Neural Network Translation Models with Multiple Subword Candidates. ACL 2018,
+ * pages 66-75</a>
+ * @see <a href="https://aclanthology.org/D18-2012/">Taku Kudo, John Richardson (2018):
+ * SentencePiece: A simple and language independent subword tokenizer and detokenizer for
+ * Neural Text Processing. EMNLP 2018 (system demonstrations), pages 66-71</a>
* @since 3.0.0
*/
public interface SubwordTokenizer {
diff --git a/opennlp-api/src/main/java/opennlp/tools/tokenize/WordpieceEncoder.java b/opennlp-api/src/main/java/opennlp/tools/tokenize/WordpieceEncoder.java
index 0dcd50f80..dedfb236c 100644
--- a/opennlp-api/src/main/java/opennlp/tools/tokenize/WordpieceEncoder.java
+++ b/opennlp-api/src/main/java/opennlp/tools/tokenize/WordpieceEncoder.java
@@ -36,6 +36,15 @@ import opennlp.tools.util.StringUtil;
* ranges at the text boundaries, so {@link #encode(CharSequence)} includes both control
* entries.</p>
*
+ * <p>The wordpiece inventory was introduced by Schuster and Nakajima (2012) as the
+ * {@code WordPieceModel}: word units learned greedily from unsegmented text to maximize the
+ * language-model likelihood, so that no input is out of vocabulary. Wu et al. (2016), section
+ * 4.1, adopt it for neural machine translation, and Devlin et al. (2019), section 3, build BERT
+ * on a 30,000 entry WordPiece vocabulary with a leading classification token. Those papers
+ * describe how an inventory is <i>trained</i>; the greedy longest-match-first segmentation and
+ * the {@code ##} continuation marker applied here are the <i>inference</i> conventions of the
+ * BERT reference implementation. Wu et al. instead mark word starts with {@code _}.</p>
+ *
* <p>Ids follow the line-number convention of BERT {@code vocab.txt} files. List constructors use
* the list index, while the map constructor uses the supplied ids. The classification, separator,
* and unknown tokens must all be present in the
@@ -47,11 +56,21 @@ import opennlp.tools.util.StringUtil;
* implementation.</p>
*
* <p>A word exceeding the configured maximum number of normalized Unicode code points becomes
- * the unknown piece. The default is 100, matching the BERT reference implementation.</p>
+ * the unknown piece. The default is 100, the value used by the Hugging Face {@code transformers}
+ * BERT tokenizer; the original {@code google-research/bert} code uses 200. Both count code
+ * points, and a constructor parameter selects another limit.</p>
*
* @see WordpieceTokenizer
* @see <a href="https://github.com/google-research/bert/blob/master/tokenization.py">
* BERT tokenization reference</a>
+ * @see <a href="https://research.google/pubs/pub37842/">Mike Schuster, Kaisuke Nakajima (2012):
+ * Japanese and Korean Voice Search. ICASSP 2012, pages 5149-5152</a>
+ * @see <a href="https://arxiv.org/abs/1609.08144">Yonghui Wu et al. (2016): Google's Neural
+ * Machine Translation System: Bridging the Gap between Human and Machine Translation.
+ * arXiv:1609.08144</a>
+ * @see <a href="https://aclanthology.org/N19-1423/">Jacob Devlin, Ming-Wei Chang, Kenton Lee,
+ * Kristina Toutanova (2019): BERT: Pre-training of Deep Bidirectional Transformers for
+ * Language Understanding. NAACL-HLT 2019, pages 4171-4186</a>
* @since 3.0.0
*/
@ThreadSafe