[jira] [Closed] (OPENNLP-1885) Pure-Java SentencePiece inference with exact original-text spans

["Richard Zowalla (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/OPENNLP-1885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Richard Zowalla closed OPENNLP-1885.
------------------------------------
    Resolution: Fixed

> Pure-Java SentencePiece inference with exact original-text spans 
> -----------------------------------------------------------------
>
>                 Key: OPENNLP-1885
>                 URL: https://issues.apache.org/jira/browse/OPENNLP-1885
>             Project: OpenNLP
>          Issue Type: New Feature
>          Components: Sentence Detector
>            Reporter: Kristian Rickert
>            Assignee: Kristian Rickert
>            Priority: Major
>             Fix For: 3.0.0-M6, 3.0.0
>
>          Time Spent: 6h
>  Remaining Estimate: 0h
>
> Adds a new opennlp-extensions/opennlp-subword module implementing SentencePiece model inference purely in Java: the model file reader (hand-written protobuf wire parsing, no new dependency), the model-embedded normalizer (precompiled character map over the double-array trie format, whitespace collapsing and escaping, word-boundary marker), unigram best-path segmentation, BPE agenda merging, byte fallback, and user-defined symbol handling. The public contract is SubwordTokenizer/SubwordPiece; every piece reports the exact UTF-16 span of the caller's original text it came from, and the model normalizer is also exposed as an OffsetAwareNormalizer producing AlignedText.
> Parity with the reference implementation is asserted, not assumed. Five tiny trained models are bundled with fixtures generated by the sentencepiece Python package (unigram, unigram with byte fallback, BPE, identity normalization, whitespace-as-suffix), checked piece for piece, id for id, span for span, plus each model's embedded self-test samples. An opt-in test (-Dopennlp.subword.eval.dir) runs the same assertions against real downloaded models; t5-small (32k vocabulary) and albert-base-v2 (30k) pass exactly, including mixed scripts, emoji ZWJ sequences, BOM, and CRLF inputs. Fixture regeneration scripts live in the test resources.
> Measured single-thread throughput on the t5-small vocabulary is about 2.8M pieces/s; the native reference is about 1.6x faster. The value here is zero native dependencies, one shareable thread-safe instance, and exact original-text offsets. A non-breaking performance follow-up with identified wins is planned separately.
> JIRA to follow; title gets the key once filed.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)