You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@opennlp.apache.org by "Martin Wiesner (Jira)" <ji...@apache.org> on 2026/06/18 10:38:00 UTC
[jira] [Commented] (OPENNLP-1850) Unicode aware tokenization for whitespace and dashes with utility classes
[ https://issues.apache.org/jira/browse/OPENNLP-1850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18089896#comment-18089896 ]
Martin Wiesner commented on OPENNLP-1850:
-----------------------------------------
The enhanced impl should be (natural) language agnostic - so it does not just cover “en” cases.
> Unicode aware tokenization for whitespace and dashes with utility classes
> -------------------------------------------------------------------------
>
> Key: OPENNLP-1850
> URL: https://issues.apache.org/jira/browse/OPENNLP-1850
> Project: OpenNLP
> Issue Type: New Feature
> Components: dl, Tokenizer
> Reporter: Kristian Rickert
> Assignee: Kristian Rickert
> Priority: Major
> Fix For: 3.0.0
>
>
> h2. Summary
> Make OpenNLP DL text preprocessing Unicode-aware for two character classes: whitespace and dashes. Drive both off explicit, standards-sourced code point sets that the user can extend from an input file, do all the work with cursor based scanning instead of regex, and add a real set of helpers (split, normalize, squish, trim, normalize-dashes) instead of scattering {{s+}} regexes around the code.
> I've run into this issue a lot with large sets of documents. Originating data files and PDFs slip a lot of these in their raw data and it trips up the software, often resulting in me rolling my own.
> Cursors should prove to vastly improve performance as well and greatly reduce the memory footprint of a lot of these operations.
> h2. Why
> Java's default{{{}s{}}} only matches 6 ASCII characters: space, tab, LF, VT, FF, CR. Real input has a lot more, especially anything pasted from the web, PDFs, Office docs, or non-English sources. NBSP and friends look like spaces to a human but are invisible to {{{}s+{}}}, so we get wrong token boundaries, failed span lookups, and inconsistent model inputs depending on where the text came from.
> Same story with dashes. There are many dash code points (en dash, em dash, figure dash, non-breaking hyphen, fullwidth hyphen, minus sign, and more) and we only ever recognize the ASCII hyphen. Normalizing them to a single ASCII hyphen, and optionally treating any dash as a token delimiter, would let users tokenize ideas more consistently no matter where the text came from.
> We also do this whitespace work in more than one place, each slightly differently, so a fix in one spot does not help the others. Current offenders:
> - {{NameFinderDL}} line ~619: {{{}text.split("{}}}{{{}s+"){}}} for chunking
> - {{NameFinderDL}} line ~583: {{{}Pattern.quote(span).replace(" ", "\\E\\s*{}}}{{{}Q"){}}} for matching a reconstructed span back to the source
> - {{DocumentCategorizerDL}} line ~337: {{{}text.split("{}}}{{{}s+"){}}} for chunking
> The real goal is to pin down what "whitespace" and "dash" actually mean per published standards, make those choices explicit and configurable, and implement them in a way that is cheap on memory and predictable.
> h2. Hard requirements
> These are recommended "musts" for this work:
> - *Do not use Java's built-in character assumptions.* No {{{}Character.isWhitespace{}}}, no {{{}Character.isSpaceChar{}}}, no {{{}String.trim{}}}/{{{}String.strip{}}}, no {{{}\\s{}}}/\{{p {Pd}}}/{{{}p\{IsWhite_Space}{}}} as the source of truth. They disagree with each other and with the standards (details below).
> - *No regex.* All splitting, matching, and normalization is done with cursor based scanning over code points, single pass, one output buffer. Regex is error prone here and costs us {{{}Pattern{}}}/{{{}Matcher{}}} allocation, backtracking, and intermediate arrays we do not need.
> - *Standards-sourced, explicit lists.* Default sets come from the Unicode Character Database ({{{}PropList.txt{}}}: {{White_Space}} and {{Dash}} properties), checked in as explicit code point lists, cross-checked against POSIX / IEEE Std 1003.1, W3C, XML, and JSON. Users can extend or override via an input file.
> h2. Research: whitespace across standards
> There is no single definition and they do not agree. We anchor on the Unicode {{White_Space}} property and treat the rest as references.
> ||Standard / source||What counts||Notes||
> |Unicode {{White_Space}} (UCD {{{}PropList.txt{}}}, UAX #44)|25 code points|Our anchor. Includes NBSP (U+00A0), narrow NBSP (U+202F), ideographic space (U+3000), NEL (U+0085), line/paragraph separators. Does NOT include zero-width space (U+200B) or U+180E.|
> |POSIX / IEEE Std 1003.1 {{{}[:space:]{}}}|tab, LF, VT, FF, CR, space (6)|The classic ASCII set in the C locale. Subset of Unicode.|
> |W3C HTML "ASCII whitespace"|TAB, LF, FF, CR, SPACE (5)|No VT.|
> |XML 1.0 {{S}} production|space, tab, CR, LF (4)|Narrow.|
> |JSON (RFC 8259)|space, tab, LF, CR (4)|Narrow.|
> Why we do NOT use Java's APIs as the definition:
> - {{{}Character.isWhitespace(int){}}}: EXCLUDES the non-breaking spaces (U+00A0, U+2007, U+202F) and NEL (U+0085), but INCLUDES the FS/GS/RS/US controls (U+001C-U+001F). That is not the Unicode list and not any other standard.
> - {{{}Character.isSpaceChar(int){}}}: only Zs/Zl/Zp separators, so it EXCLUDES tab, newline, CR, FF, VT. Opposite trade-off.
> - {{String.trim()}} strips code points <= U+0020 only. {{String.strip()}} inherits the {{isWhitespace}} quirks.
> - Regex {{s}} default is the 6 ASCII chars; {{s}} with {{UNICODE_CHARACTER_CLASS}} maps to the Unicode property but still drags in regex machinery we are avoiding.
> Default whitespace set (the 25, all in the BMP):
> - {{U+0009}} TAB, {{U+000A}} LF, {{U+000B}} VT, {{U+000C}} FF, {{U+000D}} CR
> - {{U+0020}} SPACE
> - {{U+0085}} NEXT LINE
> - {{U+00A0}} NO-BREAK SPACE
> - {{U+1680}} OGHAM SPACE MARK
> - {{{}U+2000{}}}-{{{}U+200A{}}} typographic spaces
> - {{U+2028}} LINE SEPARATOR, {{U+2029}} PARAGRAPH SEPARATOR
> - {{U+202F}} NARROW NO-BREAK SPACE
> - {{U+205F}} MEDIUM MATHEMATICAL SPACE
> - {{U+3000}} IDEOGRAPHIC SPACE
> Not whitespace, must not be removed or treated as space by default (zero-width / invisible format chars): {{U+200B}} ZWSP, {{U+200C}} ZWNJ, {{U+200D}} ZWJ, {{U+2060}} WORD JOINER, {{U+FEFF}} BOM.
> h2. Research: dashes across standards
> Anchor on the Unicode {{Dash}} property (UCD {{{}PropList.txt{}}}). Same lesson as whitespace: do not lean on a built-in.
> - Java has NO built-in {{Dash}} predicate. There is no {{{}Character.isDash{}}}, and
> {{p\{IsDash}}} is not a supported Java regex property. The closest is the general category {{
> p\{Pd}}} (Dash_Punctuation), which MISSES {{U+2212}} MINUS SIGN (category Sm) and the super/subscript minuses. So an explicit set is required regardless.
> Default dash set (normalize to {{U+002D}} HYPHEN-MINUS), representative members:
> - {{U+002D}} HYPHEN-MINUS (the target)
> - {{U+2010}} HYPHEN, {{U+2011}} NON-BREAKING HYPHEN
> - {{U+2012}} FIGURE DASH, {{U+2013}} EN DASH, {{U+2014}} EM DASH, {{U+2015}} HORIZONTAL BAR
> - {{U+2E3A}} TWO-EM DASH, {{U+2E3B}} THREE-EM DASH
> - {{U+FE58}} SMALL EM DASH, {{U+FE63}} SMALL HYPHEN-MINUS, {{U+FF0D}} FULLWIDTH HYPHEN-MINUS
> - script hyphens: {{U+058A}} ARMENIAN HYPHEN, {{U+05BE}} HEBREW MAQAF, {{U+1400}} CANADIAN SYLLABICS HYPHEN, {{U+1806}} MONGOLIAN TODO SOFT HYPHEN, {{U+301C}} WAVE DASH, {{U+3030}} WAVY DASH, {{U+30A0}} KATAKANA-HIRAGANA DOUBLE HYPHEN
> Special cases (handle deliberately, not by default):
> - {{U+00AD}} SOFT HYPHEN is a format char (Cf), an invisible line-break hint, NOT a visible dash. Do not convert it to {{{}-{}}}. Treat it like the zero-width format chars: leave it or strip it, do not normalize it to a hyphen.
> - {{U+2212}} MINUS SIGN is mathematical (Sm). Flattening it to ASCII {{-}} can change meaning in math/code, so dash normalization for {{U+2212}} (and {{{}U+207B{}}}/{{{}U+208B{}}} super/subscript minus) is opt-in.
> h2. What I want to build
> A shared foundation, then two profiles on top of it.
> *Shared foundation*
> - {{{}CodePointSet{}}}: an explicit code point set with O(1) membership, built from a standards-derived list and optionally extended from a user file.
> - {{{}CharClass{}}}: one generic cursor based engine (split, normalize, collapse, trim, remove) configured by a member set plus a canonical ASCII replacement. Whitespace and dashes ship as built-in presets, not separate hand-written classes, so a new character class later is just a new preset with no new engine code. Class-specific quirks (whitespace line-break preservation, dash soft-hyphen/math-minus exclusions) live in the preset config, not the engine.
> *Whitespace profile*
> - Split text on the whitespace set (replaces the {{split("}} {{s+")}} sites).
> - Cursor based source matching for {{NameFinderDL}} span lookup (replaces the {{\\E\\s*}}
> {{Q}} regex).
> - Normalize: collapse any run of whitespace to a single ASCII space.
> - Squish: collapse runs but keep structure. {{\n\n\n\n\t\t\n}} becomes a single {{\n}} when the run contains a line break, otherwise a single space.
> - Trim leading/trailing whitespace, and remove-all.
> *Dash profile*
> - Normalize all dashes in the set to {{U+002D}} (1:1 replacement).
> - Optional dash delimiter: treat any dash in the set as a token boundary for tokenization.
> h2. Implementation approach: cursors, no regex
> *Set representation.* Back the set with a bitmap over the BMP (U+0000-U+FFFF) for O(1) {{{}contains{}}}, plus a tiny overflow structure (sorted {{int[]}} + binary search) for any supplementary code points. Whitespace is entirely BMP; dashes are almost entirely BMP, so the overflow path is rarely touched.
> *Scanning.* Every operation is a single forward pass with an index cursor: read {{{}cp = Character.codePointAt(text, i){}}}, test membership, act, advance {{{}i += Character.charCount(cp){}}}.
> Examples:
> - split: walk the text, mark token start, on a whitespace code point close the current token (if non-empty), skip the run, continue. Return offset spans so callers keep character offsets.
> - normalize/squish: copy non-members straight through; per run of members emit one space (normalize) or one {{{}\n{}}}/space depending on whether the run held a line break (squish).
> - trim: cursor in from each end to the first/last non-member, then substring.
> - normalize-dashes: copy non-members, emit {{-}} for each in-set dash.
> - {{NameFinderDL}} match: a two-cursor matcher walking source and reconstructed span together, treating an in-set whitespace code point as flexible (zero or more), returning {{{}[start, end){}}}. No {{{}Pattern{}}}, no {{{}quote{}}}, no {{\\E\\s*}}{{{}Q{}}}.
> *Why this is better here.* One O(n) pass, one output buffer, no Pattern/Matcher allocation, no backtracking, no intermediate arrays from {{{}String.split{}}}, and membership is a bitmap hit. It is also correct and configurable because it uses our set, not a JDK assumption.
> h2. User-defined input file
> Users extend or override either set from a simple file. Default with no file = the standards-derived sets above.
> - one code point per line, hex form, e.g. {{U+00A0}} or {{0x00A0}}
> - ranges allowed, e.g. {{U+2000-U+200A}}
> - {{#}} starts a comment
> - a section header selects which class the entries extend
> {code:java}
> [whitespace]
> U+00A0 # no-break space
> U+2028 # line separator
> [dash]
> U+2E5D # oblique hyphen (example)
> U+2212 # opt in to flattening the math minus
> {code}
> h2. API sketch
> {code:java}
> // shared foundation: the member set
> public final class CodePointSet {
> public static CodePointSet of(int... codePoints);
> public static CodePointSet fromFile(File definitions, String section) throws IOException;
> public CodePointSet union(CodePointSet other);
> public boolean contains(int codePoint); // O(1) BMP bitmap + small overflow
> }
> // shared foundation: one generic cursor engine, configured by (members, replacement)
> public final class CharClass {
> public static CharClass of(CodePointSet members, int replacement);
> // standards-sourced presets
> public static CharClass whitespace(); // Unicode White_Space -> U+0020
> public static CharClass dashes(); // Unicode Dash (curated) -> U+002D
> public boolean contains(int codePoint);
> public List<Span> splitSpans(CharSequence text); // members as delimiters, offset-preserving
> public String normalize(CharSequence text); // each member -> replacement (1:1)
> public String collapse(CharSequence text); // runs of members -> one replacement
> public String collapsePreserving(CharSequence text, CodePointSet keep, int keepReplacement); // squish
> public String trim(CharSequence text);
> public String removeAll(CharSequence text);
> }
> {code}
> So whitespace squish is {{CharClass.whitespace().collapsePreserving(text, lineBreaks, '\n')}} and dash flattening is {{{}CharClass.dashes().normalize(text){}}}. A future class is one more {{CharClass.xxx()}} preset.
> For DL components, start opt-in so default behavior does not change under anyone:
> {code:java}
> InferenceOptions#setNormalizeWhitespace(boolean)
> InferenceOptions#setNormalizeDashes(boolean)
> {code}
> h2. The offsets edge
> {{NameFinderDL}} resolves spans back to character offsets in the original text, so any transform that changes length before span lookup moves offsets and breaks {{{}Span.getCoveredText{}}}.
> - Whitespace normalize/squish collapse runs, so they DO shift offsets. Keep them on model-input prep only, or maintain an original-to-normalized offset map.
> - Dash normalize is 1:1 for BMP dashes (one code point to one {{{}-{}}}), so it is length preserving and offset safe. Supplementary dashes (e.g. none common today) would not be 1:1, so guard that path.
> - First cut: limit run-collapsing normalization to input prep, resolve span offsets against the untouched original.
> h2. Acceptance criteria
> - Shared {{CodePointSet}} with O(1) membership and unit tests, no regex, no JDK whitespace/dash APIs in the hot path.
> - Whitespace default = Unicode {{{}White_Space{}}}; dash default = Unicode {{Dash}} (excluding soft hyphen and, by default, math minus). Lists checked in explicitly and traceable to the UCD.
> - Cursor based split, normalize, squish, trim, remove-all, and a cursor based source matcher for {{{}NameFinderDL{}}}.
> - Dash normalize-to-hyphen and optional dash delimiter splitting.
> - User file extends/overrides either set, with ranges, comments, and sections.
> - Zero-width / invisible format chars are not whitespace and not removed by default. Soft hyphen is not normalized to a visible hyphen by default.
> - Normalization opt-in is documented; offsets stay correct where spans are produced.
> h2. Test plan
> - Membership: every default code point in, representative neighbors out, supplementary overflow path.
> - Split/normalize/squish/trim/remove-all over mixed ASCII and Unicode whitespace; squish structure-preservation ({{{}\n\n\t\t\n{}}} to {{{}\n{}}}).
> - NBSP and narrow NBSP inside entity/document text (most common real-world break).
> - Dash normalize across the full default set to {{{}-{}}}; verify soft hyphen and math minus are left alone unless opted in.
> - User-file parsing: single code points, ranges, comments, sections, bad lines.
> - {{NameFinderDL}} span-offset tests proving the cursor matcher and dash normalize do not move offsets.
> - A quick benchmark sanity check that the cursor path beats the regex it replaces on large inputs (no regression).
> h2. Delivery / PRs
> One JIRA as the design doc. Split delivery by risk, not by feature, since whitespace and dashes share the foundation:
> - PR 1: foundation + helpers. {{{}CodePointSet{}}}, the cursor engine, the standards-derived lists, the file loader, and all helpers (whitespace and dash), fully unit tested. Pure addition, no behavior change, nothing else imports it yet.
> - PR 2: integration. Swap the {{split("}}
> {{s+")}} sites and the {{NameFinderDL}} matcher to the cursor based helpers, add the {{InferenceOptions}} opt-ins, handle offsets. This is the only behavior-changing part, so it gets the careful review.
> - PR 3 (fallback if PR 2 is large): split the safe correctness fix (Unicode-aware split + matcher) from the new opt-in normalization (whitespace collapse, dash flatten).
> h2. Future work (separate tickets, out of scope here)
> This ticket is whitespace + dashes only. The generic {{CharClass}} engine makes these cheap follow-ups once it lands. File them as their own JIRAs after this PR merges.
> Fits the same set-to-ASCII mechanism (new {{CharClass}} preset):
> - Quotes and apostrophes to ASCII {{'}} and {{"}} (Unicode {{{}Quotation_Mark{}}}: smart quotes, guillemets, primes, fullwidth). Highest value for tokenization, e.g. {{don't}} variants.
> - Ellipsis to {{...}} ({{{}U+2026{}}} and relatives).
> - Bullets / list markers ({{{}U+2022{}}} and friends).
> - Non-ASCII decimal digits to ASCII (Unicode {{{}Nd{}}}: Arabic-Indic, Devanagari, fullwidth).
> - Strip invisible format and bidi controls (Unicode {{Default_Ignorable_Code_Point}} plus bidi controls {{{}U+200E{}}}/{{{}U+200F{}}}/{{{}U+202A{}}}{-}{{{}U+202E{}}}/{{{}U+2066{}}}{-}{{{}U+2069{}}}). Complements the "leave zero-width alone" rule by offering an explicit cleaner.
> Different mechanism, evaluate {{java.text.Normalizer}} (UAX #15) instead of a hand-rolled set:
> - Fullwidth/halfwidth forms, ligatures ({{{}fi{}}} to {{{}fi{}}}), super/subscripts: NFKC.
> - Accent / diacritic folding ({{{}café{}}} to {{{}cafe{}}}): NFD then strip combining marks.
> - Confusables / homoglyphs (Cyrillic {{а}} vs Latin {{{}a{}}}): UTS #39, larger and security-flavored.
> h2. Notes
> Keep this separate from the NameFinderDL BIO decoding work. It touches broader preprocessing semantics and can change model inputs, so it lands as its own behavior-focused change with its own review.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)