[jira] [Resolved] (OPENNLP-1852) Offset-aware normalization pipeline: handle length-changing folds (expansion and contraction) with a single offset model

["Martin Wiesner (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/OPENNLP-1852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Martin Wiesner resolved OPENNLP-1852.
-------------------------------------
    Resolution: Delivered

> Offset-aware normalization pipeline: handle length-changing folds (expansion and contraction) with a single offset model
> ------------------------------------------------------------------------------------------------------------------------
>
>                 Key: OPENNLP-1852
>                 URL: https://issues.apache.org/jira/browse/OPENNLP-1852
>             Project: OpenNLP
>          Issue Type: Epic
>          Components: Tokenizer
>            Reporter: Kristian Rickert
>            Assignee: Kristian Rickert
>            Priority: Major
>             Fix For: 3.0.0-M5, 3.0.0
>
>
> h2. Summary
> Normalization can change string length in both directions, and today only one corner of the library accounts for it. Canonical and compatibility decompositions, full case folding, ligature and fraction expansion, and run-collapsing all change the number of characters between the input and the normalized output. The {{CharClass}} engine already produces an {{Alignment}} (carried in an {{AlignedText}}) for its whitespace/dash folds, used by the DL components to map entity spans back to the original text, but the broader {{CharSequenceNormalizer}} family returns a plain {{CharSequence}} with no offset information. The goal of this issue is to make the whole normalization pipeline offset-aware so that any chain of expanding or contracting rungs can map a result position (an entity span, a search hit, a highlight) back to original-document coordinates by construction, not case by case.
> This is a follow-up to OPENNLP-1850. That work delivered the offset *model* this issue builds on (see "Already delivered" below) and made the only offset-sensitive path that applies length-changing folds today (the DL whitespace/dash chunking in {{NameFinderDL}}) offset-safe. This issue generalizes the model across the rest of the normalizer family so it can be used safely in offset-sensitive contexts.
> h2. Already delivered in OPENNLP-1850 (do not re-scope here)
> The keystone is in place; this issue should build on it rather than reinvent it.
> * {{Alignment}} -- a bidirectional edit-sequence offset model ({{equal}}/{{replace}} runs) that represents expansion, contraction, deletion (as gaps), and 1:1 changes uniformly, and maps span-to-span with {{toOriginalSpan}}/{{toNormalizedSpan}}. It replaced the earlier per-output-character {{OffsetMap}}, which assumed the normalized text contiguously covered the original and over-covered deletions. ({{OffsetMap}} and {{NormalizedText}} were never released and are removed.)
> * {{Alignment.andThen(Alignment)}} -- associative composition of two stages into one original-to-final alignment. Built and tested, including expand-then-contract and contract-then-expand chains.
> * The offset-aware *contract* on {{CharClass}}: {{normalizeAligned}}, {{collapseAligned}}, {{collapsePreservingAligned}}, {{trimAligned}}, {{removeAllAligned}}, each returning an {{AlignedText}} (normalized text plus {{Alignment}}).
> * The DL path consuming it: {{AbstractDL.normalizeInputAligned}} and {{NameFinderDL.findInOriginal}} map decoded spans back via {{Alignment.toOriginalSpan}}.
> So the remaining work is to lift the offset-aware contract from {{CharClass}} up to the whole {{CharSequenceNormalizer}} family and the {{TextNormalizer}} pipeline, add the provenance-tagged data, and add the emoji/emoticon rung and annotation layer.
> h2. Background: there is a standard, and it classifies every character for us
> We do not need to hand-enumerate "special characters." Unicode already catalogs which characters expand or contract and to what, so the work is to consume that data, not invent it.
> || Source standard || What it covers || Direction || Notes ||
> | UAX #15 Normalization Forms (NFC/NFD/NFKC/NFKD) | Canonical and compatibility (de)composition | both | Implemented by {{java.text.Normalizer}} -- but see the edit-tracking caveat under Proposed architecture: it returns a plain string with no edit information |
> | UCD {{UnicodeData.txt}} Decomposition_Mapping + Decomposition_Type (UAX #44) | Per-character expansions, tagged by type: {{<fraction>}} (1/2), {{<super>}}/{{<sub>}} (squared to 2), {{<compat>}} (ligature fi, ellipsis to three dots, Roman numerals), {{<square>}}, {{<wide>}}/{{<narrow>}} | expand | This is the authoritative expansion catalog and the basis for NFKC |
> | CaseFolding.txt (UTS #21 section 5.18) | Case folding, including the full (expanding) folds with status F: eszett to ss, ligature ff to ff | expand and 1:1 | Java's {{toLowerCase}} is locale lowercasing, not Unicode case folding, so full folds genuinely need this data |
> | Project folds (dash, quote, ellipsis, digit, confusable) | Our own target choices, some derived from Unicode properties (Dash, White_Space) and some hand-authored | both | These are the ones with no single governing standard; see provenance below |
> Contractions are the mirror image of the same data: NFC composition (combining marks fold into one precomposed character), our whitespace and dash run-collapses, and the UTF-16 case where a supplementary code point (two UTF-16 units) collapses to one unit.
> h2. The core idea: one offset model, not a per-character table
> Expansion and contraction are the same problem stated twice: the output length differs from the input length. {{Alignment}} solves it generically. It records the edits between input and output as a sequence of _equal_ runs (text copied through, length unchanged) and _replace_ runs (M original characters that produced P normalized characters), so 1-to-3 (ellipsis), 3-to-1 (run collapse), 2-to-1 (supplementary dash), and N-to-0 (deletion) are all just runs. It maps span-to-span, so a match that ends next to deleted text reports a tight original span instead of over-covering the deletion, and it composes with {{andThen}}. The character data only decides _what_ each fold produces; {{Alignment}} handles _where everything went_.
> h2. Proposed architecture
> # *Offset-aware normalizer contract.* Give the {{CharSequenceNormalizer}} family an offset-aware mode that returns an {{AlignedText}} (normalized text plus {{Alignment}}), in addition to the existing {{CharSequence}} method. A length-preserving rung returns an identity alignment; a length-changing rung builds a real one. The {{CharClass}}-backed rungs derive it cheaply during the pass they already make; the standard-engine rungs are the hard case (next point).
> # *Map composition.* {{Alignment.andThen}} -- *already delivered*. A pipeline of N rungs collapses to a single original-to-final alignment by folding {{andThen}} over the stages.
> # *Offset-aware pipeline.* Add an offset-aware build/apply mode to {{TextNormalizer}} that runs the configured rungs in order, composes their alignments, and returns one {{AlignedText}} for the whole chain.
> # *Reuse the standard engines where possible -- with a known edit-tracking gap.* NFC/NFD/NFKC/NFKD and full case folding are the value of delegating to standards, but {{java.text.Normalizer.normalize(s, form)}} returns only the result string with *no edit information*, so an {{Alignment}} cannot be read out of it directly. The realistic options, to be decided in design review: (a) depend on ICU4J's {{Normalizer2}}, which can emit an {{Edits}} object that maps directly onto our {{equal}}/{{replace}} runs; (b) compute the alignment by diffing input against output at decomposition boundaries; or (c) restrict the offset-aware path to folds we generate ourselves (the {{CharClass}} family plus a bundled-table case fold) and treat {{java.text.Normalizer}}-only NFC/NFKC as offset-opaque. This is the single biggest design decision in the issue and should be settled first.
> The {{CharClass}} {{*Aligned}} family and the DL {{normalizeInputAligned}} are the first instances of the offset-aware contract; this issue lifts the pattern up to the whole family.
> h2. Data files and provenance metadata
> We will bundle a CaseFolding lookup (sourced from the Unicode {{CaseFolding.txt}}) and aim for completeness against the upstream file. The key addition is *provenance metadata*: every mapping records which standard it follows, so we can audit coverage, distinguish standard-mandated folds from project choices, and re-derive the table when Unicode revises.
> Proposed move from the raw {{.txt}} to a small CSV (or a {{.txt}} with a documented header schema) with columns roughly:
> {noformat}
> source       ; target         ; fold_type ; standard                   ; sentiment ; unicode_version ; notes
> 00DF         ; 0073 0073      ; CASE_FOLD ; UTS21:CaseFolding:F        ;           ; 17.0.0          ; sharp s -> ss
> 2026         ; 002E 002E 002E ; NFKC      ; UAX44:Decomposition:compat ;           ; 17.0.0          ; horizontal ellipsis
> 2014 2013    ; 002D           ; DASH      ; UNSPECIFIED                ;           ; -               ; dash-to-hyphen target choice
> 1F600        ; 003A 0029      ; EMOJI     ; CLDR:annotation            ; positive  ; 17.0.0          ; grinning face -> :)  (expansion)
> 003A 0029    ; 1F642          ; EMOTICON  ; UNSPECIFIED                ; positive  ; -               ; :) -> slightly smiling face (contraction)
> {noformat}
> The {{standard}} field is the important one. Suggested values:
> * {{UAX15:NFC}} / {{UAX15:NFKC}} (and the D forms) for normalization-form folds
> * {{UAX44:Decomposition:<type>}} for a specific decomposition type
> * {{UTS21:CaseFolding:<status>}} where status is C (common), F (full), S (simple), T (Turkic)
> * {{UNSPECIFIED}} meaning "we chose this mapping; it is not mandated by a published standard." This is the explicit "we just did that" marker the team wants, so a reviewer or auditor immediately knows the entry is a project decision (for example our choice of ASCII hyphen-minus as the dash target, or expanding the ellipsis ourselves rather than relying on NFKC).
> h2. Emoji and emoticons, both directions, sentiment, and token classification
> Emoji-to-emoticon and emoticon-to-emoji are exactly the bidirectional, length-changing case this issue is built for: an emoji rendered as {{:)}} is an expansion, the reverse a contraction, and a supplementary-plane emoji is also the two-UTF-16-units-collapsing case. Both directions are in scope, riding the same offset-aware rung and lookup table. There is no governing standard for the mapping itself (emoticons are informal and the relation is many-to-many and culturally variable), so most rows are tagged {{UNSPECIFIED}}; where a direction can be anchored on published data we do (for example CLDR emoji annotations for the emoji short-name) and tag it accordingly.
> The bigger payoff is turning an opaque pictograph into *lexical signal*. Beyond the emoticon form, the same lookup can carry the emoji's actual text, its CLDR short-name (for example "grinning face"), so the rung can surface a real word sequence instead of an out-of-vocabulary symbol that most models simply drop. That text becomes the input that sentiment analysis, semantic search, embedding pipelines, and LLM-style consumers actually read and reason about, which is where this helps downstream understanding the most. The {{Term}} model is the natural carrier: one token can expose {{original}} (the emoji), a folded {{emoticon}}, a human-readable {{name}}, and a {{sentiment}} tag as parallel layers, and each consumer picks the layer it needs (a sentiment lexicon match, the name text for an embedding, or the original for display). Sentiment carries its own provenance via a {{sentiment_source}} field (an emoji sentiment ranking dataset, a project lexicon, or {{UNSPECIFIED}}), kept separate from the mapping's {{standard}} tag so each is independently auditable and replaceable.
> The same annotation mechanism generalizes well beyond sentiment. An emoji's name is frequently an entity or concept in its own right (an airplane, a tower, the Statue of Liberty, a national flag), so surfacing that name plus an optional coarse entity-type tag (for example VEHICLE, LANDMARK, ANIMAL, FLAG, FOOD) gives the Name Finder real lexical content and a gazetteer-like signal where it would otherwise see an unknown glyph. The same row can also carry a document-category hint (a pizza or an airplane is a strong cue for a food or a travel document), feeding Document Categorizer features. So the lookup is less a single sentiment column and more a small, provenance-tagged annotation layer per symbol: name, sentiment, entity type, and category, each optional and each with its own source, surfaced through the {{Term}} model and consumed by the Sentiment Detection, Name Finder, and Doccat components respectively. Each annotation dimension is opt-in, so a consumer that only wants the folded text pays nothing for the rest.
> Token classification ties in too. The UAX #29 tokenizer already gives emoji their own {{WordType}} category, so the same data lets downstream counting and feature generation treat emoji and emoticons as a consistent class rather than stray symbols, for example folding them into a punctuation-like bucket or a dedicated emoji feature for corpus statistics and the ML feature generators. That keeps a document's token counts and features stable whether sentiment is expressed as {{:)}}, as an emoji, or as the spelled-out name.
> h2. Reconciliation
> The runtime fold tables will be reconciled from several sources, so we need explicit rules:
> # *Prefer standard engines over tables.* For anything {{java.text.Normalizer}} already does (NFC/NFKC/NFD/NFKD), use it for the *text* and do not duplicate the mapping in a file (the offset-aware question for these is the edit-tracking gap above, not the mapping itself). The file is for what Java does not give us: full case folding and our project folds.
> # *Provenance on every entry.* Each bundled mapping carries its {{standard}} tag (above). A build/test step audits the table against the authoritative upstream file for that standard (for example: every CaseFolding.txt status C and F row is present and identical) and fails on drift or gaps, which is how we keep "try to make it complete" honest.
> # *Conflict precedence.* If two sources map the same code point differently for the same fold type, the more specific published standard wins over a general one, and any deviation we deliberately keep must be re-tagged {{UNSPECIFIED}} with a note, never left looking standard-backed.
> # *Do we need to roll our own?* Mostly no. The open question to settle in this issue is the exact boundary: confirm that NFC/NFKC and case folding are fully delegated to standard data, and that {{UNSPECIFIED}} is reserved for the genuinely project-specific folds (dash target, quote folding, ellipsis-if-we-want-it-independent-of-NFKC, digit folding, confusable skeletons). Minimizing the {{UNSPECIFIED}} set is a goal.
> h2. Design decisions (from OPENNLP-1850; assumed baseline on main)
> The following were settled during OPENNLP-1850 and are already shipped. This issue builds on them rather than reopening them.
> * *Offset-aware contract.* A parallel {{OffsetAwareNormalizer}} interface (extends {{CharSequenceNormalizer}}, adds {{normalizeAligned()}}). {{TextNormalizer.Builder.buildAligned()}} composes only rungs that implement it and throws {{IllegalStateException}} naming the offending rung otherwise. Callers probe with {{instanceof OffsetAwareNormalizer}} (same pattern as {{OffsetMappingNameFinder}} in the DL layer).
> * *Standard-engine folds in aligned pipelines.* NFC, NFKC, accent fold, confusable skeleton, and case fold delegate to {{java.text.Normalizer}} or JDK case mapping and *cannot* report per-character edits. They are excluded from {{buildAligned()}} in 1850; making them offset-aware is the core work of *this* issue.
> * *Default vs opt-in behavior.* {{TextNormalizer.defaultChain()}} is the conservative matching chain: strip invisible, NFC, whitespace, quotes, dashes, case fold, accent fold. DL input folding (whitespace and dashes) is opt-in via {{InferenceOptions}}. NFKC is not in {{defaultChain()}}. This issue does not change default normalization behavior; offset-aware capability and provenance-tagged lookups are additive.
> * *Emoji classification.* {{WordType.EMOJI}} is a dedicated category (not punctuation). Classification precedence is emoji > script > alphanumeric/numeric. {{Extended_Pictographic.txt}} is a filtered UCD subset bundled and cursor-parsed like the other 1850 data files.
> * *Bundled data format (1850 precedent).* Line-oriented {{.txt}} resources, cursor-parsed at load time, fail-loud on malformed input. No regex in the parse path. Provenance columns are not present in 1850 data files; they are introduced in this issue for CaseFolding and project-specific mappings.
> h2. Remaining design work
> * * *How NFC/NFKC/full case fold produce an {{Alignment}}.* {{java.text.Normalizer}} returns only a string. Choose and implement one of: ICU4J {{Normalizer2}} + {{Edits}}, input/output diff at decomposition boundaries, or a bundled CaseFolding lookup that generates the alignment during our own pass. Evaluate ICU dependency vs diff complexity in the first implementation PR.
> * *Provenance-tagged lookup format.* Extend the 1850 {{.txt}} + cursor-parse pattern with a documented header/schema for {{standard}}, {{fold_type}}, and {{UNSPECIFIED}} project choices; or introduce a small CSV if columnar provenance is clearer. One file per fold type for bundled tables (consistent with {{WordBreakProperty.txt}}, {{confusables.txt}}, etc.).
> * *{{UNSPECIFIED}} entries.* Allowed for genuine project choices (dash target, quote targets, emoji/emoticon mappings with no governing standard). Each must carry an inline note. Standard-backed rows must audit clean against the upstream file. No separate sign-off gate in the build for 1850-style project folds.
> * *Emoji/emoticon annotation layer.* Bidirectional emoji↔emoticon rung plus optional {{Term}} layers (name, sentiment, entity type, category). CLDR annotations for names where available; sentiment/entity/category sources and file layout decided in the first annotation PR. Annotations in a separate lookup file keyed by code point, not mixed into the fold table.
> h2. Scope and acceptance criteria
> * (DONE in OPENNLP-1850) {{Alignment.andThen}} composition with tests, including expand-then-contract and contract-then-expand chains.
> * (DONE in OPENNLP-1850) The offset-aware contract on {{CharClass}} returning {{AlignedText}}, with round-trip tests for collapse, supplementary-dash contraction, supplementary-replacement expansion, and edge deletions.
> * An offset-aware mode on the rest of the {{CharSequenceNormalizer}} family and on {{TextNormalizer}}, returning a composed {{AlignedText}} for a multi-rung chain (requires resolving the NFC/NFKC edit-tracking question).
> * A bundled, provenance-tagged CaseFolding lookup with a completeness audit test against upstream {{CaseFolding.txt}}.
> * Round-trip tests: for representative expanding (ellipsis, eszett, ligature, fraction) and contracting (run collapse, supplementary dash, NFC compose) inputs, a position in the normalized text maps back to the correct original characters through the full pipeline.
> * A bidirectional, opt-in emoji/emoticon rung backed by the same provenance-tagged lookup, offset-aware like every other rung, with round-trip tests in both directions.
> * A small, provenance-tagged annotation layer on the lookup (name, sentiment, entity type, category), surfaced through the {{Term}} model and wired as optional input to the Sentiment Detection, Name Finder, and Document Categorizer components.
> * Consistent {{WordType}}-level classification of emoji and emoticons so token counts and ML feature generators treat them as one class.
> h2. Design guidelines and codebase consistency
> OPENNLP-1850 merged the normalization engine the rest of this work builds on. The following are already in place on {{main}} and are assumed by this issue:
> * {{CharClass}} / {{CodePointSet}} cursor engine — UCD-sourced {{White_Space}} and {{Dash}} sets, no regex on text paths
> * {{Alignment}} / {{AlignedText}}, {{OffsetAwareNormalizer}}, {{TextNormalizer.buildAligned()}}, and the {{CharClass}} {{*Aligned}} variants
> * UAX #29 word tokenizer ({{WordSegmenter}}, {{WordTokenizer}}, {{WordType}})
> * {{Term}} / {{TermAnalyzer}} and per-language {{NormalizationProfile(s)}}
> * DL integration: Unicode {{White_Space}} chunking, optional input folding, {{NameFinderDL.findInOriginal()}} via {{Alignment}}
> h3. Design principals
> Three design goals govern that engine and should govern new normalization work going forward:
> # *Standards-sourced character classes.* UCD properties ({{White_Space}}, {{Dash}}, …), not {{Character.isWhitespace}} or regex character classes like {{\s}}.
> # *Cursor-based text transforms.* Single forward code-point scans for normalization and token-boundary logic on user text. Regex is fine for config parsing and other non-user-text utilities.
> # *Offset preservation where spans matter.* Length-changing folds compose through {{Alignment}} so consumers can map normalized positions back to the original input. That is this issue's job for the standard-engine folds {{buildAligned()}} rejects today; the {{CharClass}} rungs and DL path already do it.
> The manual (OPENNLP-1850 docs) should describe these as *design goals* the engine follows and that new code should follow. Do not claim every legacy class in {{opennlp.tools.util.normalizer}} or all of OpenNLP already meets them — several pre-existing normalizers still use regex and {{StringUtil}} still uses JVM whitespace predicates. That gap is tracked below, not a defect in the merged engine.
> Target manual wording ({{normalizer.xml}} intro):
> {quote}
> The normalization engine follows three design goals: standards-sourced character classes, cursor-based text transforms, and offset preservation where spans matter. New normalization code should align with these goals; older components are migrated on a best-effort basis (tracked in OPENNLP-1852).
> {quote}
> State the no-regex / UCD rationale once in the normalizer chapter; DL and tokenizer chapters should xref rather than restate.
> h3. Consistency scoreboard
> Sub-tasks to file under this epic when ready; until then this table is the scoreboard:
> || # || Summary || Component || Priority || Status ||
> | 1 | Audit {{split("\\s+")}} and {{Character.isWhitespace}} call sites outside the 1850 engine | codebase-wide | medium | todo |
> | 2 | Migrate {{StringUtil}} whitespace to UCD-backed API or document intentional exception | {{opennlp-api}} | medium | todo |
> | 3 | Replace regex in legacy {{TwitterCharSequenceNormalizer}} | {{opennlp-runtime}} | low | todo |
> | 4 | Replace regex in legacy {{UrlCharSequenceNormalizer}} | {{opennlp-runtime}} | low | todo |
> | 5 | Replace regex in legacy {{ShrinkCharSequenceNormalizer}} | {{opennlp-runtime}} | low | todo |
> | 6 | Replace regex in legacy {{NumberCharSequenceNormalizer}} | {{opennlp-runtime}} | low | todo |
> | 7 | Replace regex in legacy {{EmojiCharSequenceNormalizer}} | {{opennlp-runtime}} | low | todo |
> | 8 | Audit spellcheck / extension normalizers for regex-on-text paths | extensions | low | todo |
> | 9 | Soften manual intro to design-goals wording; trim duplicated no-regex/ReDoS prose across chapters | {{opennlp-docs}} | medium | todo |
> | 10 | Add design-guidelines xref in {{normalizer.xml}} pointing here | {{opennlp-docs}} | low | todo |
> When the scoreboard is substantially green, tighten the manual from "design goals" to stronger statements. Scoreboard items do not block the offset-aware fold work in this issue.
> h2. Out of scope
> * * Changing default normalization behavior; this issue adds offset-aware capability and the provenance model, it does not turn expanding folds on by default.
> * Legacy regex normalizer migration, {{StringUtil}} whitespace standardization, and manual wording — tracked in the scoreboard above, not acceptance criteria for this issue.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)