[jira] [Created] (OPENNLP-1876) Remove regex over cursors TwitterCharSequenceNormalizer, UrlCharSequenceNormalizer, ShrinkCharSequenceNormalizer, and NumberCharSequenceNormalizer

["Kristian Rickert (Jira)" <ji...@apache.org>]

Kristian Rickert created OPENNLP-1876:
-----------------------------------------

             Summary: Remove regex over cursors TwitterCharSequenceNormalizer, UrlCharSequenceNormalizer, ShrinkCharSequenceNormalizer, and NumberCharSequenceNormalizer 
                 Key: OPENNLP-1876
                 URL: https://issues.apache.org/jira/browse/OPENNLP-1876
             Project: OpenNLP
          Issue Type: Improvement
    Affects Versions: 3.0.0-M4
            Reporter: Kristian Rickert
            Assignee: Kristian Rickert
             Fix For: 3.0.0-M5


h2. Summary

The legacy normalizers {{TwitterCharSequenceNormalizer}}, {{UrlCharSequenceNormalizer}}, {{ShrinkCharSequenceNormalizer}}, and {{NumberCharSequenceNormalizer}} apply regular expressions to user text, which runs against the OPENNLP-1850 design goal of cursor-based text transforms (a single forward code-point scan on user text, with regex reserved for config parsing). This ticket replaces those regex passes with cursor scans on the {{CharClass}} engine where they operate on user text, and audits the spellcheck and extension normalizers for the same pattern. These are pre-1850 rungs and are not in the blessed default chain, so the change is low-risk and output-preserving rather than a redesign.

h2. Scope

* Replace regex-on-text with a cursor scan (on the {{CharClass}}/{{CodePointSet}} primitives from OPENNLP-1850) in {{TwitterCharSequenceNormalizer}}, {{UrlCharSequenceNormalizer}}, {{ShrinkCharSequenceNormalizer}}, and {{NumberCharSequenceNormalizer}}.
* Where a normalizer's behavior is subtle, add characterization tests that pin the pre-change output first, then refactor, so the output is preserved byte for byte.
* Audit the spellcheck and extension normalizers for regex-on-text paths; convert the clear cases and file follow-ups for anything larger.

h2. Acceptance criteria

* The four normalizers use cursor scans on the text path; no regex is applied to user text in them.
* Output-preserving: characterization tests capture the prior behavior and pass unchanged after the refactor.
* The spellcheck and extension audit is complete, with conversions applied or follow-ups filed.

h2. Out of scope

* Making these rungs offset-aware. They are not in the blessed chain, and offset-safety for them is a separate concern.
* The {{StringUtil}} whitespace migration, which is a separate ticket.
* Regex in config parsing and non-user-text utilities.
* Adding or removing any of these rungs from a default chain.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)