[jira] [Resolved] (OPENNLP-1876) Replace regex with cursor scans in the legacy CharSequenceNormalizers

["Martin Wiesner (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/OPENNLP-1876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Martin Wiesner resolved OPENNLP-1876.
-------------------------------------
    Resolution: Delivered

> Replace regex with cursor scans in the legacy CharSequenceNormalizers
> ---------------------------------------------------------------------
>
>                 Key: OPENNLP-1876
>                 URL: https://issues.apache.org/jira/browse/OPENNLP-1876
>             Project: OpenNLP
>          Issue Type: Improvement
>    Affects Versions: 3.0.0-M4
>            Reporter: Kristian Rickert
>            Assignee: Kristian Rickert
>            Priority: Major
>             Fix For: 3.0.0-M5
>
>          Time Spent: 6h
>  Remaining Estimate: 0h
>
> h2. Summary
> The legacy normalizers {{TwitterCharSequenceNormalizer}}, {{UrlCharSequenceNormalizer}}, {{ShrinkCharSequenceNormalizer}}, and {{NumberCharSequenceNormalizer}} apply regular expressions to user text, which runs against the OPENNLP-1850 design goal of cursor-based text transforms (a single forward code-point scan on user text, with regex reserved for config parsing). This ticket replaces those regex passes with cursor scans on the {{CharClass}} engine where they operate on user text, and audits the spellcheck and extension normalizers for the same pattern. These are pre-1850 rungs and are not in the blessed default chain, so the change is low-risk and output-preserving rather than a redesign.
> h2. Scope
> * Replace regex-on-text with a cursor scan (on the {{CharClass}}/{{CodePointSet}} primitives from OPENNLP-1850) in {{TwitterCharSequenceNormalizer}}, {{UrlCharSequenceNormalizer}}, {{ShrinkCharSequenceNormalizer}}, and {{NumberCharSequenceNormalizer}}.
> * Where a normalizer's behavior is subtle, add characterization tests that pin the pre-change output first, then refactor, so the output is preserved byte for byte.
> * Audit the spellcheck and extension normalizers for regex-on-text paths; convert the clear cases and file follow-ups for anything larger.
> h2. Acceptance criteria
> * The four normalizers use cursor scans on the text path; no regex is applied to user text in them.
> * Output-preserving: characterization tests capture the prior behavior and pass unchanged after the refactor.
> * The spellcheck and extension audit is complete, with conversions applied or follow-ups filed.
> h2. Out of scope
> * Making these rungs offset-aware. They are not in the blessed chain, and offset-safety for them is a separate concern.
> * The {{StringUtil}} whitespace migration, which is a separate ticket.
> * Regex in config parsing and non-user-text utilities.
> * Adding or removing any of these rungs from a default chain.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)