[jira] [Updated] (OPENNLP-1928) RegEx removal part 1: Deterministic text scans replace trivial regular expressions

["Kristian Rickert (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/OPENNLP-1928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Kristian Rickert updated OPENNLP-1928:
--------------------------------------
    Summary: RegEx removal part 1:  Deterministic text scans replace trivial regular expressions  (was: RegEx removal part 1:  trival batch)

> RegEx removal part 1:  Deterministic text scans replace trivial regular expressions
> -----------------------------------------------------------------------------------
>
>                 Key: OPENNLP-1928
>                 URL: https://issues.apache.org/jira/browse/OPENNLP-1928
>             Project: OpenNLP
>          Issue Type: Improvement
>            Reporter: Kristian Rickert
>            Assignee: Kristian Rickert
>            Priority: Major
>          Time Spent: 1.5h
>  Remaining Estimate: 0h
>
> Regex isn't compatible with the correct whitespace characters and punctuation marks.  Every language's parser slightly changes.  This ticket focuses on all the trivial spots we found regex:
>  
> ||Site||Pattern removed||
> |{{DefaultPOSContextGenerator}}, {{DefaultLemmatizerContextGenerator}}|{{[A-Z]}}, {{[0-9]}} capital and digit detection|
> |{{FeatureGeneratorUtil}}|{{^[A-ZÄÖÜ]\.$}} capital-period check|
> |{{TokenPatternFeatureGenerator}}|{{[^a-zA-Z]}} non-letter check|
> |{{BioCodec}}, {{NameFinderME}}|{{(.+)-\w+}} typed-outcome split|
> |{{BrownCluster}}|tab split|
> |{{TokenSampleStream}}|{{\s+}} split and {{[A-Za-z0-9]}} check|
> |{{EmojiCharSequenceNormalizer}}|surrogate-range emoji pattern|
> |{{ConlluStream}}|token-id split on {{-}} and {{text_([a-z]\{2,3\})}} language comment|
> |{{ParserTool}}|the two untokenized-bracket spacing patterns|
> |{{DownloadUtil}}|checksum-line split and the {{<a href>}} link pattern|
> |{{ADNameSampleStream}}|tag, whitespace, underscore, hyphen, and alphanumeric patterns|
> |{{ADPOSSampleStream}}, {{ADSentenceSampleStream}}, {{ADSentenceStream}}|whitespace, metadata, and punctuation patterns|
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)