[jira] [Closed] (OPENNLP-1975) RegEx removal part 9: Correct the output of the regex sites in TokenSampleStream, ParserTool, ConlluStream, DownloadUtil and FeatureGeneratorUtil

["Richard Zowalla (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/OPENNLP-1975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Richard Zowalla closed OPENNLP-1975.
------------------------------------
    Resolution: Fixed

> RegEx removal part 9: Correct the output of the regex sites in TokenSampleStream, ParserTool, ConlluStream, DownloadUtil and FeatureGeneratorUtil
> -------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: OPENNLP-1975
>                 URL: https://issues.apache.org/jira/browse/OPENNLP-1975
>             Project: OpenNLP
>          Issue Type: Bug
>            Reporter: Richard Zowalla
>            Assignee: Kristian Rickert
>            Priority: Major
>             Fix For: 3.0.0
>
>          Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> Split out of OPENNLP\-1928 \(PR #1275\). Depends on OPENNLP\-1928 \(RegEx removal part 1\).
> Replaces the regular expressions of these sites with scans that follow the intended format instead of copying the regex quirks. The output changes:* {{TokenSampleStream}} \(English\): tokens are split on Unicode White\_Space; a leading space no longer yields an empty first token; a line without a token resets the open\-quote state; a token with a letter or digit of any script is a word.
> * {{ParserTool}}: brackets that follow each other are each separated \({{a\)\)b}} gives {{a \) \) b}}\).
> * {{ConlluStream}}: a malformed multiword token id \({{3\-}}, {{\-4}}, {{3\-\-4}}, {{1\-2\-3}}, {{4\-3}}\) fails with an {{InvalidFormatException}} naming the id.
> * {{DownloadUtil}}: a blank checksum file is an {{IOException}} saying so.
> * {{FeatureGeneratorUtil}}: the {{cp}} class requires exactly one capital and one period.
> Before merge: tokenizer eval build that reads through {{TokenSampleStream}}.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)