You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@opennlp.apache.org by "Richard Zowalla (Jira)" <ji...@apache.org> on 2026/09/18 10:05:00 UTC

[jira] [Created] (OPENNLP-1975) RegEx removal part 9: Correct the output of the regex sites in TokenSampleStream, ParserTool, ConlluStream, DownloadUtil and FeatureGeneratorUtil

Richard Zowalla created OPENNLP-1975:
----------------------------------------

             Summary: RegEx removal part 9: Correct the output of the regex sites in TokenSampleStream, ParserTool, ConlluStream, DownloadUtil and FeatureGeneratorUtil
                 Key: OPENNLP-1975
                 URL: https://issues.apache.org/jira/browse/OPENNLP-1975
             Project: OpenNLP
          Issue Type: Bug
            Reporter: Richard Zowalla
            Assignee: Kristian Rickert
             Fix For: 3.0.0


Split out of OPENNLP\-1928 \(PR #1275\). Depends on OPENNLP\-1928 \(RegEx removal part 1\).

Replaces the regular expressions of these sites with scans that follow the intended format instead of copying the regex quirks. The output changes:* {{TokenSampleStream}} \(English\): tokens are split on Unicode White\_Space; a leading space no longer yields an empty first token; a line without a token resets the open\-quote state; a token with a letter or digit of any script is a word.
* {{ParserTool}}: brackets that follow each other are each separated \({{a\)\)b}} gives {{a \) \) b}}\).
* {{ConlluStream}}: a malformed multiword token id \({{3\-}}, {{\-4}}, {{3\-\-4}}, {{1\-2\-3}}, {{4\-3}}\) fails with an {{InvalidFormatException}} naming the id.
* {{DownloadUtil}}: a blank checksum file is an {{IOException}} saying so.
* {{FeatureGeneratorUtil}}: the {{cp}} class requires exactly one capital and one period.

Before merge: tokenizer eval build that reads through {{TokenSampleStream}}.





--
This message was sent by Atlassian Jira
(v8.20.10#820010)