You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@opennlp.apache.org by "Richard Zowalla (Jira)" <ji...@apache.org> on 2026/09/18 10:05:00 UTC
[jira] [Created] (OPENNLP-1975) RegEx removal part 9: Correct the output of the regex sites in TokenSampleStream, ParserTool, ConlluStream, DownloadUtil and FeatureGeneratorUtil
Richard Zowalla created OPENNLP-1975:
----------------------------------------
Summary: RegEx removal part 9: Correct the output of the regex sites in TokenSampleStream, ParserTool, ConlluStream, DownloadUtil and FeatureGeneratorUtil
Key: OPENNLP-1975
URL: https://issues.apache.org/jira/browse/OPENNLP-1975
Project: OpenNLP
Issue Type: Bug
Reporter: Richard Zowalla
Assignee: Kristian Rickert
Fix For: 3.0.0
Split out of OPENNLP\-1928 \(PR #1275\). Depends on OPENNLP\-1928 \(RegEx removal part 1\).
Replaces the regular expressions of these sites with scans that follow the intended format instead of copying the regex quirks. The output changes:* {{TokenSampleStream}} \(English\): tokens are split on Unicode White\_Space; a leading space no longer yields an empty first token; a line without a token resets the open\-quote state; a token with a letter or digit of any script is a word.
* {{ParserTool}}: brackets that follow each other are each separated \({{a\)\)b}} gives {{a \) \) b}}\).
* {{ConlluStream}}: a malformed multiword token id \({{3\-}}, {{\-4}}, {{3\-\-4}}, {{1\-2\-3}}, {{4\-3}}\) fails with an {{InvalidFormatException}} naming the id.
* {{DownloadUtil}}: a blank checksum file is an {{IOException}} saying so.
* {{FeatureGeneratorUtil}}: the {{cp}} class requires exactly one capital and one period.
Before merge: tokenizer eval build that reads through {{TokenSampleStream}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)