You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@opennlp.apache.org by GitBox <gi...@apache.org> on 2026/09/15 14:40:09 UTC
[GH] (opennlp/OPENNLP-1928-regex-removal-trivial): Workflow run "Java CI" is working again!
The GitHub Actions job "Java CI" on opennlp.git/OPENNLP-1928-regex-removal-trivial has succeeded.
Run started by GitHub user krickert (triggered by krickert).
Head commit for run:
ccea67059c9e6ec4473a0e4986392adfe7bb1ac2 / Kristian Rickert <kr...@apache.org>
OPENNLP-1928: Code the intended behavior where scans had copied regex quirks
Scans now do what the code meant instead of what the regular expression
would accept:
- EmojiCharSequenceNormalizer replaces only supplementary-plane code
points. BMP characters, hyphens and unpaired surrogates are kept.
- ParserTool separates brackets on Unicode whitespace in one pass, so
brackets that follow each other are all spaced and no space is doubled.
- The FeatureGeneratorUtil capital-period feature needs one capital and
one period; a trailing line break no longer qualifies.
- BioCodec rejects a line terminator at any position of an outcome type.
- DownloadUtil accepts a checksum file with leading Unicode whitespace
and treats a blank file as missing.
- The AD streams split on the toolkit whitespace definition, drop empty
underscore parts and judge punctuation lines by code point.
- ConlluStream fails a malformed multiword token id with an
InvalidFormatException instead of skipping it.
StringUtil adds isAsciiLetter, isAsciiDigit, endOfAsciiDigits,
isLineTerminator and indexOfLineTerminator, shared by BioCodec,
ADSentenceSampleStream and ConlluStream here and needed by the related
OPENNLP-1929 to OPENNLP-1935 branches. isAsciiWhitespace and
splitOnAsciiWhitespace are removed; callers use the Unicode-aware
StringUtil.isWhitespace and WhitespaceTokenizer.
The manual documents the bracket handling of the parser tool, the
CoNLL-U multiword id and language code rules, the AD reader rules and
the emoji normalizer in the language detector chapter. Tests cover the
accept and reject side of each scan, the Unicode plane boundaries and
Unicode whitespace separators.
Report URL: https://github.com/apache/opennlp/actions/runs/34981287276
With regards,
GitHub Actions via GitBox