You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@opennlp.apache.org by "rzo1 (via GitHub)" <gi...@apache.org> on 2026/10/02 08:37:07 UTC
[PR] Minor: Reuse StringUtil split, BOM, digit and line-terminator helpers (opennlp)
rzo1 opened a new pull request, #1352:
URL: https://github.com/apache/opennlp/pull/1352
Replaces hand-written copies of split, BOM stripping, ASCII digit scanning and line-terminator checks in ml-commons, formats, runtime and spellcheck with the existing StringUtil helpers, and adds StringUtil.isUnpairedSurrogate for the two surrogate checks. The only semantic change is that the post-M5 MeCab and unigram lexicon parsers now split on Unicode White_Space independent of WhitespaceMode, matching the trim they already apply.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opennlp.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Minor: Reuse StringUtil split, BOM, digit and line-terminator helpers (opennlp)
Posted by "rzo1 (via GitHub)" <gi...@apache.org>.
rzo1 commented on PR #1352:
URL: https://github.com/apache/opennlp/pull/1352#issuecomment-5954436252
Updated the description. Thx for review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opennlp.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Minor: Reuse StringUtil split, BOM, digit and line-terminator helpers (opennlp)
Posted by "krickert (via GitHub)" <gi...@apache.org>.
krickert merged PR #1352:
URL: https://github.com/apache/opennlp/pull/1352
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opennlp.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Minor: Reuse StringUtil split, BOM, digit and line-terminator helpers (opennlp)
Posted by "mawiesne (via GitHub)" <gi...@apache.org>.
mawiesne commented on code in PR #1352:
URL: https://github.com/apache/opennlp/pull/1352#discussion_r4172936009
##########
opennlp-api/src/main/java/opennlp/tools/util/StringUtil.java:
##########
@@ -419,6 +419,19 @@ public static boolean isLineTerminator(int codePoint) {
|| codePoint == '\u2028' || codePoint == '\u2029';
}
+ /**
+ * Tests whether a code point taken from a {@link CharSequence} is a lone surrogate,
Review Comment:
"Tests" should read "Checks" here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opennlp.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Minor: Reuse StringUtil split, BOM, digit and line-terminator helpers (opennlp)
Posted by "krickert (via GitHub)" <gi...@apache.org>.
krickert commented on PR #1352:
URL: https://github.com/apache/opennlp/pull/1352#issuecomment-5952062091
Checked every replaced loop against the StringUtil method it now calls, on empty input, leading and trailing separators, supplementary code points, lone surrogates, BOM, all five line terminators and the odd whitespace code points. Identical under UNICODE mode everywhere; the only difference is the disclosed LEGACY-mode change in the two lattice parsers.
The 4 tests ran only in UNICODE, so added LEGACY as well. It parameterizes them over WhitespaceMode with a reset after each test, so the Javadoc is right now. Both classes green.
Nit: the description should also list opennlp-api (MecabDictionary, UnigramSegmenter, StringUtil) and ml-maxent (BasicContextGenerator).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opennlp.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Minor: Reuse StringUtil split, BOM, digit and line-terminator helpers (opennlp)
Posted by "mawiesne (via GitHub)" <gi...@apache.org>.
mawiesne commented on PR #1352:
URL: https://github.com/apache/opennlp/pull/1352#issuecomment-5968683432
@rzo1 needs rebase and resolve.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opennlp.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Minor: Reuse StringUtil split, BOM, digit and line-terminator helpers (opennlp)
Posted by "rzo1 (via GitHub)" <gi...@apache.org>.
rzo1 commented on PR #1352:
URL: https://github.com/apache/opennlp/pull/1352#issuecomment-5968863409
Addressed your comments. Thx @mawiesne
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opennlp.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org