You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@opennlp.apache.org by sm...@apache.org on 2026/08/30 13:11:13 UTC
(opennlp) branch main updated: OPENNLP-1922: Fix ReDoS / stack exhaustion in RegexNameFinderFactory EMAIL and URL patterns (#1250)
This is an automated email from the ASF dual-hosted git repository.
smarthi pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/opennlp.git
The following commit(s) were added to refs/heads/main by this push:
new 8f7befe44 OPENNLP-1922: Fix ReDoS / stack exhaustion in RegexNameFinderFactory EMAIL and URL patterns (#1250)
8f7befe44 is described below
commit 8f7befe4439530eda66221b1f4a927a00fd91a20
Author: Suneel Marthi <sm...@apache.org>
AuthorDate: Sun Aug 30 09:11:08 2026 -0400
OPENNLP-1922: Fix ReDoS / stack exhaustion in RegexNameFinderFactory EMAIL and URL patterns (#1250)
* Fix ReDoS / stack exhaustion in RegexNameFinderFactory EMAIL and URL patterns
The built-in EMAIL and URL regexes in DEFAULT_REGEX_NAME_FINDER were
vulnerable to catastrophic backtracking and deep recursion
(CWE-1333 / CWE-400, and CWE-674 for URL). RegexNameFinder feeds the
concatenated token string straight into Matcher.find(), so a single
crafted long token drives the matcher into quadratic backtracking
(EMAIL) or a StackOverflowError (URL).
- EMAIL: bounded, possessive quantifiers with RFC 5321 length limits;
drop the malformed host:garbage sub-branch of the IP-literal
alternative. Quoted-string local part is retained.
- URL: flatten the nested (((/(...)+)+|/)+ ...) and (...(&...)*)*
groups to single-level possessive repetition with segment caps.
Also fixes two latent bugs: unescaped dot in "www." and the
"%[a-f\d{2}]" character class that never matched a percent octet.
- Capturing groups made non-capturing (unused by RegexNameFinder).
Adds a regression test asserting the built-in finders complete
promptly on 100k-char attack inputs.
* Fix ReDoS / stack exhaustion in RegexNameFinderFactory EMAIL and URL patterns
The built-in EMAIL and URL regexes in DEFAULT_REGEX_NAME_FINDER were
vulnerable to catastrophic backtracking and deep recursion
(CWE-1333 / CWE-400, and CWE-674 for URL). RegexNameFinder feeds the
concatenated token string straight into Matcher.find(), so a single
crafted long token drives the matcher into quadratic backtracking
(EMAIL) or a StackOverflowError (URL).
- EMAIL: bounded, possessive quantifiers with RFC 5321 length limits;
drop the malformed host:garbage sub-branch of the IP-literal
alternative. Quoted-string local part is retained.
- URL: flatten the nested (((/(...)+)+|/)+ ...) and (...(&...)*)*
groups to single-level possessive repetition with segment caps.
Also fixes two latent bugs: unescaped dot in "www." and the
"%[a-f\d{2}]" character class that never matched a percent octet.
- Capturing groups made non-capturing (unused by RegexNameFinder).
Adds a regression test asserting the built-in finders complete
promptly on 100k-char attack inputs.
# Conflicts:
# opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/namefind/RegexNameFinderFactoryTest.java
* Fix ReDoS / stack exhaustion in RegexNameFinderFactory EMAIL and URL patterns
The built-in EMAIL and URL regexes in DEFAULT_REGEX_NAME_FINDER were
vulnerable to catastrophic backtracking and deep recursion
(CWE-1333 / CWE-400, and CWE-674 for URL). RegexNameFinder feeds the
concatenated token string straight into Matcher.find(), so a single
crafted long token drives the matcher into quadratic backtracking
(EMAIL) or a StackOverflowError (URL).
Both patterns are rewritten so that every quantifier is bounded by a
constant. Bounds alone remove the exponential backtracking and cap the
recursion depth; the repetitions stay bounded ({0,255} / {1,63}) rather
than becoming * / + so that the trailing \b can still backtrack into a
matched run (a plain * reintroduces the overflow on a long path segment).
- EMAIL: bounded quantifiers with RFC 5321 length limits; drop the
malformed host:garbage sub-branch of the IP-literal alternative.
Quoted-string local part is retained.
- URL: flatten the nested (((/(...)+)+|/)+ ...) and (...(&...)*)* groups
to single-level bounded repetition with segment caps. Also fixes two
latent bugs: unescaped dot in "www." and the "%[a-f\d{2}]" character
class that never matched a percent octet. Capturing groups made
non-capturing (unused by RegexNameFinder).
Adds regression tests: the built-in finders complete promptly on
100k-char attack inputs (incl. the reported ?a&a&a&... query string),
and the URL finder still matches URLs with a trailing slash, sentence
period, '#' or '&'.
* Apply suggestion from @rzo1
Co-authored-by: Richard Zowalla <13...@users.noreply.github.com>
* Apply suggestion from @rzo1
Co-authored-by: Richard Zowalla <13...@users.noreply.github.com>
---------
Co-authored-by: Richard Zowalla <13...@users.noreply.github.com>
---
.../tools/namefind/RegexNameFinderFactory.java | 39 +++++----
.../tools/namefind/RegexNameFinderFactoryTest.java | 94 ++++++++++++++++++++++
2 files changed, 118 insertions(+), 15 deletions(-)
diff --git a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/namefind/RegexNameFinderFactory.java b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/namefind/RegexNameFinderFactory.java
index 7c2bc79ec..0890ae051 100644
--- a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/namefind/RegexNameFinderFactory.java
+++ b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/namefind/RegexNameFinderFactory.java
@@ -107,12 +107,18 @@ public class RegexNameFinderFactory {
@Override
public Map<String, Pattern[]> getRegexMap() {
Pattern[] p = new Pattern[1];
- p[0] = Pattern.compile("([a-z0-9!#$%&'*+/=?^_`{|}~-]+(?:\\.[a-z0-9!#$%&'*+/=?^_`{|}~-]+)*" +
- "|\"([\\x01-\\x08\\x0b\\x0c\\x0e-\\x1f\\x21\\x23-\\x5b\\x5d-\\x7f]|\\\\[\\x01-\\x09" +
- "\\x0b\\x0c\\x0e-\\x7f])*\")@(?:(?:[a-z0-9]([a-z0-9-]*[a-z0-9])?\\.)+[a-z0-9](?:[a-z0-9-]" +
- "*[a-z0-9])?|\\[((25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\\.){3}(?:25[0-5]|2[0-4][0-9]|[01]" +
- "?[0-9][0-9]?|[a-z0-9-]*[a-z0-9]:(?:[\\x01-\\x08\\x0b\\x0c\\x0e-\\x1f\\x21-\\x5a\\x53-\\x7f]" +
- "|\\\\[\\x01-\\x09\\x0b\\x0c\\x0e-\\x7f])+)\\])", Pattern.CASE_INSENSITIVE);
+ // Every quantifier is bounded by a constant, which removes both the exponential
+ // backtracking and the recursion depth that made the old pattern a ReDoS vector.
+ // Limits follow RFC 5321: local part <= 64 chars, each domain label <= 63 chars.
+ p[0] = Pattern.compile(
+ "(?<![a-z0-9!#$%&'*+/=?^_`{|}~.-])" +
+ "(?:[a-z0-9!#$%&'*+/=?^_`{|}~-]{1,64}(?:\\.[a-z0-9!#$%&'*+/=?^_`{|}~-]{1,64}){0,10}" +
+ "|\"(?:[\\x01-\\x08\\x0b\\x0c\\x0e-\\x1f\\x21\\x23-\\x5b\\x5d-\\x7f]" +
+ "|\\\\[\\x01-\\x09\\x0b\\x0c\\x0e-\\x7f]){0,255}\")" +
+ "@(?:(?:[a-z0-9](?:[a-z0-9-]{0,61}[a-z0-9])?\\.){1,20}" +
+ "[a-z0-9](?:[a-z0-9-]{0,61}[a-z0-9])?" +
+ "|\\[(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\\.){3}" +
+ "(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\\])", Pattern.CASE_INSENSITIVE);
Map<String, Pattern[]> regexMap = new HashMap<>();
regexMap.put(getType(), p);
return regexMap;
@@ -127,16 +133,19 @@ public class RegexNameFinderFactory {
@Override
public Map<String, Pattern[]> getRegexMap() {
Pattern[] p = new Pattern[1];
- p[0] = Pattern.compile("\\b(((ht|f)tp(s?)\\:\\/\\/|~\\/|\\/)|www.)"
- + "(\\w+:\\w+@)?(([-\\w]+\\.)+(com|org|net|gov"
+ // Flattened to single-level groups and every quantifier bounded by a constant.
+ // The bounds alone remove the nested-quantifier backtracking and the recursion
+ // depth that caused the StackOverflowError; they must stay bounded ({0,255} /
+ // {1,63} rather than * / +) or a long path segment reintroduces the overflow.
+ p[0] = Pattern.compile("\\b(?:(?:ht|f)tps?://|~/|/|www\\.)"
+ + "(?:\\w{1,63}:\\w{1,63}@)?"
+ + "(?:[-\\w]{1,63}\\.){1,20}(?:com|org|net|gov"
+ "|mil|biz|info|mobi|name|aero|jobs|museum"
- + "|travel|[a-z]{2}))(:[\\d]{1,5})?"
- + "(((\\/([-\\w~!$+|.,=]|%[a-f\\d]{2})+)+|\\/)+|\\?|#)?"
- + "((\\?([-\\w~!$+|.,*:]|%[a-f\\d{2}])+=?"
- + "([-\\w~!$+|.,*:=]|%[a-f\\d]{2})*)"
- + "(&(?:[-\\w~!$+|.,*:]|%[a-f\\d{2}])+=?"
- + "([-\\w~!$+|.,*:=]|%[a-f\\d]{2})*)*)*"
- + "(#([-\\w~!$+|.,*:=]|%[a-f\\d]{2})*)?\\b", Pattern.CASE_INSENSITIVE);
+ + "|travel|[a-z]{2})(?::\\d{1,5})?"
+ + "(?:/(?:[-\\w~!$+|.,=]|%[a-f\\d]{2}){0,255}){0,50}"
+ + "(?:\\?(?:[-\\w~!$+|.,*:=]|%[a-f\\d]{2}){0,255}"
+ + "(?:&(?:[-\\w~!$+|.,*:=]|%[a-f\\d]{2}){0,255}){0,50})?"
+ + "(?:#(?:[-\\w~!$+|.,*:=]|%[a-f\\d]{2}){0,255})?\\b", Pattern.CASE_INSENSITIVE);
Map<String, Pattern[]> regexMap = new HashMap<>();
regexMap.put(getType(), p);
return regexMap;
diff --git a/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/namefind/RegexNameFinderFactoryTest.java b/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/namefind/RegexNameFinderFactoryTest.java
index 809429925..6fdea6414 100644
--- a/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/namefind/RegexNameFinderFactoryTest.java
+++ b/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/namefind/RegexNameFinderFactoryTest.java
@@ -17,6 +17,7 @@
package opennlp.tools.namefind;
+import java.time.Duration;
import java.util.Arrays;
import java.util.List;
@@ -90,6 +91,99 @@ public class RegexNameFinderFactoryTest {
Assertions.assertEquals("45", tokens[latLongSpan2.getStart()]);
}
+ /**
+ * Crafted inputs that used to drive the built-in EMAIL and URL patterns into
+ * catastrophic backtracking / deep recursion (ReDoS, CWE-1333 / CWE-400 / CWE-674).
+ * The hardened patterns must finish quickly regardless of input length.
+ */
+ @Test
+ void testBuiltinPatternsAreNotVulnerableToReDoS() {
+ final String emailLocalBlowup = "a".repeat(100_000) + "@ ";
+ final String emailDomainBlowup = "x@a" + "-a".repeat(60_000) + " ";
+ final String urlPathRecursion = "http://a.com/" + "a".repeat(100_000) + " ";
+ final String urlNestedBlowup = "http://a.com" + "/a".repeat(50_000) + "%z";
+ // The reported StackOverflowError repro: a long ?a&a&a&... query string, which
+ // drove the nested (&(...)+ ... )* group in the old URL pattern into deep recursion.
+ final String urlQueryRecursion = "http://a.com/p?a" + "&a".repeat(50_000) + "= ";
+
+ Assertions.assertTimeoutPreemptively(Duration.ofSeconds(2), () -> {
+ for (String attack : new String[] {
+ emailLocalBlowup, emailDomainBlowup, urlPathRecursion, urlNestedBlowup, urlQueryRecursion}) {
+ String[] tokens = WhitespaceTokenizer.INSTANCE.tokenize(attack);
+ regexNameFinder.find(tokens);
+ }
+ });
+ }
+
+ /**
+ * Regression tests for the hardened built-in patterns: a trailing slash or a
+ * sentence-final period must not cause the match to be abandoned entirely.
+ */
+ @Test
+ void testUrlWithTrailingSlashOrSentencePunctuation() {
+ RegexNameFinder urlFinder = RegexNameFinderFactory.getDefaultRegexNameFinders(
+ RegexNameFinderFactory.DEFAULT_REGEX_NAME_FINDER.URL);
+ final String input = "see http://a.com/ or http://example.com/path/ or www.google.com.";
+ Span[] spans = urlFinder.find(input);
+ Assertions.assertEquals(3, spans.length);
+ Assertions.assertEquals("http://a.com",
+ input.substring(spans[0].getStart(), spans[0].getEnd()));
+ Assertions.assertEquals("http://example.com/path",
+ input.substring(spans[1].getStart(), spans[1].getEnd()));
+ Assertions.assertEquals("www.google.com",
+ input.substring(spans[2].getStart(), spans[2].getEnd()));
+ }
+
+ @Test
+ void testUrlKeepsPortWhenFollowedBySlash() {
+ RegexNameFinder urlFinder = RegexNameFinderFactory.getDefaultRegexNameFinders(
+ RegexNameFinderFactory.DEFAULT_REGEX_NAME_FINDER.URL);
+ final String input = "http://example.com:8080/";
+ Span[] spans = urlFinder.find(input);
+ Assertions.assertEquals(1, spans.length);
+ Assertions.assertEquals("http://example.com:8080",
+ input.substring(spans[0].getStart(), spans[0].getEnd()));
+ }
+
+ @Test
+ void testEmailAtEndOfSentence() {
+ RegexNameFinder emailFinder = RegexNameFinderFactory.getDefaultRegexNameFinders(
+ RegexNameFinderFactory.DEFAULT_REGEX_NAME_FINDER.EMAIL);
+ final String input = "mail me at a@b.com. and a@sub.b.co.uk.";
+ Span[] spans = emailFinder.find(input);
+ Assertions.assertEquals(2, spans.length);
+ Assertions.assertEquals("a@b.com", input.substring(spans[0].getStart(), spans[0].getEnd()));
+ Assertions.assertEquals("a@sub.b.co.uk",
+ input.substring(spans[1].getStart(), spans[1].getEnd()));
+ }
+
+ /**
+ * The hardened URL pattern must still match a URL that is followed by a trailing
+ * delimiter (a path slash, a sentence-final period, {@code #} or {@code &}), stopping
+ * the span before the delimiter rather than abandoning the match.
+ */
+ @Test
+ void testUrlTrailingDelimiters() {
+ assertFirstUrl("http://a.com", "http://a.com/");
+ assertFirstUrl("http://example.com/path", "http://example.com/path/");
+ assertFirstUrl("ftp://files.example.com/pub", "ftp://files.example.com/pub/");
+ assertFirstUrl("www.google.com", "check www.google.com/ now");
+ assertFirstUrl("http://example.com", "I saw http://example.com. Then");
+ assertFirstUrl("www.google.com", "www.google.com.");
+ assertFirstUrl("http://a.com/path", "http://a.com/path.");
+ assertFirstUrl("http://a.com", "http://a.com/#");
+ assertFirstUrl("http://example.com:8080", "http://example.com:8080/");
+ assertFirstUrl("http://a.com/p?q=1", "http://a.com/p?q=1&");
+ }
+
+ private static void assertFirstUrl(String expected, String input) {
+ RegexNameFinder urlFinder = RegexNameFinderFactory.getDefaultRegexNameFinders(
+ RegexNameFinderFactory.DEFAULT_REGEX_NAME_FINDER.URL);
+ Span[] spans = urlFinder.find(input);
+ Assertions.assertTrue(spans.length > 0, "no URL match for: " + input);
+ Assertions.assertEquals(expected, input.substring(spans[0].getStart(), spans[0].getEnd()));
+ }
+
@Test
void testMgrs() {
String[] tokens = WhitespaceTokenizer.INSTANCE.tokenize(text);