You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@trafficserver.apache.org by "bryancall (via GitHub)" <gi...@apache.org> on 2026/09/15 04:43:31 UTC
[PR] Regex: hold the JIT stack in a pthread key and inherit the shared match context [trafficserver]
bryancall opened a new pull request, #13683:
URL: https://github.com/apache/trafficserver/pull/13683
Fixes #13660.
First of four, split out of #13671 at review request.
Two of the remaining three, #PR3 and #PR4, are independent of this one and of each
other. The fourth, making the pcre2 contexts process-wide, builds directly on this
change and cannot stand without it, so it follows once this merges rather than
carrying this commit along for the ride.
### A caller-supplied `RegexMatchContext` ran on a 32 KiB JIT stack
`RegexMatchContext`'s constructor called `pcre2_match_context_create(nullptr)`, which
builds a context that configures nothing. A caller who wanted only to set a match limit
therefore silently gave up everything the shared context provides, including its 1 MiB
JIT stack, and PCRE2 fell back to its own 32 KiB machine-stack block. That block resolves
about 1,362 bytes of a subject that backtracks once per character; a production
`regex_remap` rule hit the bound at 1,377 bytes of query string. `regex_remap` and the
`esi` URL validator are the only two callers, and both were affected.
Copy the shared context instead, so a caller overrides only what it means to override
and anything added to the shared context later propagates on its own.
### The JIT stack moves to a pthread key
A shared match context needs a per thread JIT stack, and PCRE2 supplies one through a
callback invoked at match time rather than a pointer baked in when the context is built.
The obvious place to keep that stack is a `thread_local`, but a `thread_local` with a
destructor registers it through `__cxa_thread_atexit`, which takes the dynamic loader
lock. Doing that from a match inverts lock order against a `dlopen` caller running a
plugin's static initialization; `Diags::tag_activated` documents that exact deadlock.
A pthread key registers its destructor once, at key creation, and never from the matching
path. `pthread_key_create` failure is handled: `jit_stack_key` is zero initialized and key
0 can belong to another subsystem, so a failed create returns null and PCRE2 falls back to
its own stack, which `pcre2jit` documents as thread safe. A failed `pthread_setspecific`
frees the stack rather than leaking one per match.
The 1 MiB maximum is now recorded as measured rather than assumed: the maximum costs
nothing per match at any size, and 1 MiB already resolves a longer subject than
`proxy.config.http.request_header_max_size` lets a client send.
### The autest expectation changes
`regex_remap`'s 3 KB URL case asserted a 200 fall-through, which is what a 32 KiB stack
produced. With the rule matching as written it is a 301, so the run now expects the
redirect and its `Location`. The crash property that case was really guarding, from
#5762, moves to a unit test that asserts it directly rather than inferring it from a
status code.
### Tests
- `RegexMatchContext matches the shared context`: a quantified alternation of capture
groups over 1,000 characters, run through the shared context and through a caller
context, must return the same result. Gated on `PCRE2_INFO_JITSIZE`, because without
JIT code PCRE2 never consults the stack and the test would pass whether or not the
behaviour is present.
- `Regex reports resource exhaustion rather than crashing`: the #5762 pattern against a
256 KiB subject must return an error, not take down the thread.
### Verification
Fedora 44, gcc 16.2.1, PCRE2 10.47, dev-asan.
- `test_tsutil "[Regex]"`: 386 assertions in 20 cases, clean under AddressSanitizer with
UBSan.
- Negative control, this branch's tests against the unfixed `Regex.cc`:
`shared_rc := 2`, `own_rc := -46`. The caller context hits `JIT_STACKLIMIT` exactly
where the shared context matches.
- `regex_remap` autest: every behavioural assertion passes, including the 3 KB URL
returning `301` with its `Location` and both deliberate resource-limit errors appearing
in `diags.log`. The run is marked failed only by `traffic_server` exiting 1 on a
LeakSanitizer report, and that leak is 104 bytes in
`ConfigReloadTask::start_progress_checker`, which is #13662 and present on unmodified
master in the same run. CI's autest lane does not build with ASan.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Regex: hold the JIT stack in a pthread key and inherit the shared match context [trafficserver]
Posted by "bryancall (via GitHub) via github" <gi...@trafficserver.apache.org>.
bryancall commented on PR #13683:
URL: https://github.com/apache/trafficserver/pull/13683#issuecomment-6750673242
[approve ci urtest 3]
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Regex: hold the JIT stack in a pthread key and inherit the shared match context [trafficserver]
Posted by "bryancall (via GitHub)" <gi...@apache.org>.
bryancall commented on code in PR #13683:
URL: https://github.com/apache/trafficserver/pull/13683#discussion_r4066776157
##########
src/tsutil/unit_tests/test_Regex.cc:
##########
@@ -1050,3 +1055,163 @@ TEST_CASE("Regex end-anchor with alternation", "[libts][Regex]")
CHECK(r.exec("cdn.example.com.evil.com", matches) == RE_ERROR_NOMATCH);
CHECK(r.exec("prefix.cdn.example.com", matches) == RE_ERROR_NOMATCH);
}
+
+namespace
+{
+/** Does PCRE2 have JIT code for this pattern?
+ *
+ * The two tests below are about the JIT stack, and PCRE2 consults it only when it
+ * has JIT code to run. Without it both a blank context and the shared one take the
+ * interpreter and return the same answer, so the tests would pass whether or not
+ * the behaviour they describe is present. Ask PCRE2 rather than assume.
+ */
+bool
+pattern_has_jit(char const *pattern)
+{
+ int errnum = 0;
+ PCRE2_SIZE erroffset = 0;
+ pcre2_code *code = pcre2_compile(reinterpret_cast<PCRE2_SPTR>(pattern), PCRE2_ZERO_TERMINATED, 0, &errnum, &erroffset, nullptr);
+ if (code == nullptr) {
+ return false;
+ }
Review Comment:
Agreed, and taken: it is a `REQUIRE(code != nullptr)` now, which is the release-assert alternative you offered. A pattern that will not compile is a broken test rather than a build without a JIT, and reporting it as "no JIT" would turn a typo into a silent skip, so the comment says that too.
##########
src/tsutil/unit_tests/test_Regex.cc:
##########
@@ -1050,3 +1055,163 @@ TEST_CASE("Regex end-anchor with alternation", "[libts][Regex]")
CHECK(r.exec("cdn.example.com.evil.com", matches) == RE_ERROR_NOMATCH);
CHECK(r.exec("prefix.cdn.example.com", matches) == RE_ERROR_NOMATCH);
}
+
+namespace
+{
+/** Does PCRE2 have JIT code for this pattern?
+ *
+ * The two tests below are about the JIT stack, and PCRE2 consults it only when it
+ * has JIT code to run. Without it both a blank context and the shared one take the
+ * interpreter and return the same answer, so the tests would pass whether or not
+ * the behaviour they describe is present. Ask PCRE2 rather than assume.
+ */
+bool
+pattern_has_jit(char const *pattern)
+{
+ int errnum = 0;
+ PCRE2_SIZE erroffset = 0;
+ pcre2_code *code = pcre2_compile(reinterpret_cast<PCRE2_SPTR>(pattern), PCRE2_ZERO_TERMINATED, 0, &errnum, &erroffset, nullptr);
+ if (code == nullptr) {
+ return false;
+ }
+ pcre2_jit_compile(code, PCRE2_JIT_COMPLETE);
+ size_t jit_size = 0;
+ pcre2_pattern_info(code, PCRE2_INFO_JITSIZE, &jit_size);
+ pcre2_code_free(code);
+ return jit_size > 0;
+}
+} // namespace
+
+// A caller-supplied RegexMatchContext must behave like the shared context that
+// Regex::exec uses when none is supplied. A context built from scratch silently
+// drops everything the shared one configures, which is how regex_remap came to
+// run with PCRE2's fallback 32KiB JIT stack instead of the 1MiB one.
+TEST_CASE("RegexMatchContext matches the shared context", "[libts][Regex][RegexMatchContext]")
+{
+ // Quantified alternation of capture groups: every subject character pushes a
+ // backtracking frame, so the JIT stack size is what bounds this.
+ char const *const pattern = R"(^(?:(a)|(b))+$)";
+ if (!pattern_has_jit(pattern)) {
+ SKIP("PCRE2 has no JIT for this pattern, so the JIT stack is never consulted");
+ }
+
+ Regex re;
+ REQUIRE(re.compile(pattern));
+
+ std::string const subject(1000, 'a');
+
+ RegexMatches shared_matches;
+ RegexMatchContext match_context;
+ RegexMatches own_matches;
+
+ int const shared_rc = re.exec(subject, shared_matches);
+ int const own_rc = re.exec(subject, own_matches, 0, &match_context);
+ CAPTURE(shared_rc, own_rc);
+
+ REQUIRE(shared_rc > 0);
+ REQUIRE(own_rc == shared_rc);
+}
+
+// The guard from #5762: a pattern that backtracks once per character must fail
+// cleanly rather than run the thread out of stack. PCRE1 recursed on the machine
+// stack and a long enough subject crashed the server; PCRE2 must report an error
+// instead. If this ever crashes rather than fails, that regression is back.
Review Comment:
Removed.
##########
src/tsutil/unit_tests/test_Regex.cc:
##########
@@ -1050,3 +1055,163 @@ TEST_CASE("Regex end-anchor with alternation", "[libts][Regex]")
CHECK(r.exec("cdn.example.com.evil.com", matches) == RE_ERROR_NOMATCH);
CHECK(r.exec("prefix.cdn.example.com", matches) == RE_ERROR_NOMATCH);
}
+
+namespace
+{
+/** Does PCRE2 have JIT code for this pattern?
+ *
+ * The two tests below are about the JIT stack, and PCRE2 consults it only when it
+ * has JIT code to run. Without it both a blank context and the shared one take the
+ * interpreter and return the same answer, so the tests would pass whether or not
+ * the behaviour they describe is present. Ask PCRE2 rather than assume.
+ */
+bool
+pattern_has_jit(char const *pattern)
+{
+ int errnum = 0;
+ PCRE2_SIZE erroffset = 0;
+ pcre2_code *code = pcre2_compile(reinterpret_cast<PCRE2_SPTR>(pattern), PCRE2_ZERO_TERMINATED, 0, &errnum, &erroffset, nullptr);
+ if (code == nullptr) {
+ return false;
+ }
+ pcre2_jit_compile(code, PCRE2_JIT_COMPLETE);
+ size_t jit_size = 0;
+ pcre2_pattern_info(code, PCRE2_INFO_JITSIZE, &jit_size);
+ pcre2_code_free(code);
+ return jit_size > 0;
+}
+} // namespace
+
+// A caller-supplied RegexMatchContext must behave like the shared context that
+// Regex::exec uses when none is supplied. A context built from scratch silently
+// drops everything the shared one configures, which is how regex_remap came to
+// run with PCRE2's fallback 32KiB JIT stack instead of the 1MiB one.
+TEST_CASE("RegexMatchContext matches the shared context", "[libts][Regex][RegexMatchContext]")
+{
+ // Quantified alternation of capture groups: every subject character pushes a
+ // backtracking frame, so the JIT stack size is what bounds this.
+ char const *const pattern = R"(^(?:(a)|(b))+$)";
+ if (!pattern_has_jit(pattern)) {
+ SKIP("PCRE2 has no JIT for this pattern, so the JIT stack is never consulted");
+ }
+
+ Regex re;
+ REQUIRE(re.compile(pattern));
+
+ std::string const subject(1000, 'a');
+
+ RegexMatches shared_matches;
+ RegexMatchContext match_context;
+ RegexMatches own_matches;
+
+ int const shared_rc = re.exec(subject, shared_matches);
+ int const own_rc = re.exec(subject, own_matches, 0, &match_context);
+ CAPTURE(shared_rc, own_rc);
+
+ REQUIRE(shared_rc > 0);
+ REQUIRE(own_rc == shared_rc);
+}
+
+// The guard from #5762: a pattern that backtracks once per character must fail
+// cleanly rather than run the thread out of stack. PCRE1 recursed on the machine
+// stack and a long enough subject crashed the server; PCRE2 must report an error
+// instead. If this ever crashes rather than fails, that regression is back.
+TEST_CASE("Regex reports resource exhaustion rather than crashing", "[libts][Regex][limits]")
+{
+ // Only the JIT path has a bound to exhaust here. PCRE2's interpreter keeps its
+ // backtracking frames on the heap, so it matches this subject rather than running
+ // out of anything, and there is no resource error to assert.
+ char const *const pattern = R"(^/alpha/bravo/[?]((?!action=(newsfeed|calendar|contacts|notepad)).)*$)";
+ if (!pattern_has_jit(pattern)) {
+ SKIP("PCRE2 has no JIT for this pattern, so there is no stack bound to exhaust");
+ }
+
+ Regex re;
+ REQUIRE(re.compile(pattern));
+
+ // This pattern starts failing at roughly 43KiB of subject against a 1MiB JIT
+ // stack, measured identically on x86_64 and arm64. 256KiB keeps a six times
+ // margin for a platform whose JIT frames are larger, without allocating more
+ // than the bound needs. Do not trim this to just above 43KiB.
+ std::string subject{"/alpha/bravo/?"};
+ subject.append(256 * 1024, 'x');
+
+ RegexMatches matches;
+ int const rc = re.exec(subject, matches);
+ CAPTURE(rc);
+
+ // Reaching this line at all is the crash assertion.
+ REQUIRE(rc < 0);
+ REQUIRE(rc != RE_ERROR_NOMATCH);
+}
+
+// The header promises that exec() may be called concurrently on one instance, and nothing
+// tested that. Every thread must reach the same verdict, whether it matches through the
+// shared context or through one it built itself, and each thread must get its own JIT
+// stack from the callback rather than share one. Run this under ThreadSanitizer to get the
+// second half of the guarantee.
+TEST_CASE("Regex matches concurrently on one instance", "[libts][Regex][threads]")
+{
Review Comment:
Done in `9e73cb6`. The three inlined assertions in the thread body move to `concurrent_match_round()`, so the loop is now just the call and the failure count.
One behavioural note rather than bury it: the helper returns false on the first disagreement, so the counter tallies rounds that went wrong rather than individual assertions. The case asserts on zero either way, so the oracle is unchanged, but the denominator is different if you ever read the number.
##########
src/tsutil/unit_tests/test_Regex.cc:
##########
@@ -1050,3 +1055,163 @@ TEST_CASE("Regex end-anchor with alternation", "[libts][Regex]")
CHECK(r.exec("cdn.example.com.evil.com", matches) == RE_ERROR_NOMATCH);
CHECK(r.exec("prefix.cdn.example.com", matches) == RE_ERROR_NOMATCH);
}
+
+namespace
+{
+/** Does PCRE2 have JIT code for this pattern?
+ *
+ * The two tests below are about the JIT stack, and PCRE2 consults it only when it
+ * has JIT code to run. Without it both a blank context and the shared one take the
+ * interpreter and return the same answer, so the tests would pass whether or not
+ * the behaviour they describe is present. Ask PCRE2 rather than assume.
+ */
+bool
+pattern_has_jit(char const *pattern)
+{
+ int errnum = 0;
+ PCRE2_SIZE erroffset = 0;
+ pcre2_code *code = pcre2_compile(reinterpret_cast<PCRE2_SPTR>(pattern), PCRE2_ZERO_TERMINATED, 0, &errnum, &erroffset, nullptr);
+ if (code == nullptr) {
+ return false;
+ }
+ pcre2_jit_compile(code, PCRE2_JIT_COMPLETE);
Review Comment:
You asked for a reference, and your own caveat turns out to be the reason I cannot use it. From `pcre2jit(3)`:
> having the JIT code available does not guarantee that it will be used for any particular match. One reason for this is that there are a number of options and pattern items that are not supported by JIT (see below).
The UNSUPPORTED OPTIONS AND PATTERN ITEMS section names `\C` in a UTF mode and a callout immediately before an assertion condition in a conditional group. So the two calls answer different questions: `PCRE2_JIT_TEST_ALLOC` answers "is JIT compiled in and can it allocate executable memory", and what the gate needs is "did *this* pattern get a JIT block", because the JIT stack is only consulted for a pattern that was actually JIT compiled. Querying `PCRE2_INFO_JITSIZE` after `pcre2_jit_compile()` answers that directly.
There is a second, independent reason. `PCRE2_JIT_TEST_ALLOC` arrived in PCRE2 10.45. ATS pins no minimum version (`pkg_check_modules(PCRE2 REQUIRED IMPORTED_TARGET libpcre2-8)`) and the `rocky` CI lane is Rocky Linux 8.10, whose pcre2 predates 10.45, so it would not compile there.
Leaving this thread open rather than resolving it, since it is your call whether that answer satisfies the question.
##########
src/tsutil/Regex.cc:
##########
@@ -278,11 +343,16 @@ RegexMatchContext::operator=(RegexMatchContext const &other)
{
if (&other != this) {
auto ptr = _MatchContext::get(other._match_context);
- if (nullptr != ptr) {
- pcre2_match_context *const ctx = pcre2_match_context_copy(ptr);
- _MatchContext::set(_match_context, ctx);
- } else {
- _MatchContext::set(_match_context, nullptr);
+
+ // Take the copy before releasing what this object already holds, so a failing
+ // copy leaves it holding its old context rather than a freed one. Releasing it
+ // is what this operator used to omit, and every assignment leaked one context.
+ pcre2_match_context *const ctx = nullptr != ptr ? pcre2_match_context_copy(ptr) : nullptr;
+ pcre2_match_context *const old = _MatchContext::get(_match_context);
+
+ _MatchContext::set(_match_context, ctx);
+ if (old != nullptr) {
+ pcre2_match_context_free(old);
}
Review Comment:
Correct, and fixed in `9e73cb6`.
Worse than the report: the comment directly above claimed that taking the copy first leaves the object holding its old context on failure, and the code then assigned the null and freed the old one anyway. The ordering was already right; what was missing was declining to swap when the copy comes back null, which it now does.
The copy constructor has the same null return but no old context to lose, so it is left alone deliberately.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Regex: hold the JIT stack in a pthread key and inherit the shared match context [trafficserver]
Posted by "bryancall (via GitHub) via github" <gi...@trafficserver.apache.org>.
bryancall commented on PR #13683:
URL: https://github.com/apache/trafficserver/pull/13683#issuecomment-6622815390
@JosiahWI Thanks for making time for this. To help split it up, here is what changed since your review on 15 Sep (at c3b4f4d4):
- **Tests.** Each of your comments has an answer in its thread: `pattern_has_jit` now `REQUIRE`s a compiled pattern, the extra sentence is gone, the concurrent round is a helper, and the `PCRE2_JIT_TEST_ALLOC` thread is still open for you. The 3 KB case needs no gate, because both engines redirect it. The concurrent test's deep subjects now need the 1 MiB stack, and every round starts at a barrier.
- **`RegexMatchContext`.** The constructor assigns the JIT stack callback directly instead of copying the shared context, so it never creates the `thread_local`. Copy assignment frees the old context, and keeps it if the copy can't be allocated. The move operations are real now, since the defaulted ones freed the context twice. The copy and move paths have tests.
- **`traffic_layout info`.** Below PCRE2 10.45 it decides `TS_HAS_PCRE2_JIT` by JIT compiling a pattern, so a runtime that can't allocate executable memory reads false.
- **regex_remap test.** It moved to Uranium along with master. The checks are the same: the 3 KB query expects the 301, and the 64,000 byte query, on a JIT build only, expects the 200 and the resource error.
- **Last two commits.** They only touch comments, to fit the new comment rules, plus one test fix: the self move test now sets a match limit, so it fails if the self move guard goes away. Before, it passed either way.
If you want three sittings, I'd take test_Regex.cc first, then Regex.cc and Regex.h, then info.cc and the Uranium test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Regex: hold the JIT stack in a pthread key and inherit the shared match context [trafficserver]
Posted by "bryancall (via GitHub)" <gi...@apache.org>.
bryancall commented on PR #13683:
URL: https://github.com/apache/trafficserver/pull/13683#issuecomment-6019585402
@JosiahWI Sorry this one sat so long. The test was enforcing a bug. This PR doesn't change any documented behavior.
The rule is `^/alpha/bravo/[?]((?!action=(newsfeed|calendar|contacts|notepad)).)*$ https://redirect.com/ @status=301`. The query is `action=newsfed;...` (newsfed, not newsfeed), so the lookahead never trips. The pattern matches that URL, and the docs say a matched rule redirects. The old run 3 got a 200 only because the match never finished. The rule backtracks once per subject character. The plugin's match context carried no JIT stack, so PCRE2 fell back to its 32 KiB default. A 3 KB query used that up, the match returned an error, and the rule was skipped (#13660). You can see the test knew this: next to the 200 it required `Bad regular expression result -46/-47/-53/-63` in diags.log, so it passed only when the regex failed.
With the context now inheriting the shared 1 MB stack, the same request matches and returns the 301. Run 3 is what fails if the fix is reverted.
The #5762 crash guard is still there as run 3b. It sends a 64,000-byte query, past the roughly 44 KB that the 1 MB stack can handle, and asserts the same 200 plus resource-exhaustion error as before. It's gated on `TS_HAS_PCRE2_JIT`, because the interpreter keeps its frames on the heap and just matches. test_Regex.cc covers the same property directly by bounding the stack.
If that answers it, could you take another look? Your review from 9/15 still shows changes requested, and I believe every thread from it is addressed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Regex: hold the JIT stack in a pthread key and inherit the shared match context [trafficserver]
Posted by "bryancall (via GitHub) via github" <gi...@trafficserver.apache.org>.
bryancall commented on PR #13683:
URL: https://github.com/apache/trafficserver/pull/13683#issuecomment-6748840083
[approve ci doxygen]
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Regex: hold the JIT stack in a pthread key and inherit the shared match context [trafficserver]
Posted by "JosiahWI (via GitHub)" <gi...@apache.org>.
JosiahWI commented on code in PR #13683:
URL: https://github.com/apache/trafficserver/pull/13683#discussion_r4082820061
##########
src/tsutil/Regex.cc:
##########
@@ -79,6 +80,67 @@ my_free(void *ptr, void * /*caller*/)
free(ptr);
}
+//----------------------------------------------------------------------------
+// One match context is shared by every thread that matches through it, and PCRE2
+// requires a distinct JIT stack per thread, so the stack comes from a callback
+// invoked at match time rather than a pointer baked in when the context is built.
+//
+// The per thread stack is held in a pthread key rather than a thread_local. A
+// thread_local with a destructor registers it through __cxa_thread_atexit, which
+// takes the dynamic loader lock; doing that from a match would invert lock order
+// against a dlopen caller running a plugin's static initialization. See the same
+// hazard described at Diags::tag_activated. A pthread key registers its destructor
+// once, at key creation, and never from the matching path.
+pthread_key_t jit_stack_key;
+bool jit_stack_key_valid = false;
+pthread_once_t jit_stack_key_once = PTHREAD_ONCE_INIT;
Review Comment:
Should these perhaps have internal linkage?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Regex: hold the JIT stack in a pthread key and inherit the shared match context [trafficserver]
Posted by "bryancall (via GitHub) via github" <gi...@trafficserver.apache.org>.
bryancall commented on PR #13683:
URL: https://github.com/apache/trafficserver/pull/13683#issuecomment-6553439757
[approve ci urtest clang-tidy]
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Regex: hold the JIT stack in a pthread key and inherit the shared match context [trafficserver]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #13683:
URL: https://github.com/apache/trafficserver/pull/13683#discussion_r4052242010
##########
src/tsutil/Regex.cc:
##########
@@ -278,11 +343,16 @@ RegexMatchContext::operator=(RegexMatchContext const &other)
{
if (&other != this) {
auto ptr = _MatchContext::get(other._match_context);
- if (nullptr != ptr) {
- pcre2_match_context *const ctx = pcre2_match_context_copy(ptr);
- _MatchContext::set(_match_context, ctx);
- } else {
- _MatchContext::set(_match_context, nullptr);
+
+ // Take the copy before releasing what this object already holds, so a failing
+ // copy leaves it holding its old context rather than a freed one. Releasing it
+ // is what this operator used to omit, and every assignment leaked one context.
+ pcre2_match_context *const ctx = nullptr != ptr ? pcre2_match_context_copy(ptr) : nullptr;
+ pcre2_match_context *const old = _MatchContext::get(_match_context);
+
+ _MatchContext::set(_match_context, ctx);
+ if (old != nullptr) {
+ pcre2_match_context_free(old);
}
Review Comment:
`pcre2_match_context_copy()` can return `nullptr` when it cannot allocate memory. In that case this code still replaces the destination with `nullptr` and frees `old`, so a transient copy failure discards the existing match limit and JIT callback—the opposite of the comment's promised strong-failure behavior. Only clear/free `old` when `ptr` was null, or when `ctx` is a successful copy; otherwise leave the destination unchanged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Regex: test concurrent matching and JIT stack exhaustion [trafficserver]
Posted by "bryancall (via GitHub) via github" <gi...@trafficserver.apache.org>.
bryancall merged PR #13683:
URL: https://github.com/apache/trafficserver/pull/13683
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Regex: test concurrent matching and JIT stack exhaustion [trafficserver]
Posted by "bryancall (via GitHub) via github" <gi...@trafficserver.apache.org>.
bryancall commented on PR #13683:
URL: https://github.com/apache/trafficserver/pull/13683#issuecomment-6783038178
I've rescoped this PR. #13748 landed while it was open and replaced `RegexMatchContext` with `Regex::Options`, so each thread now owns its match contexts and JIT stack. That fixed the 32KiB JIT stack problem this PR was opened for, and the pthread key and inherited match context here aren't needed anymore. After merging master, `Regex.h` and `Regex.cc` match master exactly.
What's left is test coverage for #13748's fix: the 8-thread concurrency case through default and non-default `Options`, the exhaustion case gated on real JIT code instead of `PCRE2_CONFIG_JIT`, `TS_HAS_PCRE2_JIT` in `traffic_layout info`, and the Uranium JIT stack exhaustion check. I updated the title and description to match and dropped the "Fixes #13660" line.
#13660 looks fixed by #13748. I'll leave closing it to its owner.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Regex: hold the JIT stack in a pthread key and inherit the shared match context [trafficserver]
Posted by "bryancall (via GitHub)" <gi...@apache.org>.
bryancall commented on code in PR #13683:
URL: https://github.com/apache/trafficserver/pull/13683#discussion_r4066777398
##########
src/tsutil/unit_tests/test_Regex.cc:
##########
@@ -1147,3 +1152,260 @@ TEST_CASE("Regex copies answer the same as their original", "[libts][Regex][copy
CHECK(original.exec(ordinary, original_ordinary) == copy.exec(ordinary, copy_ordinary));
}
}
+
+namespace
+{
+/** Does PCRE2 have JIT code for this pattern?
+ *
+ * The two tests below are about the JIT stack, and PCRE2 consults it only when it
+ * has JIT code to run. Without it both a blank context and the shared one take the
+ * interpreter and return the same answer, so the tests would pass whether or not
+ * the behaviour they describe is present. Ask PCRE2 rather than assume.
+ *
+ * This asks about one pattern, deliberately: PCRE2 declines some pattern items and
+ * leaves PCRE2_INFO_JITSIZE at zero on a build whose JIT is otherwise fine, so the
+ * library-wide probes (pcre2_config, PCRE2_JIT_TEST_ALLOC) answer a different
+ * question. Call it from the test thread only; Catch2 assertions are not thread safe.
+ */
+bool
+pattern_has_jit(char const *pattern)
+{
+ int errnum = 0;
+ PCRE2_SIZE erroffset = 0;
+ pcre2_code *code = pcre2_compile(reinterpret_cast<PCRE2_SPTR>(pattern), PCRE2_ZERO_TERMINATED, 0, &errnum, &erroffset, nullptr);
+
+ // A pattern that will not compile is a broken test, not a build without a JIT.
+ // Reporting it as "no JIT" would turn a typo into a silent skip.
+ REQUIRE(code != nullptr);
+
+ pcre2_jit_compile(code, PCRE2_JIT_COMPLETE);
+ size_t jit_size = 0;
+
+ pcre2_pattern_info(code, PCRE2_INFO_JITSIZE, &jit_size);
+ pcre2_code_free(code);
+ return jit_size > 0;
+}
+} // namespace
+
+// A caller-supplied RegexMatchContext must behave like the shared context that
+// Regex::exec uses when none is supplied. A context built from scratch silently
+// drops everything the shared one configures, which is how regex_remap came to
+// run with PCRE2's fallback 32KiB JIT stack instead of the 1MiB one.
+TEST_CASE("RegexMatchContext matches the shared context", "[libts][Regex][RegexMatchContext]")
+{
+ // Quantified alternation of capture groups: every subject character pushes a
+ // backtracking frame, so the JIT stack size is what bounds this.
+ char const *const pattern = R"(^(?:(a)|(b))+$)";
+ if (!pattern_has_jit(pattern)) {
+ SKIP("PCRE2 has no JIT for this pattern, so the JIT stack is never consulted");
+ }
+
+ Regex re;
+ REQUIRE(re.compile(pattern));
+
+ // 40 bytes of JIT stack per subject character, so PCRE2's 32KiB fallback stops at
+ // 818 characters and the shared context's 1MiB stack at 26,213. 5,000 sits six
+ // times above the first and five times below the second, so a blank context and
+ // the shared one give different answers with margin either way.
+ std::string const subject(5000, 'a');
+
+ RegexMatches shared_matches;
+ RegexMatchContext match_context;
+ RegexMatches own_matches;
+
+ int const shared_rc = re.exec(subject, shared_matches);
+ int const own_rc = re.exec(subject, own_matches, 0, &match_context);
+ CAPTURE(shared_rc, own_rc);
+
+ REQUIRE(shared_rc > 0);
+ REQUIRE(own_rc == shared_rc);
+
+ // The copy paths must carry the inherited configuration too. Building a blank
+ // context in either of them is the same defect, and nothing else here notices.
+ RegexMatchContext const copied{match_context};
+ RegexMatchContext assigned;
+
+ assigned = match_context;
Review Comment:
Fixed. `operator=` releases the old context before taking the new one as of `e4cafab`, which is the leak this flagged, and `9e73cb6` adds the OOM case on top so a failing copy leaves the object untouched rather than empty.
##########
src/tsutil/Regex.cc:
##########
@@ -257,8 +310,20 @@ struct RegexMatchContext::_MatchContext {
//----------------------------------------------------------------------------
RegexMatchContext::RegexMatchContext()
{
- auto ctx = pcre2_match_context_create(nullptr);
- debug_assert_message(ctx, "Failed to allocate custom pcre2 match context");
+ // Copy the shared context rather than building a blank one. A blank context
+ // silently drops everything the shared context configures, which is how this
+ // type came to run with PCRE2's fallback 32KiB JIT stack instead of the 1MiB
+ // one. Callers override only the fields they mean to.
+ //
+ // pcre2_match_context_copy dereferences its argument rather than returning null
+ // for one, and RegexContext's constructor does not check its allocations, so a
+ // shared context that failed to allocate would crash here. Fall back to a blank
+ // context, which is what this constructor built before and which no reader of
+ // _match_context dereferences unchecked.
+ auto *shared = RegexContext::get_instance()->get_match_context();
+ auto *ctx = shared != nullptr ? pcre2_match_context_copy(shared) : pcre2_match_context_create(nullptr);
Review Comment:
This one was right and it was self inflicted. Fixed in `d5029bc`.
The old constructor called `pcre2_match_context_create(nullptr)` and touched no thread-local. Copying the shared context meant calling `RegexContext::get_instance()`, which constructs a `thread_local` whose destructor registers through `__cxa_thread_atexit` and takes the dynamic loader lock. That is the exact inversion documented above `jit_stack_key` that this change exists to avoid, and the constructor reintroduced it for any thread that builds a context without matching, including a plugin doing so during static initialization while `dlopen` already holds the lock.
Nothing the shared context carries is actually needed there. The JIT stack callback is thread independent because it resolves its stack through the pthread key, so the constructor now assigns the callback directly and gets the 1MiB stack without the thread-local. A null general context is what it used before this branch. The copy and its null-shared fallback are gone, so the constructor is shorter than the version it replaces.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Regex: test concurrent matching and JIT stack exhaustion [trafficserver]
Posted by "bryancall (via GitHub) via github" <gi...@trafficserver.apache.org>.
bryancall commented on PR #13683:
URL: https://github.com/apache/trafficserver/pull/13683#issuecomment-6787258798
Thanks for catching that. `<barrier>` first shipped in libstdc++ 11, and the CentOS 7 and Ubuntu 20.04 images build against libstdc++ 10 and 9. I replaced `std::barrier` with a small mutex and condition_variable barrier in the test, so it only needs library features those images have. I built both CI presets in those images, and `test_tsutil` passes on each.
While checking that the case still catches what it's meant to on those builds, I found that on the CentOS Release build, a match limit leaking between threads was caught in only 1 of 100 runs. Each round now matches the hit 64 times instead of once, which catches it in every run with no measurable change in runtime. That's in its own commit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Regex: hold the JIT stack in a pthread key and inherit the shared match context [trafficserver]
Posted by "JosiahWI (via GitHub)" <gi...@apache.org>.
JosiahWI commented on PR #13683:
URL: https://github.com/apache/trafficserver/pull/13683#issuecomment-6446068633
It will take me a bit to get to this PR as I will have to break it into at least three review sessions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Regex: hold the JIT stack in a pthread key and inherit the shared match context [trafficserver]
Posted by "bryancall (via GitHub) via github" <gi...@trafficserver.apache.org>.
bryancall commented on code in PR #13683:
URL: https://github.com/apache/trafficserver/pull/13683#discussion_r4463696162
##########
src/tsutil/Regex.cc:
##########
@@ -279,22 +355,58 @@ RegexMatchContext::operator=(RegexMatchContext const &other)
{
if (&other != this) {
auto ptr = _MatchContext::get(other._match_context);
- if (nullptr != ptr) {
- pcre2_match_context *const ctx = pcre2_match_context_copy(ptr);
- _MatchContext::set(_match_context, ctx);
- } else {
- _MatchContext::set(_match_context, nullptr);
+
+ // Take the copy before releasing what this object already holds, so a failing
+ // copy leaves it holding its old context rather than a freed one. Releasing it
+ // is what this operator used to omit, and every assignment leaked one context.
+ pcre2_match_context *const ctx = nullptr != ptr ? pcre2_match_context_copy(ptr) : nullptr;
+
+ // pcre2_match_context_copy() returns nullptr when it cannot allocate. Assigning it
+ // anyway would free the old context and leave this object empty, which is the very
+ // thing the ordering above exists to prevent, so leave the object untouched instead.
+ if (nullptr != ptr && nullptr == ctx) {
Review Comment:
I've also put it in the header now, where a caller will see it, in 7890206bda: if the copy can't be allocated, the object keeps its own settings.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Regex: hold the JIT stack in a pthread key and inherit the shared match context [trafficserver]
Posted by "bryancall (via GitHub)" <gi...@apache.org>.
bryancall commented on PR #13683:
URL: https://github.com/apache/trafficserver/pull/13683#issuecomment-5686751708
@JosiahWI you flagged comment verbosity on #13684 and #13685, so I applied the same pass here before you had to say it a third time: the eight lines of stack-size measurements above one call are down to the two facts needed to judge the number, and the table is in the description instead.
I deliberately left the comment on the pthread key at full length. It is the one that explains why the stack is not in a `thread_local`, which is the change someone would otherwise make back into a loader-lock inversion. Happy to cut it too if you disagree.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Regex: hold the JIT stack in a pthread key and inherit the shared match context [trafficserver]
Posted by "JosiahWI (via GitHub)" <gi...@apache.org>.
JosiahWI commented on PR #13683:
URL: https://github.com/apache/trafficserver/pull/13683#issuecomment-5688786231
@bryancall I want to clarify one thing just to make sure we are on the same page. The AuTest that was expecting a 200, and has been modified to expect a redirect - was that test enforcing a bug, or are we changing the documented behavior in this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Regex: test concurrent matching and JIT stack exhaustion [trafficserver]
Posted by "JosiahWI (via GitHub) via github" <gi...@trafficserver.apache.org>.
JosiahWI commented on PR #13683:
URL: https://github.com/apache/trafficserver/pull/13683#issuecomment-6786013333
Sadly, it seems the `<barrier>` header does not exist on our CentOS and Ubuntu CI environments. That may be a C++20 feature that we cannot use until there is wider compiler support.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Regex: test concurrent matching and JIT stack exhaustion [trafficserver]
Posted by "JosiahWI (via GitHub) via github" <gi...@trafficserver.apache.org>.
JosiahWI commented on PR #13683:
URL: https://github.com/apache/trafficserver/pull/13683#issuecomment-6784169307
[approve ci centos ubuntu]
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Regex: hold the JIT stack in a pthread key and inherit the shared match context [trafficserver]
Posted by "bryancall (via GitHub)" <gi...@apache.org>.
bryancall commented on PR #13683:
URL: https://github.com/apache/trafficserver/pull/13683#issuecomment-5724436527
[approve ci]
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Regex: hold the JIT stack in a pthread key and inherit the shared match context [trafficserver]
Posted by "JosiahWI (via GitHub)" <gi...@apache.org>.
JosiahWI commented on code in PR #13683:
URL: https://github.com/apache/trafficserver/pull/13683#discussion_r4020690035
##########
tests/gold_tests/pluginTest/regex_remap/regex_remap.test.py:
##########
@@ -123,19 +125,48 @@
tr.Processes.Default.Streams.stdout = "gold/regex_remap_simple.gold"
tr.StillRunningAfter = ts
-# 3 Test - Preserve the original crash guard from #5762. This request must
-# survive resource exhaustion without redirecting, regardless of which matching
-# resource limit is reached (JIT stack, match work, depth, or heap).
-tr = Test.AddTestRun("resource exhaustion does not crash ATS")
+# 3 Test - A 3 KB query redirects. This rule backtracks once per subject
+# character, so it used to exhaust the 32 KB stack PCRE2 falls back to when a
+# match context carries none, and the rule was skipped. The plugin's context now
+# inherits the shared 1 MB stack, so the rule matches and the redirect fires.
+#
+# This run needs no JIT gate. With JIT the 1 MB stack resolves the subject, and
+# without JIT the interpreter resolves it on the heap, so the redirect is the
+# answer either way. It only demonstrates the fix on a build that has a JIT.
Review Comment:
It's easy - see line 158. But I'm fine with no gate for now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Regex: hold the JIT stack in a pthread key and inherit the shared match context [trafficserver]
Posted by "bryancall (via GitHub)" <gi...@apache.org>.
bryancall commented on code in PR #13683:
URL: https://github.com/apache/trafficserver/pull/13683#discussion_r4017439356
##########
tests/gold_tests/pluginTest/regex_remap/regex_remap.test.py:
##########
@@ -123,12 +125,29 @@
tr.Processes.Default.Streams.stdout = "gold/regex_remap_simple.gold"
tr.StillRunningAfter = ts
-# 3 Test - Preserve the original crash guard from #5762. This request must
+# 3 Test - A 3 KB query redirects. This rule backtracks once per subject
+# character, so it used to exhaust the 32 KB stack PCRE2 falls back to when a
+# match context carries none, and the rule was skipped. The plugin's context now
+# inherits the shared 1 MB stack, so the rule matches and the redirect fires.
+tr = Test.AddTestRun("long query redirects rather than exhausting the JIT stack")
+creq = replay_txns[1]['client-request']
+tr.MakeCurlCommand(
+ curl_and_args + f"--header 'uuid: {creq['headers']['fields'][1][1]}' '{creq['url']}'" + " | grep -e '^HTTP/' -e '^Location'",
+ ts=ts)
+tr.Processes.Default.ReturnCode = 0
+tr.Processes.Default.Streams.stdout = "gold/regex_remap_redirect.gold"
+tr.StillRunningAfter = ts
+
+# 3b Test - Preserve the original crash guard from #5762. This request must
# survive resource exhaustion without redirecting, regardless of which matching
-# resource limit is reached (JIT stack, match work, depth, or heap).
+# resource limit is reached (JIT stack, match work, depth, or heap). Against the
+# shared 1 MB stack this rule needs a subject past 43 KB to exhaust it, which is
+# why the request header limit is raised above. Shortening this query silently
+# turns the run into a plain redirect test.
+crash_guard_query = 'x' * 64000
tr = Test.AddTestRun("resource exhaustion does not crash ATS")
-creq = replay_txns[1]['client-request']
-tr.MakeCurlCommand(curl_and_args + f"--header 'uuid: {creq['headers']['fields'][1][1]}' '{creq['url']}'", ts=ts)
+tr.MakeCurlCommand(
+ curl_and_args + "--header 'uuid: 180' " + f"'http://example.one/alpha/bravo/?action=newsfed;{crash_guard_query}'", ts=ts)
Review Comment:
Fixed in 6efbf77f3e, and you were right that the unit tests were the tell: they skip on `pattern_has_jit()` and this run did not, so a PCRE2 built without JIT would have failed the suite on a rule that behaves exactly as designed.
The run now gates on a new `TS_HAS_PCRE2_JIT` feature, reported by `traffic_layout info` from `pcre2_config(PCRE2_CONFIG_JIT, ...)`. `traffic_layout` already includes `pcre2.h` and gets PCRE2 through `tsutil`'s PUBLIC link, so this needed no build change. The gate reads the way the QUIC and Brotli runs read.
I checked the gate is wired rather than decorative. On a JIT build the suite executes 14 test runs and `diags.log` carries both `-46` and `-47`. With the condition pointed at a feature that does not exist, it executes 13, `diags.log` carries only `-47`, and nothing fails for a missing assertion, because the `ContainsExpression` is scoped inside the same branch.
Nothing else in the file needs the gate. The 3 KB redirect run gets the same answer from either engine, so it passes without a JIT, and I said so in a comment rather than leaving the next reader to work it out. The crash property is not lost on a build without JIT either: the match-limit run reaches `-47` through the interpreter, and `test_Regex.cc` asserts the property directly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Regex: test concurrent matching and JIT stack exhaustion [trafficserver]
Posted by "JosiahWI (via GitHub) via github" <gi...@trafficserver.apache.org>.
JosiahWI commented on code in PR #13683:
URL: https://github.com/apache/trafficserver/pull/13683#discussion_r4532804675
##########
tests/uranium_tests/pluginTest/regex_remap/test_regex_remap.py:
##########
@@ -178,6 +194,31 @@ def run_matching_checks(*, _ats: ATS, _curl: Curl, _transactions: list[dict[str,
)
assert "HTTP/1.1 200 OK" in output and "uuid: 180" in output
wait_for_file_lines(_ats.diags_log, r"ERROR: .regex_remap. Bad regular expression result -47", 1)
+ run_jit_stack_exhaustion_check(_ats=_ats, _curl=_curl)
Review Comment:
Nit: this call is at the wrong level of abstraction, I think. It should go in the same place as the `run_matching_checks` call, not inside `run_matching_checks`. Conceptually, `run_jit_stack_exhaustion_check` is testing something a bit different, and the naming suggests the two functions are at the same level of abstraction, so it's weird for one to call the other.
##########
src/traffic_layout/info.cc:
##########
@@ -182,6 +182,37 @@ produce_features(bool json)
print_feature("TS_HAS_128BIT_CAS", TS_HAS_128BIT_CAS, json);
print_feature("TS_HAS_128BIT_CAS_LIBATOMIC", TS_HAS_128BIT_CAS_LIBATOMIC, json);
print_feature("TS_HAS_TESTS", TS_HAS_TESTS, json);
+ // Whether PCRE2 can run a pattern on the just-in-time engine. This is a property of the
+ // PCRE2 that ATS is linked against, not of ATS. Only the JIT has a stack to exhaust: the
+ // interpreter keeps its backtracking frames on the heap, so a subject that fails with a
+ // JIT stack error can match on the interpreter. Tests that assert on that error need to
+ // know which engine runs.
+ //
+ // PCRE2_JIT_TEST_ALLOC (PCRE2 10.45) also confirms the JIT can allocate executable
+ // memory. PCRE2_CONFIG_JIT does not, and reports 1 on a hardened runtime where every JIT
+ // compile then fails, which is the direction that misleads a test gate.
+ {
Review Comment:
Nit: this block could be extracted to a helper function so that the `print_feature` section flows cleanly without this low-level interruption.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Regex: hold the JIT stack in a pthread key and inherit the shared match context [trafficserver]
Posted by "JosiahWI (via GitHub) via github" <gi...@trafficserver.apache.org>.
JosiahWI commented on code in PR #13683:
URL: https://github.com/apache/trafficserver/pull/13683#discussion_r4516965766
##########
src/tsutil/unit_tests/test_Regex.cc:
##########
@@ -1646,3 +1651,358 @@ TEST_CASE("Regex copies answer the same as their original", "[libts][Regex][copy
CHECK(original.exec(ordinary, original_ordinary) == copy.exec(ordinary, copy_ordinary));
}
}
+
+namespace
+{
+/** Does PCRE2 have JIT code for this pattern?
+ *
+ * The two tests below are about the JIT stack, and PCRE2 consults it only when it
+ * has JIT code to run. Without it both a blank context and the shared one take the
+ * interpreter and return the same answer, so the tests would pass whether or not
+ * the behaviour they describe is present. Ask PCRE2 rather than assume.
+ *
+ * This asks about one pattern, deliberately: PCRE2 declines some pattern items and
+ * leaves PCRE2_INFO_JITSIZE at zero on a build whose JIT is otherwise fine, so the
+ * library-wide probes (pcre2_config, PCRE2_JIT_TEST_ALLOC) answer a different
+ * question. Call it from the test thread only; Catch2 assertions are not thread safe.
+ */
Review Comment:
I heard @zwoop wants to have a discussion on where we use Doxygen comments. In the context of concern about clear guidelines, and given that we now have tooling to build Doxygen, please change this to a non-Doxygen comment. There is no module boundary here.
##########
src/tsutil/unit_tests/test_Regex.cc:
##########
@@ -1646,3 +1651,358 @@ TEST_CASE("Regex copies answer the same as their original", "[libts][Regex][copy
CHECK(original.exec(ordinary, original_ordinary) == copy.exec(ordinary, copy_ordinary));
}
}
+
+namespace
+{
+/** Does PCRE2 have JIT code for this pattern?
+ *
+ * The two tests below are about the JIT stack, and PCRE2 consults it only when it
+ * has JIT code to run. Without it both a blank context and the shared one take the
+ * interpreter and return the same answer, so the tests would pass whether or not
+ * the behaviour they describe is present. Ask PCRE2 rather than assume.
+ *
+ * This asks about one pattern, deliberately: PCRE2 declines some pattern items and
+ * leaves PCRE2_INFO_JITSIZE at zero on a build whose JIT is otherwise fine, so the
+ * library-wide probes (pcre2_config, PCRE2_JIT_TEST_ALLOC) answer a different
+ * question. Call it from the test thread only; Catch2 assertions are not thread safe.
+ */
+bool
+pattern_has_jit(char const *pattern)
+{
+ int errnum = 0;
+ PCRE2_SIZE erroffset = 0;
+ pcre2_code *code = pcre2_compile(reinterpret_cast<PCRE2_SPTR>(pattern), PCRE2_ZERO_TERMINATED, 0, &errnum, &erroffset, nullptr);
+
+ // A pattern that will not compile is a broken test, not a build without a JIT.
+ // Reporting it as "no JIT" would turn a typo into a silent skip.
Review Comment:
Nit: This comment is confusing in my opinion. The "not a build without a JIT" information makes it sound like `pcre2_compile` did some kind of JIT testing, which it did not. Mentioning JIT here is irrelevant and distracting information.
##########
src/tsutil/unit_tests/test_Regex.cc:
##########
@@ -1646,3 +1651,358 @@ TEST_CASE("Regex copies answer the same as their original", "[libts][Regex][copy
CHECK(original.exec(ordinary, original_ordinary) == copy.exec(ordinary, copy_ordinary));
}
}
+
+namespace
+{
+/** Does PCRE2 have JIT code for this pattern?
+ *
+ * The two tests below are about the JIT stack, and PCRE2 consults it only when it
+ * has JIT code to run. Without it both a blank context and the shared one take the
+ * interpreter and return the same answer, so the tests would pass whether or not
+ * the behaviour they describe is present. Ask PCRE2 rather than assume.
+ *
+ * This asks about one pattern, deliberately: PCRE2 declines some pattern items and
+ * leaves PCRE2_INFO_JITSIZE at zero on a build whose JIT is otherwise fine, so the
+ * library-wide probes (pcre2_config, PCRE2_JIT_TEST_ALLOC) answer a different
+ * question. Call it from the test thread only; Catch2 assertions are not thread safe.
+ */
+bool
+pattern_has_jit(char const *pattern)
+{
+ int errnum = 0;
+ PCRE2_SIZE erroffset = 0;
+ pcre2_code *code = pcre2_compile(reinterpret_cast<PCRE2_SPTR>(pattern), PCRE2_ZERO_TERMINATED, 0, &errnum, &erroffset, nullptr);
+
+ // A pattern that will not compile is a broken test, not a build without a JIT.
+ // Reporting it as "no JIT" would turn a typo into a silent skip.
+ REQUIRE(code != nullptr);
+
+ pcre2_jit_compile(code, PCRE2_JIT_COMPLETE);
+ size_t jit_size = 0;
+
+ pcre2_pattern_info(code, PCRE2_INFO_JITSIZE, &jit_size);
Review Comment:
Is it not enough to check the return value of `pcre2_jit_compile`, which returns an appropriate error code if the pattern cannot compile?
##########
src/tsutil/unit_tests/test_Regex.cc:
##########
@@ -1646,3 +1651,358 @@ TEST_CASE("Regex copies answer the same as their original", "[libts][Regex][copy
CHECK(original.exec(ordinary, original_ordinary) == copy.exec(ordinary, copy_ordinary));
}
}
+
+namespace
+{
+/** Does PCRE2 have JIT code for this pattern?
+ *
+ * The two tests below are about the JIT stack, and PCRE2 consults it only when it
+ * has JIT code to run. Without it both a blank context and the shared one take the
+ * interpreter and return the same answer, so the tests would pass whether or not
+ * the behaviour they describe is present. Ask PCRE2 rather than assume.
+ *
+ * This asks about one pattern, deliberately: PCRE2 declines some pattern items and
+ * leaves PCRE2_INFO_JITSIZE at zero on a build whose JIT is otherwise fine, so the
+ * library-wide probes (pcre2_config, PCRE2_JIT_TEST_ALLOC) answer a different
+ * question. Call it from the test thread only; Catch2 assertions are not thread safe.
+ */
+bool
+pattern_has_jit(char const *pattern)
+{
+ int errnum = 0;
+ PCRE2_SIZE erroffset = 0;
+ pcre2_code *code = pcre2_compile(reinterpret_cast<PCRE2_SPTR>(pattern), PCRE2_ZERO_TERMINATED, 0, &errnum, &erroffset, nullptr);
+
+ // A pattern that will not compile is a broken test, not a build without a JIT.
+ // Reporting it as "no JIT" would turn a typo into a silent skip.
+ REQUIRE(code != nullptr);
+
+ pcre2_jit_compile(code, PCRE2_JIT_COMPLETE);
+ size_t jit_size = 0;
+
+ pcre2_pattern_info(code, PCRE2_INFO_JITSIZE, &jit_size);
+ pcre2_code_free(code);
+ return jit_size > 0;
+}
+} // namespace
+
+// A caller-supplied RegexMatchContext must behave like the shared context that
+// Regex::exec uses when none is supplied. A context built from scratch drops
+// everything the shared one configures, the JIT stack callback included, and a
+// match through it then runs on PCRE2's fallback 32KiB JIT stack instead of the
+// 1MiB one.
+TEST_CASE("RegexMatchContext matches the shared context", "[libts][Regex][RegexMatchContext]")
+{
+ // Quantified alternation of capture groups: every subject character pushes a
+ // backtracking frame, so the JIT stack size is what bounds this.
+ char const *const pattern = R"(^(?:(a)|(b))+$)";
+ if (!pattern_has_jit(pattern)) {
+ SKIP("PCRE2 has no JIT for this pattern, so the JIT stack is never consulted");
+ }
+
+ Regex re;
+ REQUIRE(re.compile(pattern));
+
+ // 40 bytes of JIT stack per subject character, so PCRE2's 32KiB fallback stops at
+ // 818 characters and the shared context's 1MiB stack at 26,213. 5,000 sits six
+ // times above the first and five times below the second, so a blank context and
+ // the shared one give different answers with margin either way.
+ std::string const subject(5000, 'a');
+
+ RegexMatches shared_matches;
+ RegexMatchContext match_context;
+ RegexMatches own_matches;
+
+ int const shared_rc = re.exec(subject, shared_matches);
+ int const own_rc = re.exec(subject, own_matches, 0, &match_context);
+ CAPTURE(shared_rc, own_rc);
+
+ REQUIRE(shared_rc > 0);
+ REQUIRE(own_rc == shared_rc);
+
+ // The copy paths must carry the inherited configuration too. Building a blank
+ // context in either of them is the same defect, and nothing else here notices.
+ RegexMatchContext const copied{match_context};
+ RegexMatchContext assigned;
+
+ assigned = match_context;
+
+ RegexMatches copied_matches;
+ RegexMatches assigned_matches;
+
+ REQUIRE(re.exec(subject, copied_matches, 0, &copied) == shared_rc);
+ REQUIRE(re.exec(subject, assigned_matches, 0, &assigned) == shared_rc);
+}
+
+// Ownership only, and deliberately not inside the case above.
+//
+// That case SKIPs when pattern_has_jit() is false, which is right for what it tests and
+// wrong for this: move construction and move assignment have nothing to do with the JIT,
+// and a double free matters as much on a PCRE2 without JIT, or on a hardened runtime
+// where JIT allocation fails, as anywhere else. Inside that case this coverage would
+// skip on exactly those builds and still report green.
+TEST_CASE("RegexMatchContext move operations transfer ownership", "[libts][Regex][RegexMatchContext]")
+{
+ Regex re;
+
+ REQUIRE(re.compile("^(a+)(b+)$"));
+
+ std::string_view const subject{"aaabbb"};
+
+ auto matches_through = [&](RegexMatchContext const *ctx) {
+ RegexMatches m;
+ return re.exec(subject, m, 0, ctx);
+ };
+
+ // Baseline through a plain context, so a later mismatch means the move broke something
+ // rather than the pattern never having matched.
+ RegexMatchContext plain;
+ int const expected = matches_through(&plain);
+
+ REQUIRE(expected > 0);
+
+ SECTION("move construction")
+ {
+ RegexMatchContext donor;
+ RegexMatchContext const moved{std::move(donor)};
+
+ REQUIRE(matches_through(&moved) == expected);
+ // A double free aborts when donor and moved are both destroyed, so leaving this
+ // section is itself the second assertion.
+ }
+
+ SECTION("move assignment onto a destination that already holds a context")
+ {
+ // On an ordinary build the assertions below do not detect a dropped
+ // pcre2_match_context_free(old): removing that call leaves this section passing,
+ // because a leak has no observer here. The oracle for the leak is LeakSanitizer,
+ // which AddressSanitizer turns on by default on Linux but not on macOS.
+ //
+ // The section still earns its place: it is what gives LSan a move assignment onto a
+ // live context to observe, which no other case in this file produces. Without it
+ // the leak is unreachable and therefore undetectable anywhere.
+ RegexMatchContext destination;
+
+ REQUIRE(matches_through(&destination) == expected);
+
+ RegexMatchContext donor;
+
+ destination = std::move(donor);
+
+ REQUIRE(matches_through(&destination) == expected);
+ }
+
+ SECTION("self move assignment")
+ {
+ RegexMatchContext ctx;
+
+ // A match limit of one fails this match, so the result after the move shows whether
+ // ctx still holds its own context. Without the this != &that guard the operator frees
+ // the context and then nulls itself, and the match runs on PCRE2's defaults instead.
+ ctx.set_match_limit(1);
+ REQUIRE(matches_through(&ctx) == PCRE2_ERROR_MATCHLIMIT);
+
+ auto &alias = ctx;
+
+ ctx = std::move(alias);
+
+ REQUIRE(matches_through(&ctx) == PCRE2_ERROR_MATCHLIMIT);
+ }
+}
+
+// A pattern that backtracks once per character must fail cleanly rather than run
+// the thread out of stack. A regex engine that recurses on the machine stack crashes
+// the server on a long enough subject; PCRE2 must report an error instead.
+TEST_CASE("Regex reports resource exhaustion rather than crashing", "[libts][Regex][limits]")
+{
+ // Only the JIT path has a bound to exhaust here. PCRE2's interpreter keeps its
+ // backtracking frames on the heap, so it matches this subject rather than running
+ // out of anything, and there is no resource error to assert.
+ char const *const pattern = R"(^/alpha/bravo/[?]((?!action=(newsfeed|calendar|contacts|notepad)).)*$)";
+ if (!pattern_has_jit(pattern)) {
+ SKIP("PCRE2 has no JIT for this pattern, so there is no stack bound to exhaust");
+ }
+
+ Regex re;
+ REQUIRE(re.compile(pattern));
+
+ // This pattern starts failing at roughly 43KiB of subject against a 1MiB JIT
+ // stack, measured identically on x86_64 and arm64. Smaller JIT frames get more
+ // subject out of the same stack and so push that threshold up; 256KiB keeps a
+ // six times margin against that. Do not trim this to just above 43KiB.
+ std::string subject{"/alpha/bravo/?"};
+ subject.append(256 * 1024, 'x');
+
+ RegexMatches matches;
+ int const rc = re.exec(subject, matches);
+ CAPTURE(rc);
+
+ // Reaching this line at all is the crash assertion.
+ REQUIRE(rc < 0);
+ REQUIRE(rc != RE_ERROR_NOMATCH);
+
+ // The paired positive control, and the only part of this case that can tell a
+ // 1MiB stack from PCRE2's 32KiB fallback: 256KiB overruns both bounds, so every
+ // assertion above also holds with the JIT stack callback deleted. 10,000 is seven
+ // times over the fallback's bound and four times under the 1MiB stack's. Neither
+ // bound pins 1MiB itself; the suite only requires a maximum in roughly
+ // [240KB, 6MB].
+ std::string smaller{"/alpha/bravo/?"};
+ smaller.append(10000, 'x');
+
+ RegexMatches small_matches;
+ int const small_rc = re.exec(smaller, small_matches);
+ CAPTURE(small_rc);
+
+ REQUIRE(small_rc > 0);
+}
+
+namespace
+{
+/** A reusable barrier, so every round starts with all threads alive and inside it. The
Review Comment:
Please change this to a non-doxygen comment, along with the other comments in this namespace.
##########
src/tsutil/unit_tests/test_Regex.cc:
##########
@@ -1646,3 +1651,358 @@ TEST_CASE("Regex copies answer the same as their original", "[libts][Regex][copy
CHECK(original.exec(ordinary, original_ordinary) == copy.exec(ordinary, copy_ordinary));
}
}
+
+namespace
+{
+/** Does PCRE2 have JIT code for this pattern?
+ *
+ * The two tests below are about the JIT stack, and PCRE2 consults it only when it
+ * has JIT code to run. Without it both a blank context and the shared one take the
+ * interpreter and return the same answer, so the tests would pass whether or not
+ * the behaviour they describe is present. Ask PCRE2 rather than assume.
+ *
+ * This asks about one pattern, deliberately: PCRE2 declines some pattern items and
+ * leaves PCRE2_INFO_JITSIZE at zero on a build whose JIT is otherwise fine, so the
+ * library-wide probes (pcre2_config, PCRE2_JIT_TEST_ALLOC) answer a different
+ * question. Call it from the test thread only; Catch2 assertions are not thread safe.
+ */
+bool
+pattern_has_jit(char const *pattern)
+{
+ int errnum = 0;
+ PCRE2_SIZE erroffset = 0;
+ pcre2_code *code = pcre2_compile(reinterpret_cast<PCRE2_SPTR>(pattern), PCRE2_ZERO_TERMINATED, 0, &errnum, &erroffset, nullptr);
+
+ // A pattern that will not compile is a broken test, not a build without a JIT.
+ // Reporting it as "no JIT" would turn a typo into a silent skip.
+ REQUIRE(code != nullptr);
+
+ pcre2_jit_compile(code, PCRE2_JIT_COMPLETE);
+ size_t jit_size = 0;
+
+ pcre2_pattern_info(code, PCRE2_INFO_JITSIZE, &jit_size);
+ pcre2_code_free(code);
+ return jit_size > 0;
+}
+} // namespace
+
+// A caller-supplied RegexMatchContext must behave like the shared context that
+// Regex::exec uses when none is supplied. A context built from scratch drops
+// everything the shared one configures, the JIT stack callback included, and a
+// match through it then runs on PCRE2's fallback 32KiB JIT stack instead of the
+// 1MiB one.
+TEST_CASE("RegexMatchContext matches the shared context", "[libts][Regex][RegexMatchContext]")
+{
+ // Quantified alternation of capture groups: every subject character pushes a
+ // backtracking frame, so the JIT stack size is what bounds this.
+ char const *const pattern = R"(^(?:(a)|(b))+$)";
+ if (!pattern_has_jit(pattern)) {
+ SKIP("PCRE2 has no JIT for this pattern, so the JIT stack is never consulted");
+ }
+
+ Regex re;
+ REQUIRE(re.compile(pattern));
+
+ // 40 bytes of JIT stack per subject character, so PCRE2's 32KiB fallback stops at
+ // 818 characters and the shared context's 1MiB stack at 26,213. 5,000 sits six
+ // times above the first and five times below the second, so a blank context and
+ // the shared one give different answers with margin either way.
+ std::string const subject(5000, 'a');
+
+ RegexMatches shared_matches;
+ RegexMatchContext match_context;
+ RegexMatches own_matches;
+
+ int const shared_rc = re.exec(subject, shared_matches);
+ int const own_rc = re.exec(subject, own_matches, 0, &match_context);
+ CAPTURE(shared_rc, own_rc);
+
+ REQUIRE(shared_rc > 0);
+ REQUIRE(own_rc == shared_rc);
+
+ // The copy paths must carry the inherited configuration too. Building a blank
+ // context in either of them is the same defect, and nothing else here notices.
+ RegexMatchContext const copied{match_context};
+ RegexMatchContext assigned;
+
+ assigned = match_context;
+
+ RegexMatches copied_matches;
+ RegexMatches assigned_matches;
+
+ REQUIRE(re.exec(subject, copied_matches, 0, &copied) == shared_rc);
+ REQUIRE(re.exec(subject, assigned_matches, 0, &assigned) == shared_rc);
+}
+
+// Ownership only, and deliberately not inside the case above.
+//
+// That case SKIPs when pattern_has_jit() is false, which is right for what it tests and
+// wrong for this: move construction and move assignment have nothing to do with the JIT,
+// and a double free matters as much on a PCRE2 without JIT, or on a hardened runtime
+// where JIT allocation fails, as anywhere else. Inside that case this coverage would
+// skip on exactly those builds and still report green.
+TEST_CASE("RegexMatchContext move operations transfer ownership", "[libts][Regex][RegexMatchContext]")
+{
+ Regex re;
+
+ REQUIRE(re.compile("^(a+)(b+)$"));
+
+ std::string_view const subject{"aaabbb"};
+
+ auto matches_through = [&](RegexMatchContext const *ctx) {
+ RegexMatches m;
+ return re.exec(subject, m, 0, ctx);
+ };
+
+ // Baseline through a plain context, so a later mismatch means the move broke something
+ // rather than the pattern never having matched.
+ RegexMatchContext plain;
+ int const expected = matches_through(&plain);
+
+ REQUIRE(expected > 0);
+
+ SECTION("move construction")
+ {
+ RegexMatchContext donor;
+ RegexMatchContext const moved{std::move(donor)};
+
+ REQUIRE(matches_through(&moved) == expected);
+ // A double free aborts when donor and moved are both destroyed, so leaving this
+ // section is itself the second assertion.
+ }
+
+ SECTION("move assignment onto a destination that already holds a context")
+ {
+ // On an ordinary build the assertions below do not detect a dropped
+ // pcre2_match_context_free(old): removing that call leaves this section passing,
+ // because a leak has no observer here. The oracle for the leak is LeakSanitizer,
+ // which AddressSanitizer turns on by default on Linux but not on macOS.
+ //
+ // The section still earns its place: it is what gives LSan a move assignment onto a
+ // live context to observe, which no other case in this file produces. Without it
+ // the leak is unreachable and therefore undetectable anywhere.
+ RegexMatchContext destination;
+
+ REQUIRE(matches_through(&destination) == expected);
+
+ RegexMatchContext donor;
+
+ destination = std::move(donor);
+
+ REQUIRE(matches_through(&destination) == expected);
+ }
+
+ SECTION("self move assignment")
+ {
+ RegexMatchContext ctx;
+
+ // A match limit of one fails this match, so the result after the move shows whether
+ // ctx still holds its own context. Without the this != &that guard the operator frees
+ // the context and then nulls itself, and the match runs on PCRE2's defaults instead.
+ ctx.set_match_limit(1);
+ REQUIRE(matches_through(&ctx) == PCRE2_ERROR_MATCHLIMIT);
+
+ auto &alias = ctx;
+
+ ctx = std::move(alias);
+
+ REQUIRE(matches_through(&ctx) == PCRE2_ERROR_MATCHLIMIT);
+ }
+}
+
+// A pattern that backtracks once per character must fail cleanly rather than run
+// the thread out of stack. A regex engine that recurses on the machine stack crashes
+// the server on a long enough subject; PCRE2 must report an error instead.
+TEST_CASE("Regex reports resource exhaustion rather than crashing", "[libts][Regex][limits]")
+{
+ // Only the JIT path has a bound to exhaust here. PCRE2's interpreter keeps its
+ // backtracking frames on the heap, so it matches this subject rather than running
+ // out of anything, and there is no resource error to assert.
+ char const *const pattern = R"(^/alpha/bravo/[?]((?!action=(newsfeed|calendar|contacts|notepad)).)*$)";
+ if (!pattern_has_jit(pattern)) {
+ SKIP("PCRE2 has no JIT for this pattern, so there is no stack bound to exhaust");
+ }
+
+ Regex re;
+ REQUIRE(re.compile(pattern));
+
+ // This pattern starts failing at roughly 43KiB of subject against a 1MiB JIT
+ // stack, measured identically on x86_64 and arm64. Smaller JIT frames get more
+ // subject out of the same stack and so push that threshold up; 256KiB keeps a
+ // six times margin against that. Do not trim this to just above 43KiB.
+ std::string subject{"/alpha/bravo/?"};
+ subject.append(256 * 1024, 'x');
+
+ RegexMatches matches;
+ int const rc = re.exec(subject, matches);
+ CAPTURE(rc);
+
+ // Reaching this line at all is the crash assertion.
+ REQUIRE(rc < 0);
+ REQUIRE(rc != RE_ERROR_NOMATCH);
+
+ // The paired positive control, and the only part of this case that can tell a
+ // 1MiB stack from PCRE2's 32KiB fallback: 256KiB overruns both bounds, so every
+ // assertion above also holds with the JIT stack callback deleted. 10,000 is seven
+ // times over the fallback's bound and four times under the 1MiB stack's. Neither
+ // bound pins 1MiB itself; the suite only requires a maximum in roughly
+ // [240KB, 6MB].
+ std::string smaller{"/alpha/bravo/?"};
+ smaller.append(10000, 'x');
+
+ RegexMatches small_matches;
+ int const small_rc = re.exec(smaller, small_matches);
+ CAPTURE(small_rc);
+
+ REQUIRE(small_rc > 0);
+}
+
+namespace
+{
+/** A reusable barrier, so every round starts with all threads alive and inside it. The
+ * matches of one round then start together on every thread. std::barrier would say this
+ * directly; it is hand-rolled here to match notstd::barrier in benchmark_LogObject.cc.
+ *
+ * Each completed round advances the generation, which is what releases the waiters, so
+ * the barrier can be crossed any number of times. Exactly `expected` threads must call
+ * arrive_and_wait() every round or the rest block forever, and the barrier must outlive
+ * all of them, since destroying a condition_variable with waiters on it is undefined.
+ */
+class RoundBarrier
Review Comment:
I'm curious whether you considered implementing the barrier with a counting semaphore as an alternative and if so why you preferred the mutex.
##########
src/tsutil/unit_tests/test_Regex.cc:
##########
@@ -1646,3 +1651,358 @@ TEST_CASE("Regex copies answer the same as their original", "[libts][Regex][copy
CHECK(original.exec(ordinary, original_ordinary) == copy.exec(ordinary, copy_ordinary));
}
}
+
+namespace
+{
+/** Does PCRE2 have JIT code for this pattern?
+ *
+ * The two tests below are about the JIT stack, and PCRE2 consults it only when it
+ * has JIT code to run. Without it both a blank context and the shared one take the
+ * interpreter and return the same answer, so the tests would pass whether or not
+ * the behaviour they describe is present. Ask PCRE2 rather than assume.
+ *
+ * This asks about one pattern, deliberately: PCRE2 declines some pattern items and
+ * leaves PCRE2_INFO_JITSIZE at zero on a build whose JIT is otherwise fine, so the
+ * library-wide probes (pcre2_config, PCRE2_JIT_TEST_ALLOC) answer a different
+ * question. Call it from the test thread only; Catch2 assertions are not thread safe.
+ */
+bool
+pattern_has_jit(char const *pattern)
+{
+ int errnum = 0;
+ PCRE2_SIZE erroffset = 0;
+ pcre2_code *code = pcre2_compile(reinterpret_cast<PCRE2_SPTR>(pattern), PCRE2_ZERO_TERMINATED, 0, &errnum, &erroffset, nullptr);
+
+ // A pattern that will not compile is a broken test, not a build without a JIT.
+ // Reporting it as "no JIT" would turn a typo into a silent skip.
+ REQUIRE(code != nullptr);
+
+ pcre2_jit_compile(code, PCRE2_JIT_COMPLETE);
+ size_t jit_size = 0;
+
+ pcre2_pattern_info(code, PCRE2_INFO_JITSIZE, &jit_size);
+ pcre2_code_free(code);
+ return jit_size > 0;
+}
+} // namespace
+
+// A caller-supplied RegexMatchContext must behave like the shared context that
+// Regex::exec uses when none is supplied. A context built from scratch drops
+// everything the shared one configures, the JIT stack callback included, and a
+// match through it then runs on PCRE2's fallback 32KiB JIT stack instead of the
+// 1MiB one.
+TEST_CASE("RegexMatchContext matches the shared context", "[libts][Regex][RegexMatchContext]")
+{
+ // Quantified alternation of capture groups: every subject character pushes a
+ // backtracking frame, so the JIT stack size is what bounds this.
+ char const *const pattern = R"(^(?:(a)|(b))+$)";
+ if (!pattern_has_jit(pattern)) {
+ SKIP("PCRE2 has no JIT for this pattern, so the JIT stack is never consulted");
+ }
+
+ Regex re;
+ REQUIRE(re.compile(pattern));
+
+ // 40 bytes of JIT stack per subject character, so PCRE2's 32KiB fallback stops at
+ // 818 characters and the shared context's 1MiB stack at 26,213. 5,000 sits six
+ // times above the first and five times below the second, so a blank context and
+ // the shared one give different answers with margin either way.
+ std::string const subject(5000, 'a');
+
+ RegexMatches shared_matches;
+ RegexMatchContext match_context;
+ RegexMatches own_matches;
+
+ int const shared_rc = re.exec(subject, shared_matches);
+ int const own_rc = re.exec(subject, own_matches, 0, &match_context);
+ CAPTURE(shared_rc, own_rc);
+
+ REQUIRE(shared_rc > 0);
+ REQUIRE(own_rc == shared_rc);
+
+ // The copy paths must carry the inherited configuration too. Building a blank
+ // context in either of them is the same defect, and nothing else here notices.
+ RegexMatchContext const copied{match_context};
+ RegexMatchContext assigned;
+
+ assigned = match_context;
+
+ RegexMatches copied_matches;
+ RegexMatches assigned_matches;
+
+ REQUIRE(re.exec(subject, copied_matches, 0, &copied) == shared_rc);
+ REQUIRE(re.exec(subject, assigned_matches, 0, &assigned) == shared_rc);
+}
+
+// Ownership only, and deliberately not inside the case above.
+//
+// That case SKIPs when pattern_has_jit() is false, which is right for what it tests and
+// wrong for this: move construction and move assignment have nothing to do with the JIT,
+// and a double free matters as much on a PCRE2 without JIT, or on a hardened runtime
+// where JIT allocation fails, as anywhere else. Inside that case this coverage would
+// skip on exactly those builds and still report green.
+TEST_CASE("RegexMatchContext move operations transfer ownership", "[libts][Regex][RegexMatchContext]")
+{
+ Regex re;
+
+ REQUIRE(re.compile("^(a+)(b+)$"));
+
+ std::string_view const subject{"aaabbb"};
+
+ auto matches_through = [&](RegexMatchContext const *ctx) {
+ RegexMatches m;
+ return re.exec(subject, m, 0, ctx);
+ };
+
+ // Baseline through a plain context, so a later mismatch means the move broke something
+ // rather than the pattern never having matched.
+ RegexMatchContext plain;
+ int const expected = matches_through(&plain);
+
+ REQUIRE(expected > 0);
+
+ SECTION("move construction")
+ {
+ RegexMatchContext donor;
+ RegexMatchContext const moved{std::move(donor)};
+
+ REQUIRE(matches_through(&moved) == expected);
+ // A double free aborts when donor and moved are both destroyed, so leaving this
+ // section is itself the second assertion.
+ }
+
+ SECTION("move assignment onto a destination that already holds a context")
+ {
+ // On an ordinary build the assertions below do not detect a dropped
+ // pcre2_match_context_free(old): removing that call leaves this section passing,
+ // because a leak has no observer here. The oracle for the leak is LeakSanitizer,
+ // which AddressSanitizer turns on by default on Linux but not on macOS.
+ //
+ // The section still earns its place: it is what gives LSan a move assignment onto a
+ // live context to observe, which no other case in this file produces. Without it
+ // the leak is unreachable and therefore undetectable anywhere.
+ RegexMatchContext destination;
+
+ REQUIRE(matches_through(&destination) == expected);
+
+ RegexMatchContext donor;
+
+ destination = std::move(donor);
+
+ REQUIRE(matches_through(&destination) == expected);
+ }
+
+ SECTION("self move assignment")
+ {
+ RegexMatchContext ctx;
+
+ // A match limit of one fails this match, so the result after the move shows whether
+ // ctx still holds its own context. Without the this != &that guard the operator frees
+ // the context and then nulls itself, and the match runs on PCRE2's defaults instead.
+ ctx.set_match_limit(1);
+ REQUIRE(matches_through(&ctx) == PCRE2_ERROR_MATCHLIMIT);
+
+ auto &alias = ctx;
+
+ ctx = std::move(alias);
+
+ REQUIRE(matches_through(&ctx) == PCRE2_ERROR_MATCHLIMIT);
+ }
+}
+
+// A pattern that backtracks once per character must fail cleanly rather than run
+// the thread out of stack. A regex engine that recurses on the machine stack crashes
+// the server on a long enough subject; PCRE2 must report an error instead.
+TEST_CASE("Regex reports resource exhaustion rather than crashing", "[libts][Regex][limits]")
+{
+ // Only the JIT path has a bound to exhaust here. PCRE2's interpreter keeps its
+ // backtracking frames on the heap, so it matches this subject rather than running
+ // out of anything, and there is no resource error to assert.
+ char const *const pattern = R"(^/alpha/bravo/[?]((?!action=(newsfeed|calendar|contacts|notepad)).)*$)";
+ if (!pattern_has_jit(pattern)) {
+ SKIP("PCRE2 has no JIT for this pattern, so there is no stack bound to exhaust");
+ }
+
+ Regex re;
+ REQUIRE(re.compile(pattern));
+
+ // This pattern starts failing at roughly 43KiB of subject against a 1MiB JIT
+ // stack, measured identically on x86_64 and arm64. Smaller JIT frames get more
+ // subject out of the same stack and so push that threshold up; 256KiB keeps a
+ // six times margin against that. Do not trim this to just above 43KiB.
+ std::string subject{"/alpha/bravo/?"};
+ subject.append(256 * 1024, 'x');
+
+ RegexMatches matches;
+ int const rc = re.exec(subject, matches);
+ CAPTURE(rc);
+
+ // Reaching this line at all is the crash assertion.
+ REQUIRE(rc < 0);
+ REQUIRE(rc != RE_ERROR_NOMATCH);
+
+ // The paired positive control, and the only part of this case that can tell a
+ // 1MiB stack from PCRE2's 32KiB fallback: 256KiB overruns both bounds, so every
+ // assertion above also holds with the JIT stack callback deleted. 10,000 is seven
+ // times over the fallback's bound and four times under the 1MiB stack's. Neither
+ // bound pins 1MiB itself; the suite only requires a maximum in roughly
+ // [240KB, 6MB].
+ std::string smaller{"/alpha/bravo/?"};
+ smaller.append(10000, 'x');
+
+ RegexMatches small_matches;
+ int const small_rc = re.exec(smaller, small_matches);
+ CAPTURE(small_rc);
+
+ REQUIRE(small_rc > 0);
+}
+
+namespace
+{
+/** A reusable barrier, so every round starts with all threads alive and inside it. The
+ * matches of one round then start together on every thread. std::barrier would say this
+ * directly; it is hand-rolled here to match notstd::barrier in benchmark_LogObject.cc.
Review Comment:
Interesting justification. Can that barrier be re-used? Why didn't the log object benchmark use the standard barrier?
##########
src/tsutil/unit_tests/test_Regex.cc:
##########
@@ -1646,3 +1651,358 @@ TEST_CASE("Regex copies answer the same as their original", "[libts][Regex][copy
CHECK(original.exec(ordinary, original_ordinary) == copy.exec(ordinary, copy_ordinary));
}
}
+
+namespace
+{
+/** Does PCRE2 have JIT code for this pattern?
+ *
+ * The two tests below are about the JIT stack, and PCRE2 consults it only when it
+ * has JIT code to run. Without it both a blank context and the shared one take the
+ * interpreter and return the same answer, so the tests would pass whether or not
+ * the behaviour they describe is present. Ask PCRE2 rather than assume.
+ *
+ * This asks about one pattern, deliberately: PCRE2 declines some pattern items and
+ * leaves PCRE2_INFO_JITSIZE at zero on a build whose JIT is otherwise fine, so the
+ * library-wide probes (pcre2_config, PCRE2_JIT_TEST_ALLOC) answer a different
+ * question. Call it from the test thread only; Catch2 assertions are not thread safe.
+ */
+bool
+pattern_has_jit(char const *pattern)
+{
+ int errnum = 0;
+ PCRE2_SIZE erroffset = 0;
+ pcre2_code *code = pcre2_compile(reinterpret_cast<PCRE2_SPTR>(pattern), PCRE2_ZERO_TERMINATED, 0, &errnum, &erroffset, nullptr);
+
+ // A pattern that will not compile is a broken test, not a build without a JIT.
+ // Reporting it as "no JIT" would turn a typo into a silent skip.
+ REQUIRE(code != nullptr);
+
+ pcre2_jit_compile(code, PCRE2_JIT_COMPLETE);
+ size_t jit_size = 0;
+
+ pcre2_pattern_info(code, PCRE2_INFO_JITSIZE, &jit_size);
+ pcre2_code_free(code);
+ return jit_size > 0;
+}
+} // namespace
+
+// A caller-supplied RegexMatchContext must behave like the shared context that
+// Regex::exec uses when none is supplied. A context built from scratch drops
+// everything the shared one configures, the JIT stack callback included, and a
+// match through it then runs on PCRE2's fallback 32KiB JIT stack instead of the
+// 1MiB one.
+TEST_CASE("RegexMatchContext matches the shared context", "[libts][Regex][RegexMatchContext]")
+{
+ // Quantified alternation of capture groups: every subject character pushes a
+ // backtracking frame, so the JIT stack size is what bounds this.
+ char const *const pattern = R"(^(?:(a)|(b))+$)";
+ if (!pattern_has_jit(pattern)) {
+ SKIP("PCRE2 has no JIT for this pattern, so the JIT stack is never consulted");
+ }
+
+ Regex re;
+ REQUIRE(re.compile(pattern));
+
+ // 40 bytes of JIT stack per subject character, so PCRE2's 32KiB fallback stops at
+ // 818 characters and the shared context's 1MiB stack at 26,213. 5,000 sits six
+ // times above the first and five times below the second, so a blank context and
+ // the shared one give different answers with margin either way.
+ std::string const subject(5000, 'a');
+
+ RegexMatches shared_matches;
+ RegexMatchContext match_context;
+ RegexMatches own_matches;
+
+ int const shared_rc = re.exec(subject, shared_matches);
+ int const own_rc = re.exec(subject, own_matches, 0, &match_context);
+ CAPTURE(shared_rc, own_rc);
+
+ REQUIRE(shared_rc > 0);
+ REQUIRE(own_rc == shared_rc);
+
+ // The copy paths must carry the inherited configuration too. Building a blank
+ // context in either of them is the same defect, and nothing else here notices.
+ RegexMatchContext const copied{match_context};
+ RegexMatchContext assigned;
+
+ assigned = match_context;
+
+ RegexMatches copied_matches;
+ RegexMatches assigned_matches;
+
+ REQUIRE(re.exec(subject, copied_matches, 0, &copied) == shared_rc);
+ REQUIRE(re.exec(subject, assigned_matches, 0, &assigned) == shared_rc);
+}
+
+// Ownership only, and deliberately not inside the case above.
+//
+// That case SKIPs when pattern_has_jit() is false, which is right for what it tests and
+// wrong for this: move construction and move assignment have nothing to do with the JIT,
+// and a double free matters as much on a PCRE2 without JIT, or on a hardened runtime
+// where JIT allocation fails, as anywhere else. Inside that case this coverage would
+// skip on exactly those builds and still report green.
+TEST_CASE("RegexMatchContext move operations transfer ownership", "[libts][Regex][RegexMatchContext]")
+{
+ Regex re;
+
+ REQUIRE(re.compile("^(a+)(b+)$"));
+
+ std::string_view const subject{"aaabbb"};
+
+ auto matches_through = [&](RegexMatchContext const *ctx) {
+ RegexMatches m;
+ return re.exec(subject, m, 0, ctx);
+ };
+
+ // Baseline through a plain context, so a later mismatch means the move broke something
+ // rather than the pattern never having matched.
+ RegexMatchContext plain;
+ int const expected = matches_through(&plain);
+
+ REQUIRE(expected > 0);
+
+ SECTION("move construction")
+ {
+ RegexMatchContext donor;
+ RegexMatchContext const moved{std::move(donor)};
+
+ REQUIRE(matches_through(&moved) == expected);
+ // A double free aborts when donor and moved are both destroyed, so leaving this
+ // section is itself the second assertion.
+ }
+
+ SECTION("move assignment onto a destination that already holds a context")
+ {
+ // On an ordinary build the assertions below do not detect a dropped
+ // pcre2_match_context_free(old): removing that call leaves this section passing,
+ // because a leak has no observer here. The oracle for the leak is LeakSanitizer,
+ // which AddressSanitizer turns on by default on Linux but not on macOS.
+ //
+ // The section still earns its place: it is what gives LSan a move assignment onto a
+ // live context to observe, which no other case in this file produces. Without it
+ // the leak is unreachable and therefore undetectable anywhere.
+ RegexMatchContext destination;
+
+ REQUIRE(matches_through(&destination) == expected);
+
+ RegexMatchContext donor;
+
+ destination = std::move(donor);
+
+ REQUIRE(matches_through(&destination) == expected);
+ }
+
+ SECTION("self move assignment")
+ {
+ RegexMatchContext ctx;
+
+ // A match limit of one fails this match, so the result after the move shows whether
+ // ctx still holds its own context. Without the this != &that guard the operator frees
+ // the context and then nulls itself, and the match runs on PCRE2's defaults instead.
+ ctx.set_match_limit(1);
+ REQUIRE(matches_through(&ctx) == PCRE2_ERROR_MATCHLIMIT);
+
+ auto &alias = ctx;
+
+ ctx = std::move(alias);
+
+ REQUIRE(matches_through(&ctx) == PCRE2_ERROR_MATCHLIMIT);
+ }
+}
+
+// A pattern that backtracks once per character must fail cleanly rather than run
+// the thread out of stack. A regex engine that recurses on the machine stack crashes
+// the server on a long enough subject; PCRE2 must report an error instead.
+TEST_CASE("Regex reports resource exhaustion rather than crashing", "[libts][Regex][limits]")
+{
+ // Only the JIT path has a bound to exhaust here. PCRE2's interpreter keeps its
+ // backtracking frames on the heap, so it matches this subject rather than running
+ // out of anything, and there is no resource error to assert.
+ char const *const pattern = R"(^/alpha/bravo/[?]((?!action=(newsfeed|calendar|contacts|notepad)).)*$)";
+ if (!pattern_has_jit(pattern)) {
+ SKIP("PCRE2 has no JIT for this pattern, so there is no stack bound to exhaust");
+ }
+
+ Regex re;
+ REQUIRE(re.compile(pattern));
+
+ // This pattern starts failing at roughly 43KiB of subject against a 1MiB JIT
+ // stack, measured identically on x86_64 and arm64. Smaller JIT frames get more
+ // subject out of the same stack and so push that threshold up; 256KiB keeps a
+ // six times margin against that. Do not trim this to just above 43KiB.
+ std::string subject{"/alpha/bravo/?"};
+ subject.append(256 * 1024, 'x');
+
+ RegexMatches matches;
+ int const rc = re.exec(subject, matches);
+ CAPTURE(rc);
+
+ // Reaching this line at all is the crash assertion.
Review Comment:
Nit: odd way to phrase this. Consider removing the comment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Regex: hold the JIT stack in a pthread key and inherit the shared match context [trafficserver]
Posted by "JosiahWI (via GitHub)" <gi...@apache.org>.
JosiahWI commented on code in PR #13683:
URL: https://github.com/apache/trafficserver/pull/13683#discussion_r4020674234
##########
tests/gold_tests/pluginTest/regex_remap/regex_remap.test.py:
##########
@@ -123,19 +125,48 @@
tr.Processes.Default.Streams.stdout = "gold/regex_remap_simple.gold"
tr.StillRunningAfter = ts
-# 3 Test - Preserve the original crash guard from #5762. This request must
-# survive resource exhaustion without redirecting, regardless of which matching
-# resource limit is reached (JIT stack, match work, depth, or heap).
-tr = Test.AddTestRun("resource exhaustion does not crash ATS")
+# 3 Test - A 3 KB query redirects. This rule backtracks once per subject
+# character, so it used to exhaust the 32 KB stack PCRE2 falls back to when a
+# match context carries none, and the rule was skipped. The plugin's context now
+# inherits the shared 1 MB stack, so the rule matches and the redirect fires.
+#
+# This run needs no JIT gate. With JIT the 1 MB stack resolves the subject, and
+# without JIT the interpreter resolves it on the heap, so the redirect is the
+# answer either way. It only demonstrates the fix on a build that has a JIT.
Review Comment:
How hard would it be to gate it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Regex: hold the JIT stack in a pthread key and inherit the shared match context [trafficserver]
Posted by "JosiahWI (via GitHub)" <gi...@apache.org>.
JosiahWI commented on code in PR #13683:
URL: https://github.com/apache/trafficserver/pull/13683#discussion_r4020795421
##########
src/tsutil/unit_tests/test_Regex.cc:
##########
@@ -1050,3 +1055,163 @@ TEST_CASE("Regex end-anchor with alternation", "[libts][Regex]")
CHECK(r.exec("cdn.example.com.evil.com", matches) == RE_ERROR_NOMATCH);
CHECK(r.exec("prefix.cdn.example.com", matches) == RE_ERROR_NOMATCH);
}
+
+namespace
+{
+/** Does PCRE2 have JIT code for this pattern?
+ *
+ * The two tests below are about the JIT stack, and PCRE2 consults it only when it
+ * has JIT code to run. Without it both a blank context and the shared one take the
+ * interpreter and return the same answer, so the tests would pass whether or not
+ * the behaviour they describe is present. Ask PCRE2 rather than assume.
+ */
+bool
+pattern_has_jit(char const *pattern)
+{
+ int errnum = 0;
+ PCRE2_SIZE erroffset = 0;
+ pcre2_code *code = pcre2_compile(reinterpret_cast<PCRE2_SPTR>(pattern), PCRE2_ZERO_TERMINATED, 0, &errnum, &erroffset, nullptr);
+ if (code == nullptr) {
+ return false;
+ }
+ pcre2_jit_compile(code, PCRE2_JIT_COMPLETE);
+ size_t jit_size = 0;
+ pcre2_pattern_info(code, PCRE2_INFO_JITSIZE, &jit_size);
+ pcre2_code_free(code);
+ return jit_size > 0;
+}
+} // namespace
+
+// A caller-supplied RegexMatchContext must behave like the shared context that
+// Regex::exec uses when none is supplied. A context built from scratch silently
+// drops everything the shared one configures, which is how regex_remap came to
+// run with PCRE2's fallback 32KiB JIT stack instead of the 1MiB one.
+TEST_CASE("RegexMatchContext matches the shared context", "[libts][Regex][RegexMatchContext]")
+{
+ // Quantified alternation of capture groups: every subject character pushes a
+ // backtracking frame, so the JIT stack size is what bounds this.
+ char const *const pattern = R"(^(?:(a)|(b))+$)";
+ if (!pattern_has_jit(pattern)) {
+ SKIP("PCRE2 has no JIT for this pattern, so the JIT stack is never consulted");
+ }
+
+ Regex re;
+ REQUIRE(re.compile(pattern));
+
+ std::string const subject(1000, 'a');
+
+ RegexMatches shared_matches;
+ RegexMatchContext match_context;
+ RegexMatches own_matches;
+
+ int const shared_rc = re.exec(subject, shared_matches);
+ int const own_rc = re.exec(subject, own_matches, 0, &match_context);
+ CAPTURE(shared_rc, own_rc);
+
+ REQUIRE(shared_rc > 0);
+ REQUIRE(own_rc == shared_rc);
+}
+
+// The guard from #5762: a pattern that backtracks once per character must fail
+// cleanly rather than run the thread out of stack. PCRE1 recursed on the machine
+// stack and a long enough subject crashed the server; PCRE2 must report an error
+// instead. If this ever crashes rather than fails, that regression is back.
Review Comment:
Remove the last sentence, please. I think the programmer can discern easily enough that a failing test could mean a regression after reading the first part of the comment.
##########
src/tsutil/unit_tests/test_Regex.cc:
##########
@@ -1050,3 +1055,163 @@ TEST_CASE("Regex end-anchor with alternation", "[libts][Regex]")
CHECK(r.exec("cdn.example.com.evil.com", matches) == RE_ERROR_NOMATCH);
CHECK(r.exec("prefix.cdn.example.com", matches) == RE_ERROR_NOMATCH);
}
+
+namespace
+{
+/** Does PCRE2 have JIT code for this pattern?
+ *
+ * The two tests below are about the JIT stack, and PCRE2 consults it only when it
+ * has JIT code to run. Without it both a blank context and the shared one take the
+ * interpreter and return the same answer, so the tests would pass whether or not
+ * the behaviour they describe is present. Ask PCRE2 rather than assume.
+ */
+bool
+pattern_has_jit(char const *pattern)
+{
+ int errnum = 0;
+ PCRE2_SIZE erroffset = 0;
+ pcre2_code *code = pcre2_compile(reinterpret_cast<PCRE2_SPTR>(pattern), PCRE2_ZERO_TERMINATED, 0, &errnum, &erroffset, nullptr);
+ if (code == nullptr) {
+ return false;
+ }
Review Comment:
This error handling is a risk for future bugs. Returning `false` from `pattern_has_jit`, due to an error while compiling the pattern, communicates the wrong idea to the programmer calling the function. I think it might be best to throw an exception here, but returning a stronger error-carrying type or doing a release assert are alternatives.
##########
src/tsutil/unit_tests/test_Regex.cc:
##########
@@ -1050,3 +1055,163 @@ TEST_CASE("Regex end-anchor with alternation", "[libts][Regex]")
CHECK(r.exec("cdn.example.com.evil.com", matches) == RE_ERROR_NOMATCH);
CHECK(r.exec("prefix.cdn.example.com", matches) == RE_ERROR_NOMATCH);
}
+
+namespace
+{
+/** Does PCRE2 have JIT code for this pattern?
+ *
+ * The two tests below are about the JIT stack, and PCRE2 consults it only when it
+ * has JIT code to run. Without it both a blank context and the shared one take the
+ * interpreter and return the same answer, so the tests would pass whether or not
+ * the behaviour they describe is present. Ask PCRE2 rather than assume.
+ */
+bool
+pattern_has_jit(char const *pattern)
+{
+ int errnum = 0;
+ PCRE2_SIZE erroffset = 0;
+ pcre2_code *code = pcre2_compile(reinterpret_cast<PCRE2_SPTR>(pattern), PCRE2_ZERO_TERMINATED, 0, &errnum, &erroffset, nullptr);
+ if (code == nullptr) {
+ return false;
+ }
+ pcre2_jit_compile(code, PCRE2_JIT_COMPLETE);
Review Comment:
The PCRE2 library provides a mechanism specifically for a similar kind of test, which simplifies this function implementation considerably. One caveat: the API I'm going to mention is used to test general JIT support, but the wording of your comment suggests JIT can be supported for some patterns and not others. In that case, this API would not be applicable. Do you have a reference for that?
> The availability of JIT support can be tested by calling pcre2_compile_jit() with a single option PCRE2_JIT_TEST_ALLOC (the code argument is ignored, so a NULL value is accepted). Such a call returns zero if JIT is available and has a working allocator. Otherwise it returns PCRE2_ERROR_NOMEMORY if JIT is available but cannot allocate executable memory, or PCRE2_ERROR_JIT_UNSUPPORTED if JIT support is not compiled.
\- [PCRE2 API reference](https://www.pcre.org/current/doc/html/pcre2_jit_compile.html)
##########
src/tsutil/unit_tests/test_Regex.cc:
##########
@@ -1050,3 +1055,163 @@ TEST_CASE("Regex end-anchor with alternation", "[libts][Regex]")
CHECK(r.exec("cdn.example.com.evil.com", matches) == RE_ERROR_NOMATCH);
CHECK(r.exec("prefix.cdn.example.com", matches) == RE_ERROR_NOMATCH);
}
+
+namespace
+{
+/** Does PCRE2 have JIT code for this pattern?
+ *
+ * The two tests below are about the JIT stack, and PCRE2 consults it only when it
+ * has JIT code to run. Without it both a blank context and the shared one take the
+ * interpreter and return the same answer, so the tests would pass whether or not
+ * the behaviour they describe is present. Ask PCRE2 rather than assume.
+ */
+bool
+pattern_has_jit(char const *pattern)
+{
+ int errnum = 0;
+ PCRE2_SIZE erroffset = 0;
+ pcre2_code *code = pcre2_compile(reinterpret_cast<PCRE2_SPTR>(pattern), PCRE2_ZERO_TERMINATED, 0, &errnum, &erroffset, nullptr);
+ if (code == nullptr) {
+ return false;
+ }
+ pcre2_jit_compile(code, PCRE2_JIT_COMPLETE);
+ size_t jit_size = 0;
+ pcre2_pattern_info(code, PCRE2_INFO_JITSIZE, &jit_size);
+ pcre2_code_free(code);
+ return jit_size > 0;
+}
+} // namespace
+
+// A caller-supplied RegexMatchContext must behave like the shared context that
+// Regex::exec uses when none is supplied. A context built from scratch silently
+// drops everything the shared one configures, which is how regex_remap came to
+// run with PCRE2's fallback 32KiB JIT stack instead of the 1MiB one.
+TEST_CASE("RegexMatchContext matches the shared context", "[libts][Regex][RegexMatchContext]")
+{
+ // Quantified alternation of capture groups: every subject character pushes a
+ // backtracking frame, so the JIT stack size is what bounds this.
+ char const *const pattern = R"(^(?:(a)|(b))+$)";
+ if (!pattern_has_jit(pattern)) {
+ SKIP("PCRE2 has no JIT for this pattern, so the JIT stack is never consulted");
+ }
+
+ Regex re;
+ REQUIRE(re.compile(pattern));
+
+ std::string const subject(1000, 'a');
+
+ RegexMatches shared_matches;
+ RegexMatchContext match_context;
+ RegexMatches own_matches;
+
+ int const shared_rc = re.exec(subject, shared_matches);
+ int const own_rc = re.exec(subject, own_matches, 0, &match_context);
+ CAPTURE(shared_rc, own_rc);
+
+ REQUIRE(shared_rc > 0);
+ REQUIRE(own_rc == shared_rc);
+}
+
+// The guard from #5762: a pattern that backtracks once per character must fail
+// cleanly rather than run the thread out of stack. PCRE1 recursed on the machine
+// stack and a long enough subject crashed the server; PCRE2 must report an error
+// instead. If this ever crashes rather than fails, that regression is back.
+TEST_CASE("Regex reports resource exhaustion rather than crashing", "[libts][Regex][limits]")
+{
+ // Only the JIT path has a bound to exhaust here. PCRE2's interpreter keeps its
+ // backtracking frames on the heap, so it matches this subject rather than running
+ // out of anything, and there is no resource error to assert.
+ char const *const pattern = R"(^/alpha/bravo/[?]((?!action=(newsfeed|calendar|contacts|notepad)).)*$)";
+ if (!pattern_has_jit(pattern)) {
+ SKIP("PCRE2 has no JIT for this pattern, so there is no stack bound to exhaust");
+ }
+
+ Regex re;
+ REQUIRE(re.compile(pattern));
+
+ // This pattern starts failing at roughly 43KiB of subject against a 1MiB JIT
+ // stack, measured identically on x86_64 and arm64. 256KiB keeps a six times
+ // margin for a platform whose JIT frames are larger, without allocating more
+ // than the bound needs. Do not trim this to just above 43KiB.
+ std::string subject{"/alpha/bravo/?"};
+ subject.append(256 * 1024, 'x');
+
+ RegexMatches matches;
+ int const rc = re.exec(subject, matches);
+ CAPTURE(rc);
+
+ // Reaching this line at all is the crash assertion.
+ REQUIRE(rc < 0);
+ REQUIRE(rc != RE_ERROR_NOMATCH);
+}
+
+// The header promises that exec() may be called concurrently on one instance, and nothing
+// tested that. Every thread must reach the same verdict, whether it matches through the
+// shared context or through one it built itself, and each thread must get its own JIT
+// stack from the callback rather than share one. Run this under ThreadSanitizer to get the
+// second half of the guarantee.
+TEST_CASE("Regex matches concurrently on one instance", "[libts][Regex][threads]")
+{
Review Comment:
I think some setup or other helpers could usefully be extracted from the concurrent test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Regex: hold the JIT stack in a pthread key and inherit the shared match context [trafficserver]
Posted by "bryancall (via GitHub)" <gi...@apache.org>.
bryancall commented on PR #13683:
URL: https://github.com/apache/trafficserver/pull/13683#issuecomment-5683346119
Both findings from the Copilot review are addressed. The JIT gating one had a thread and is answered and resolved there; this covers the suppressed one, which had no thread to reply in.
**Multithreaded coverage.** You were right, and the gap was mine: I had written exactly this test and then filed it with the wrong change. It sat on the follow-up branch that makes the pcre2 contexts process-wide, because that is where it first landed. The property it guards is introduced *here*, by resolving the JIT stack per thread through a callback, so here is where it belongs. Moved in c4cdac1110.
Eight threads match on one `Regex`, half of them through a single `RegexMatchContext` built before the workers start. That is deliberate and it is the shape your comment asked for: an implementation that cached one stack in the context, or handed the same stack to every thread, passes a test that gives each thread its own context and corrupts this one. The start gate is a mutex and condition variable rather than `std::latch`, which says it more directly but is not in libstdc++ before 11 and the CentOS lane runs devtoolset-10.
Verified on Fedora 44, gcc 16.2.1, PCRE2 10.47:
- `[Regex]` under ThreadSanitizer: 388 assertions in 21 cases, no warnings. `[threads]` alone is also clean.
- `[Regex]` under AddressSanitizer with UBSan: same counts, clean.
- `regex_remap` autest executes 14 runs with `TS_HAS_PCRE2_JIT 1`; `diags.log` carries both `-46` and `-47`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Regex: hold the JIT stack in a pthread key and inherit the shared match context [trafficserver]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #13683:
URL: https://github.com/apache/trafficserver/pull/13683#discussion_r4350836160
##########
src/tsutil/Regex.cc:
##########
@@ -80,6 +81,70 @@ my_free(void *ptr, void * /*caller*/)
free(ptr);
}
+//----------------------------------------------------------------------------
+// One match context is shared by every thread that matches through it, and PCRE2
+// requires a distinct JIT stack per thread, so the stack comes from a callback
+// invoked at match time rather than a pointer baked in when the context is built.
+//
+// The per thread stack is held in a pthread key rather than a thread_local. A
+// thread_local with a destructor registers it through __cxa_thread_atexit, which
+// takes the dynamic loader lock; doing that from a match would invert lock order
+// against a dlopen caller running a plugin's static initialization. See the same
+// hazard described at Diags::tag_activated. A pthread key registers its destructor
+// once, at key creation, and never from the matching path.
+pthread_key_t jit_stack_key;
+bool jit_stack_key_valid = false;
+pthread_once_t jit_stack_key_once = PTHREAD_ONCE_INIT;
Review Comment:
These file-scope symbols appear to have external linkage as written. To avoid exporting internal implementation details (and reduce risk of name collisions across translation units), give them internal linkage (e.g., place them in an unnamed namespace or mark them `static`).
##########
src/tsutil/Regex.cc:
##########
@@ -279,22 +355,58 @@ RegexMatchContext::operator=(RegexMatchContext const &other)
{
if (&other != this) {
auto ptr = _MatchContext::get(other._match_context);
- if (nullptr != ptr) {
- pcre2_match_context *const ctx = pcre2_match_context_copy(ptr);
- _MatchContext::set(_match_context, ctx);
- } else {
- _MatchContext::set(_match_context, nullptr);
+
+ // Take the copy before releasing what this object already holds, so a failing
+ // copy leaves it holding its old context rather than a freed one. Releasing it
+ // is what this operator used to omit, and every assignment leaked one context.
+ pcre2_match_context *const ctx = nullptr != ptr ? pcre2_match_context_copy(ptr) : nullptr;
+
+ // pcre2_match_context_copy() returns nullptr when it cannot allocate. Assigning it
+ // anyway would free the old context and leave this object empty, which is the very
+ // thing the ordering above exists to prevent, so leave the object untouched instead.
+ if (nullptr != ptr && nullptr == ctx) {
Review Comment:
On allocation failure, copy-assignment silently becomes a no-op. That preserves the old value (good), but it also violates typical assignment expectations without any signal to the caller. Consider adding a `debug_assert_message`/diagnostic log on this path, or documenting in the header that copy-assignment may fail and leave the destination unchanged under memory pressure.
##########
src/tsutil/Regex.cc:
##########
@@ -80,6 +81,70 @@ my_free(void *ptr, void * /*caller*/)
free(ptr);
}
+//----------------------------------------------------------------------------
+// One match context is shared by every thread that matches through it, and PCRE2
+// requires a distinct JIT stack per thread, so the stack comes from a callback
+// invoked at match time rather than a pointer baked in when the context is built.
+//
+// The per thread stack is held in a pthread key rather than a thread_local. A
+// thread_local with a destructor registers it through __cxa_thread_atexit, which
+// takes the dynamic loader lock; doing that from a match would invert lock order
+// against a dlopen caller running a plugin's static initialization. See the same
+// hazard described at Diags::tag_activated. A pthread key registers its destructor
+// once, at key creation, and never from the matching path.
+pthread_key_t jit_stack_key;
+bool jit_stack_key_valid = false;
+pthread_once_t jit_stack_key_once = PTHREAD_ONCE_INIT;
+
+void
+destroy_jit_stack(void *stack)
+{
+ if (stack != nullptr) {
+ pcre2_jit_stack_free(static_cast<pcre2_jit_stack *>(stack));
+ }
+}
+
+void
+make_jit_stack_key()
+{
+ jit_stack_key_valid = pthread_key_create(&jit_stack_key, destroy_jit_stack) == 0;
+}
+
+pcre2_jit_stack *
+jit_stack_for_this_thread(void *)
+{
+ pthread_once(&jit_stack_key_once, make_jit_stack_key);
+ if (!jit_stack_key_valid) {
+ // Without a key there is nowhere to keep a stack, and jit_stack_key holds a
+ // default value that may name an unrelated key. Returning null tells PCRE2 to
+ // use its own default stack, which pcre2jit documents as thread safe.
+ return nullptr;
+ }
+
+ auto *stack = static_cast<pcre2_jit_stack *>(pthread_getspecific(jit_stack_key));
+ if (stack == nullptr) {
+ // One page to start, one mebibyte at most. The maximum is address space reserved at
+ // creation and made resident only as deep as a match actually goes, so a larger one
+ // costs nothing per match.
+ //
+ // It does NOT cover every subject a client can send. Measured with the pattern the
+ // unit tests use, a mebibyte resolves about 26,213 characters, while
+ // proxy.config.http.request_header_max_size defaults to 32768. A longer subject
+ // fails with PCRE2_ERROR_JIT_STACKLIMIT, which exec() returns as a negative code.
+ // There is no retreat to a smaller stack and none to the interpreter: measured at
+ // 26,214 characters and again at 100,000, the match returns -46 rather than
+ // completing by another route.
+ stack = pcre2_jit_stack_create(4096, 1024 * 1024, nullptr);
+ if (pthread_setspecific(jit_stack_key, stack) != 0) {
+ // Nothing holds the stack now, so it would leak once per match. Give it back and
+ // let PCRE2 use its own default stack for this call.
+ pcre2_jit_stack_free(stack);
Review Comment:
If `pcre2_jit_stack_create(...)` returns `nullptr`, this still calls `pthread_setspecific(jit_stack_key, stack)` and then (on failure) `pcre2_jit_stack_free(stack)` with a null pointer. It’s safer to explicitly handle allocation failure by returning `nullptr` immediately before calling `pthread_setspecific`, and only freeing when `stack` is non-null.
##########
src/tsutil/unit_tests/test_Regex.cc:
##########
@@ -1646,3 +1651,362 @@ TEST_CASE("Regex copies answer the same as their original", "[libts][Regex][copy
CHECK(original.exec(ordinary, original_ordinary) == copy.exec(ordinary, copy_ordinary));
}
}
+
+namespace
+{
+/** Does PCRE2 have JIT code for this pattern?
+ *
+ * The two tests below are about the JIT stack, and PCRE2 consults it only when it
+ * has JIT code to run. Without it both a blank context and the shared one take the
+ * interpreter and return the same answer, so the tests would pass whether or not
+ * the behaviour they describe is present. Ask PCRE2 rather than assume.
+ *
+ * This asks about one pattern, deliberately: PCRE2 declines some pattern items and
+ * leaves PCRE2_INFO_JITSIZE at zero on a build whose JIT is otherwise fine, so the
+ * library-wide probes (pcre2_config, PCRE2_JIT_TEST_ALLOC) answer a different
+ * question. Call it from the test thread only; Catch2 assertions are not thread safe.
+ */
+bool
+pattern_has_jit(char const *pattern)
+{
+ int errnum = 0;
+ PCRE2_SIZE erroffset = 0;
+ pcre2_code *code = pcre2_compile(reinterpret_cast<PCRE2_SPTR>(pattern), PCRE2_ZERO_TERMINATED, 0, &errnum, &erroffset, nullptr);
+
+ // A pattern that will not compile is a broken test, not a build without a JIT.
+ // Reporting it as "no JIT" would turn a typo into a silent skip.
+ REQUIRE(code != nullptr);
+
+ pcre2_jit_compile(code, PCRE2_JIT_COMPLETE);
+ size_t jit_size = 0;
+
+ pcre2_pattern_info(code, PCRE2_INFO_JITSIZE, &jit_size);
+ pcre2_code_free(code);
+ return jit_size > 0;
+}
+} // namespace
+
+// A caller-supplied RegexMatchContext must behave like the shared context that
+// Regex::exec uses when none is supplied. A context built from scratch silently
+// drops everything the shared one configures, which is how regex_remap came to
+// run with PCRE2's fallback 32KiB JIT stack instead of the 1MiB one.
+TEST_CASE("RegexMatchContext matches the shared context", "[libts][Regex][RegexMatchContext]")
+{
+ // Quantified alternation of capture groups: every subject character pushes a
+ // backtracking frame, so the JIT stack size is what bounds this.
+ char const *const pattern = R"(^(?:(a)|(b))+$)";
+ if (!pattern_has_jit(pattern)) {
+ SKIP("PCRE2 has no JIT for this pattern, so the JIT stack is never consulted");
+ }
+
+ Regex re;
+ REQUIRE(re.compile(pattern));
+
+ // 40 bytes of JIT stack per subject character, so PCRE2's 32KiB fallback stops at
+ // 818 characters and the shared context's 1MiB stack at 26,213. 5,000 sits six
+ // times above the first and five times below the second, so a blank context and
+ // the shared one give different answers with margin either way.
+ std::string const subject(5000, 'a');
+
+ RegexMatches shared_matches;
+ RegexMatchContext match_context;
+ RegexMatches own_matches;
+
+ int const shared_rc = re.exec(subject, shared_matches);
+ int const own_rc = re.exec(subject, own_matches, 0, &match_context);
+ CAPTURE(shared_rc, own_rc);
+
+ REQUIRE(shared_rc > 0);
+ REQUIRE(own_rc == shared_rc);
+
+ // The copy paths must carry the inherited configuration too. Building a blank
+ // context in either of them is the same defect, and nothing else here notices.
+ RegexMatchContext const copied{match_context};
+ RegexMatchContext assigned;
+
+ assigned = match_context;
+
+ RegexMatches copied_matches;
+ RegexMatches assigned_matches;
+
+ REQUIRE(re.exec(subject, copied_matches, 0, &copied) == shared_rc);
+ REQUIRE(re.exec(subject, assigned_matches, 0, &assigned) == shared_rc);
+}
+
+// Ownership only, and deliberately NOT inside the case above.
+//
+// That case SKIPs when pattern_has_jit() is false, which is right for what it tests and
+// wrong for this: move construction and move assignment have nothing to do with the JIT.
+// Gating them there meant the only coverage of the double free stopped running on exactly
+// the builds it was written for, a PCRE2 without JIT or the hardened runtime where JIT
+// allocation fails. A regression test that disappears on the affected platform is worse
+// than none, because it reports green.
+TEST_CASE("RegexMatchContext move operations transfer ownership", "[libts][Regex][RegexMatchContext]")
+{
+ Regex re;
+
+ REQUIRE(re.compile("^(a+)(b+)$"));
+
+ std::string_view const subject{"aaabbb"};
+
+ auto matches_through = [&](RegexMatchContext const *ctx) {
+ RegexMatches m;
+ return re.exec(subject, m, 0, ctx);
+ };
+
+ // Baseline through a plain context, so a later mismatch means the move broke something
+ // rather than the pattern never having matched.
+ RegexMatchContext plain;
+ int const expected = matches_through(&plain);
+
+ REQUIRE(expected > 0);
+
+ SECTION("move construction")
+ {
+ RegexMatchContext donor;
+ RegexMatchContext const moved{std::move(donor)};
+
+ REQUIRE(matches_through(&moved) == expected);
+ // A double free aborts when donor and moved are both destroyed, so leaving this
+ // section is itself the second assertion.
+ }
+
+ SECTION("move assignment onto a destination that already holds a context")
+ {
+ // HONEST LABEL: on an ordinary build the assertions below do NOT detect a dropped
+ // pcre2_match_context_free(old). Measured: removing that call leaves this section
+ // passing, because a leak has no observer here. The oracle for the leak is
+ // LeakSanitizer on the ci-rocky lane, which is the only PR check inheriting asan.
+ //
+ // The section still earns its place: it is what gives LSan an assignment onto a
+ // live context to observe, which no other case in this file produces. Without it
+ // the leak is unreachable and therefore undetectable anywhere.
+ RegexMatchContext destination;
+
+ REQUIRE(matches_through(&destination) == expected);
+
+ RegexMatchContext donor;
+
+ destination = std::move(donor);
+
+ REQUIRE(matches_through(&destination) == expected);
+ }
+
+ SECTION("self move assignment")
+ {
+ RegexMatchContext ctx;
+
+ REQUIRE(matches_through(&ctx) == expected);
+
+ // Without the this != &that guard this frees the context and then reads the freed
+ // pointer back into itself, so the match below runs on freed memory.
+ //
+ // HONEST LABEL: measured, removing that guard leaves this section passing on an
+ // ordinary build, because a use after free does not reliably fault. AddressSanitizer
+ // on ci-rocky is the oracle. As above, the value here is producing the access at all.
+ auto &alias = ctx;
+
+ ctx = std::move(alias);
+
+ REQUIRE(matches_through(&ctx) == expected);
+ }
+}
+
+// The guard from #5762: a pattern that backtracks once per character must fail
+// cleanly rather than run the thread out of stack. PCRE1 recursed on the machine
+// stack and a long enough subject crashed the server; PCRE2 must report an error
+// instead.
+TEST_CASE("Regex reports resource exhaustion rather than crashing", "[libts][Regex][limits]")
+{
+ // Only the JIT path has a bound to exhaust here. PCRE2's interpreter keeps its
+ // backtracking frames on the heap, so it matches this subject rather than running
+ // out of anything, and there is no resource error to assert.
+ char const *const pattern = R"(^/alpha/bravo/[?]((?!action=(newsfeed|calendar|contacts|notepad)).)*$)";
+ if (!pattern_has_jit(pattern)) {
+ SKIP("PCRE2 has no JIT for this pattern, so there is no stack bound to exhaust");
+ }
+
+ Regex re;
+ REQUIRE(re.compile(pattern));
+
+ // This pattern starts failing at roughly 43KiB of subject against a 1MiB JIT
+ // stack, measured identically on x86_64 and arm64. Smaller JIT frames get more
+ // subject out of the same stack and so push that threshold up; 256KiB keeps a
+ // six times margin against that. Do not trim this to just above 43KiB.
+ std::string subject{"/alpha/bravo/?"};
+ subject.append(256 * 1024, 'x');
+
+ RegexMatches matches;
+ int const rc = re.exec(subject, matches);
+ CAPTURE(rc);
+
+ // Reaching this line at all is the crash assertion.
+ REQUIRE(rc < 0);
+ REQUIRE(rc != RE_ERROR_NOMATCH);
+
+ // The paired positive control, and the only part of this case that can tell a
+ // 1MiB stack from PCRE2's 32KiB fallback: 256KiB overruns both bounds, so every
+ // assertion above also holds with the JIT stack callback deleted. 10,000 is seven
+ // times over the fallback's bound and four times under the 1MiB stack's. Neither
+ // bound pins 1MiB itself; the suite only requires a maximum in roughly
+ // [240KB, 6MB].
+ std::string smaller{"/alpha/bravo/?"};
+ smaller.append(10000, 'x');
+
+ RegexMatches small_matches;
+ int const small_rc = re.exec(smaller, small_matches);
+ CAPTURE(small_rc);
+
+ REQUIRE(small_rc > 0);
+}
+
+namespace
+{
+/** A reusable barrier, so every round starts with all threads alive and inside it. The
+ * matches of one round then start together on every thread. std::barrier would say this
+ * directly; it is hand-rolled here to match notstd::barrier in benchmark_LogObject.cc.
+ *
+ * Each completed round advances the generation, which is what releases the waiters, so
+ * the barrier can be crossed any number of times. Exactly `expected` threads must call
+ * arrive_and_wait() every round or the rest block forever, and the barrier must outlive
+ * all of them, since destroying a condition_variable with waiters on it is undefined.
+ */
+class RoundBarrier
+{
+public:
+ explicit RoundBarrier(int expected) : _expected{expected} { REQUIRE(expected > 0); }
+
+ /// Count this thread in, then block until every expected thread has done the same.
+ void
+ arrive_and_wait()
+ {
+ std::unique_lock<std::mutex> lock{_mutex};
+ auto const generation = _generation;
+
+ if (++_arrived == _expected) {
+ _arrived = 0;
+ ++_generation;
+ _cv.notify_all();
+ return;
+ }
+ _cv.wait(lock, [this, generation]() { return _generation != generation; });
+ }
+
+private:
+ std::mutex _mutex;
+ std::condition_variable _cv;
+ int const _expected;
+ int _arrived = 0;
+ unsigned _generation = 0;
+};
+
+/** A subject the deep pattern below backtracks through once per character.
+ *
+ * At 40 bytes of JIT stack per character the longest of these needs about 223KiB: six
+ * times what PCRE2's 32KiB fallback resolves, and under a quarter of the shared
+ * context's 1MiB stack. Each thread gets its own length and its own content, so no two
+ * threads are matching identical bytes.
+ */
+std::string
+stack_hungry_subject(int thread_index)
+{
+ std::string subject(5000 + thread_index * 100, 'a');
+
+ subject[11 + thread_index] = 'b';
+ return subject;
+}
+
+// One pass of the three matches every thread in the concurrency case performs: a hit that
+// must yield the same three captures, a miss that must stay a miss, and a deep subject that
+// only resolves on the 1MiB stack. Returns false on the first disagreement, so a caller
+// counts rounds that went wrong rather than individual assertions. The case asserts on zero
+// either way, so the change of denominator does not weaken it.
+bool
+concurrent_match_round(Regex const &re_captures, Regex const &re_deep, std::string const &hit, std::string const &miss,
+ std::string const &deep, RegexMatchContext const *const use)
+{
+ RegexMatches matches;
+ if (re_captures.exec(hit, matches, 0, use) != 4 || matches[1] != "alpha" || matches[2] != "42" || matches[3] != "tail") {
+ return false;
+ }
+
+ RegexMatches no_matches;
+ if (re_captures.exec(miss, no_matches, 0, use) != RE_ERROR_NOMATCH) {
+ return false;
+ }
+
+ // Needs the 1MiB stack, and must consume the whole subject to have used it.
+ RegexMatches deep_matches;
+ if (re_deep.exec(deep, deep_matches, 0, use) <= 0 || deep_matches[0].size() != deep.size()) {
+ return false;
+ }
+
+ return true;
+}
+
+} // namespace
+
+// The header promises that exec() may be called concurrently on one instance. Every thread
+// must reach the same verdict, whether it matches through a caller-supplied context shared
+// by several threads or through the thread-global one.
+//
+// The deep subjects are longer than PCRE2's 32KiB fallback stack can resolve, so on a JIT
+// build this also fails if the callback stops handing every thread a 1MiB stack: deleting
+// the callback, or returning null from it, turns every deep match here into
+// PCRE2_ERROR_JIT_STACKLIMIT. Without a JIT the interpreter resolves them either way, so
+// only the concurrency half of this case carries over to such a build.
+//
+// Every round starts at a barrier, so all eight threads are alive for the whole case and
+// each round's matches start together on every thread. The test is deterministic when the
+// stacks are distinct, but it still does not prove that they are: one stack shared by all
+// eight corrupts a result only when two deep matches touch the same frames at once, which
+// the barrier invites but cannot force. Measured on arm64 macOS with the callback returning
+// one shared stack, about 70 of 100 runs fail, with or without the barrier. An oracle
+// would need a thread paused inside a match, which the public API does not offer.
+// ThreadSanitizer does not see those writes either, because they come from sljit code.
+TEST_CASE("Regex matches concurrently on one instance", "[libts][Regex][threads]")
+{
+ Regex re_captures;
+ REQUIRE(re_captures.compile(R"(^/([a-z]+)/([0-9]+)/(.*)$)"));
+
+ Regex re_deep;
+ REQUIRE(re_deep.compile(R"(^(?:(a)|(b))+$)"));
+
+ constexpr int THREADS = 8;
+ constexpr int ITERATIONS = 2000;
Review Comment:
This test runs 8 threads Ă— 2000 iterations Ă— 3 regex execs per iteration (~48k execs plus synchronization). That can noticeably increase unit-test runtime on slower/contended CI workers. Consider reducing `ITERATIONS`, or restructuring to keep coverage while lowering work (e.g., fewer iterations but multiple deep matches per iteration, or a bounded runtime/iteration count tuned to the CI budget).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Regex: hold the JIT stack in a pthread key and inherit the shared match context [trafficserver]
Posted by "bryancall (via GitHub)" <gi...@apache.org>.
bryancall commented on code in PR #13683:
URL: https://github.com/apache/trafficserver/pull/13683#discussion_r4355468408
##########
src/tsutil/Regex.cc:
##########
@@ -80,6 +81,70 @@ my_free(void *ptr, void * /*caller*/)
free(ptr);
}
+//----------------------------------------------------------------------------
+// One match context is shared by every thread that matches through it, and PCRE2
+// requires a distinct JIT stack per thread, so the stack comes from a callback
+// invoked at match time rather than a pointer baked in when the context is built.
+//
+// The per thread stack is held in a pthread key rather than a thread_local. A
+// thread_local with a destructor registers it through __cxa_thread_atexit, which
+// takes the dynamic loader lock; doing that from a match would invert lock order
+// against a dlopen caller running a plugin's static initialization. See the same
+// hazard described at Diags::tag_activated. A pthread key registers its destructor
+// once, at key creation, and never from the matching path.
+pthread_key_t jit_stack_key;
+bool jit_stack_key_valid = false;
+pthread_once_t jit_stack_key_once = PTHREAD_ONCE_INIT;
Review Comment:
These already have internal linkage. They sit inside the unnamed namespace that opens at line 69 and closes at line 199, which is too far above this hunk to show in the diff. JosiahWI asked the same thing earlier and the answer is in that thread.
##########
src/tsutil/Regex.cc:
##########
@@ -80,6 +81,70 @@ my_free(void *ptr, void * /*caller*/)
free(ptr);
}
+//----------------------------------------------------------------------------
+// One match context is shared by every thread that matches through it, and PCRE2
+// requires a distinct JIT stack per thread, so the stack comes from a callback
+// invoked at match time rather than a pointer baked in when the context is built.
+//
+// The per thread stack is held in a pthread key rather than a thread_local. A
+// thread_local with a destructor registers it through __cxa_thread_atexit, which
+// takes the dynamic loader lock; doing that from a match would invert lock order
+// against a dlopen caller running a plugin's static initialization. See the same
+// hazard described at Diags::tag_activated. A pthread key registers its destructor
+// once, at key creation, and never from the matching path.
+pthread_key_t jit_stack_key;
+bool jit_stack_key_valid = false;
+pthread_once_t jit_stack_key_once = PTHREAD_ONCE_INIT;
+
+void
+destroy_jit_stack(void *stack)
+{
+ if (stack != nullptr) {
+ pcre2_jit_stack_free(static_cast<pcre2_jit_stack *>(stack));
+ }
+}
+
+void
+make_jit_stack_key()
+{
+ jit_stack_key_valid = pthread_key_create(&jit_stack_key, destroy_jit_stack) == 0;
+}
+
+pcre2_jit_stack *
+jit_stack_for_this_thread(void *)
+{
+ pthread_once(&jit_stack_key_once, make_jit_stack_key);
+ if (!jit_stack_key_valid) {
+ // Without a key there is nowhere to keep a stack, and jit_stack_key holds a
+ // default value that may name an unrelated key. Returning null tells PCRE2 to
+ // use its own default stack, which pcre2jit documents as thread safe.
+ return nullptr;
+ }
+
+ auto *stack = static_cast<pcre2_jit_stack *>(pthread_getspecific(jit_stack_key));
+ if (stack == nullptr) {
+ // One page to start, one mebibyte at most. The maximum is address space reserved at
+ // creation and made resident only as deep as a match actually goes, so a larger one
+ // costs nothing per match.
+ //
+ // It does NOT cover every subject a client can send. Measured with the pattern the
+ // unit tests use, a mebibyte resolves about 26,213 characters, while
+ // proxy.config.http.request_header_max_size defaults to 32768. A longer subject
+ // fails with PCRE2_ERROR_JIT_STACKLIMIT, which exec() returns as a negative code.
+ // There is no retreat to a smaller stack and none to the interpreter: measured at
+ // 26,214 characters and again at 100,000, the match returns -46 rather than
+ // completing by another route.
+ stack = pcre2_jit_stack_create(4096, 1024 * 1024, nullptr);
+ if (pthread_setspecific(jit_stack_key, stack) != 0) {
+ // Nothing holds the stack now, so it would leak once per match. Give it back and
+ // let PCRE2 use its own default stack for this call.
+ pcre2_jit_stack_free(stack);
Review Comment:
Both cases are already safe. If `pcre2_jit_stack_create()` returns null, `pthread_setspecific()` stores null and this returns null, and a null from the callback makes PCRE2 use its internal 32 KiB stack for that match (`pcre2_jit_stack_assign(3)`). The next match on the thread tries to create the stack again. On the failure branch, `pcre2_jit_stack_free(NULL)` is documented to return without doing anything. So I've left it as is rather than add a branch that changes no behavior.
##########
src/tsutil/Regex.cc:
##########
@@ -279,22 +355,58 @@ RegexMatchContext::operator=(RegexMatchContext const &other)
{
if (&other != this) {
auto ptr = _MatchContext::get(other._match_context);
- if (nullptr != ptr) {
- pcre2_match_context *const ctx = pcre2_match_context_copy(ptr);
- _MatchContext::set(_match_context, ctx);
- } else {
- _MatchContext::set(_match_context, nullptr);
+
+ // Take the copy before releasing what this object already holds, so a failing
+ // copy leaves it holding its old context rather than a freed one. Releasing it
+ // is what this operator used to omit, and every assignment leaked one context.
+ pcre2_match_context *const ctx = nullptr != ptr ? pcre2_match_context_copy(ptr) : nullptr;
+
+ // pcre2_match_context_copy() returns nullptr when it cannot allocate. Assigning it
+ // anyway would free the old context and leave this object empty, which is the very
+ // thing the ordering above exists to prevent, so leave the object untouched instead.
+ if (nullptr != ptr && nullptr == ctx) {
Review Comment:
This is deliberate. A copy assignment has no way to report failure, and when `pcre2_match_context_copy()` cannot allocate the two outcomes are an object with no context or an object that keeps the one it had. The comment above the check explains why it keeps the old one. The trade is that the object keeps its previous settings instead of `other`'s, and only when allocating a match context fails.
##########
src/tsutil/unit_tests/test_Regex.cc:
##########
@@ -1646,3 +1651,362 @@ TEST_CASE("Regex copies answer the same as their original", "[libts][Regex][copy
CHECK(original.exec(ordinary, original_ordinary) == copy.exec(ordinary, copy_ordinary));
}
}
+
+namespace
+{
+/** Does PCRE2 have JIT code for this pattern?
+ *
+ * The two tests below are about the JIT stack, and PCRE2 consults it only when it
+ * has JIT code to run. Without it both a blank context and the shared one take the
+ * interpreter and return the same answer, so the tests would pass whether or not
+ * the behaviour they describe is present. Ask PCRE2 rather than assume.
+ *
+ * This asks about one pattern, deliberately: PCRE2 declines some pattern items and
+ * leaves PCRE2_INFO_JITSIZE at zero on a build whose JIT is otherwise fine, so the
+ * library-wide probes (pcre2_config, PCRE2_JIT_TEST_ALLOC) answer a different
+ * question. Call it from the test thread only; Catch2 assertions are not thread safe.
+ */
+bool
+pattern_has_jit(char const *pattern)
+{
+ int errnum = 0;
+ PCRE2_SIZE erroffset = 0;
+ pcre2_code *code = pcre2_compile(reinterpret_cast<PCRE2_SPTR>(pattern), PCRE2_ZERO_TERMINATED, 0, &errnum, &erroffset, nullptr);
+
+ // A pattern that will not compile is a broken test, not a build without a JIT.
+ // Reporting it as "no JIT" would turn a typo into a silent skip.
+ REQUIRE(code != nullptr);
+
+ pcre2_jit_compile(code, PCRE2_JIT_COMPLETE);
+ size_t jit_size = 0;
+
+ pcre2_pattern_info(code, PCRE2_INFO_JITSIZE, &jit_size);
+ pcre2_code_free(code);
+ return jit_size > 0;
+}
+} // namespace
+
+// A caller-supplied RegexMatchContext must behave like the shared context that
+// Regex::exec uses when none is supplied. A context built from scratch silently
+// drops everything the shared one configures, which is how regex_remap came to
+// run with PCRE2's fallback 32KiB JIT stack instead of the 1MiB one.
+TEST_CASE("RegexMatchContext matches the shared context", "[libts][Regex][RegexMatchContext]")
+{
+ // Quantified alternation of capture groups: every subject character pushes a
+ // backtracking frame, so the JIT stack size is what bounds this.
+ char const *const pattern = R"(^(?:(a)|(b))+$)";
+ if (!pattern_has_jit(pattern)) {
+ SKIP("PCRE2 has no JIT for this pattern, so the JIT stack is never consulted");
+ }
+
+ Regex re;
+ REQUIRE(re.compile(pattern));
+
+ // 40 bytes of JIT stack per subject character, so PCRE2's 32KiB fallback stops at
+ // 818 characters and the shared context's 1MiB stack at 26,213. 5,000 sits six
+ // times above the first and five times below the second, so a blank context and
+ // the shared one give different answers with margin either way.
+ std::string const subject(5000, 'a');
+
+ RegexMatches shared_matches;
+ RegexMatchContext match_context;
+ RegexMatches own_matches;
+
+ int const shared_rc = re.exec(subject, shared_matches);
+ int const own_rc = re.exec(subject, own_matches, 0, &match_context);
+ CAPTURE(shared_rc, own_rc);
+
+ REQUIRE(shared_rc > 0);
+ REQUIRE(own_rc == shared_rc);
+
+ // The copy paths must carry the inherited configuration too. Building a blank
+ // context in either of them is the same defect, and nothing else here notices.
+ RegexMatchContext const copied{match_context};
+ RegexMatchContext assigned;
+
+ assigned = match_context;
+
+ RegexMatches copied_matches;
+ RegexMatches assigned_matches;
+
+ REQUIRE(re.exec(subject, copied_matches, 0, &copied) == shared_rc);
+ REQUIRE(re.exec(subject, assigned_matches, 0, &assigned) == shared_rc);
+}
+
+// Ownership only, and deliberately NOT inside the case above.
+//
+// That case SKIPs when pattern_has_jit() is false, which is right for what it tests and
+// wrong for this: move construction and move assignment have nothing to do with the JIT.
+// Gating them there meant the only coverage of the double free stopped running on exactly
+// the builds it was written for, a PCRE2 without JIT or the hardened runtime where JIT
+// allocation fails. A regression test that disappears on the affected platform is worse
+// than none, because it reports green.
+TEST_CASE("RegexMatchContext move operations transfer ownership", "[libts][Regex][RegexMatchContext]")
+{
+ Regex re;
+
+ REQUIRE(re.compile("^(a+)(b+)$"));
+
+ std::string_view const subject{"aaabbb"};
+
+ auto matches_through = [&](RegexMatchContext const *ctx) {
+ RegexMatches m;
+ return re.exec(subject, m, 0, ctx);
+ };
+
+ // Baseline through a plain context, so a later mismatch means the move broke something
+ // rather than the pattern never having matched.
+ RegexMatchContext plain;
+ int const expected = matches_through(&plain);
+
+ REQUIRE(expected > 0);
+
+ SECTION("move construction")
+ {
+ RegexMatchContext donor;
+ RegexMatchContext const moved{std::move(donor)};
+
+ REQUIRE(matches_through(&moved) == expected);
+ // A double free aborts when donor and moved are both destroyed, so leaving this
+ // section is itself the second assertion.
+ }
+
+ SECTION("move assignment onto a destination that already holds a context")
+ {
+ // HONEST LABEL: on an ordinary build the assertions below do NOT detect a dropped
+ // pcre2_match_context_free(old). Measured: removing that call leaves this section
+ // passing, because a leak has no observer here. The oracle for the leak is
+ // LeakSanitizer on the ci-rocky lane, which is the only PR check inheriting asan.
+ //
+ // The section still earns its place: it is what gives LSan an assignment onto a
+ // live context to observe, which no other case in this file produces. Without it
+ // the leak is unreachable and therefore undetectable anywhere.
+ RegexMatchContext destination;
+
+ REQUIRE(matches_through(&destination) == expected);
+
+ RegexMatchContext donor;
+
+ destination = std::move(donor);
+
+ REQUIRE(matches_through(&destination) == expected);
+ }
+
+ SECTION("self move assignment")
+ {
+ RegexMatchContext ctx;
+
+ REQUIRE(matches_through(&ctx) == expected);
+
+ // Without the this != &that guard this frees the context and then reads the freed
+ // pointer back into itself, so the match below runs on freed memory.
+ //
+ // HONEST LABEL: measured, removing that guard leaves this section passing on an
+ // ordinary build, because a use after free does not reliably fault. AddressSanitizer
+ // on ci-rocky is the oracle. As above, the value here is producing the access at all.
+ auto &alias = ctx;
+
+ ctx = std::move(alias);
+
+ REQUIRE(matches_through(&ctx) == expected);
+ }
+}
+
+// The guard from #5762: a pattern that backtracks once per character must fail
+// cleanly rather than run the thread out of stack. PCRE1 recursed on the machine
+// stack and a long enough subject crashed the server; PCRE2 must report an error
+// instead.
+TEST_CASE("Regex reports resource exhaustion rather than crashing", "[libts][Regex][limits]")
+{
+ // Only the JIT path has a bound to exhaust here. PCRE2's interpreter keeps its
+ // backtracking frames on the heap, so it matches this subject rather than running
+ // out of anything, and there is no resource error to assert.
+ char const *const pattern = R"(^/alpha/bravo/[?]((?!action=(newsfeed|calendar|contacts|notepad)).)*$)";
+ if (!pattern_has_jit(pattern)) {
+ SKIP("PCRE2 has no JIT for this pattern, so there is no stack bound to exhaust");
+ }
+
+ Regex re;
+ REQUIRE(re.compile(pattern));
+
+ // This pattern starts failing at roughly 43KiB of subject against a 1MiB JIT
+ // stack, measured identically on x86_64 and arm64. Smaller JIT frames get more
+ // subject out of the same stack and so push that threshold up; 256KiB keeps a
+ // six times margin against that. Do not trim this to just above 43KiB.
+ std::string subject{"/alpha/bravo/?"};
+ subject.append(256 * 1024, 'x');
+
+ RegexMatches matches;
+ int const rc = re.exec(subject, matches);
+ CAPTURE(rc);
+
+ // Reaching this line at all is the crash assertion.
+ REQUIRE(rc < 0);
+ REQUIRE(rc != RE_ERROR_NOMATCH);
+
+ // The paired positive control, and the only part of this case that can tell a
+ // 1MiB stack from PCRE2's 32KiB fallback: 256KiB overruns both bounds, so every
+ // assertion above also holds with the JIT stack callback deleted. 10,000 is seven
+ // times over the fallback's bound and four times under the 1MiB stack's. Neither
+ // bound pins 1MiB itself; the suite only requires a maximum in roughly
+ // [240KB, 6MB].
+ std::string smaller{"/alpha/bravo/?"};
+ smaller.append(10000, 'x');
+
+ RegexMatches small_matches;
+ int const small_rc = re.exec(smaller, small_matches);
+ CAPTURE(small_rc);
+
+ REQUIRE(small_rc > 0);
+}
+
+namespace
+{
+/** A reusable barrier, so every round starts with all threads alive and inside it. The
+ * matches of one round then start together on every thread. std::barrier would say this
+ * directly; it is hand-rolled here to match notstd::barrier in benchmark_LogObject.cc.
+ *
+ * Each completed round advances the generation, which is what releases the waiters, so
+ * the barrier can be crossed any number of times. Exactly `expected` threads must call
+ * arrive_and_wait() every round or the rest block forever, and the barrier must outlive
+ * all of them, since destroying a condition_variable with waiters on it is undefined.
+ */
+class RoundBarrier
+{
+public:
+ explicit RoundBarrier(int expected) : _expected{expected} { REQUIRE(expected > 0); }
+
+ /// Count this thread in, then block until every expected thread has done the same.
+ void
+ arrive_and_wait()
+ {
+ std::unique_lock<std::mutex> lock{_mutex};
+ auto const generation = _generation;
+
+ if (++_arrived == _expected) {
+ _arrived = 0;
+ ++_generation;
+ _cv.notify_all();
+ return;
+ }
+ _cv.wait(lock, [this, generation]() { return _generation != generation; });
+ }
+
+private:
+ std::mutex _mutex;
+ std::condition_variable _cv;
+ int const _expected;
+ int _arrived = 0;
+ unsigned _generation = 0;
+};
+
+/** A subject the deep pattern below backtracks through once per character.
+ *
+ * At 40 bytes of JIT stack per character the longest of these needs about 223KiB: six
+ * times what PCRE2's 32KiB fallback resolves, and under a quarter of the shared
+ * context's 1MiB stack. Each thread gets its own length and its own content, so no two
+ * threads are matching identical bytes.
+ */
+std::string
+stack_hungry_subject(int thread_index)
+{
+ std::string subject(5000 + thread_index * 100, 'a');
+
+ subject[11 + thread_index] = 'b';
+ return subject;
+}
+
+// One pass of the three matches every thread in the concurrency case performs: a hit that
+// must yield the same three captures, a miss that must stay a miss, and a deep subject that
+// only resolves on the 1MiB stack. Returns false on the first disagreement, so a caller
+// counts rounds that went wrong rather than individual assertions. The case asserts on zero
+// either way, so the change of denominator does not weaken it.
+bool
+concurrent_match_round(Regex const &re_captures, Regex const &re_deep, std::string const &hit, std::string const &miss,
+ std::string const &deep, RegexMatchContext const *const use)
+{
+ RegexMatches matches;
+ if (re_captures.exec(hit, matches, 0, use) != 4 || matches[1] != "alpha" || matches[2] != "42" || matches[3] != "tail") {
+ return false;
+ }
+
+ RegexMatches no_matches;
+ if (re_captures.exec(miss, no_matches, 0, use) != RE_ERROR_NOMATCH) {
+ return false;
+ }
+
+ // Needs the 1MiB stack, and must consume the whole subject to have used it.
+ RegexMatches deep_matches;
+ if (re_deep.exec(deep, deep_matches, 0, use) <= 0 || deep_matches[0].size() != deep.size()) {
+ return false;
+ }
+
+ return true;
+}
+
+} // namespace
+
+// The header promises that exec() may be called concurrently on one instance. Every thread
+// must reach the same verdict, whether it matches through a caller-supplied context shared
+// by several threads or through the thread-global one.
+//
+// The deep subjects are longer than PCRE2's 32KiB fallback stack can resolve, so on a JIT
+// build this also fails if the callback stops handing every thread a 1MiB stack: deleting
+// the callback, or returning null from it, turns every deep match here into
+// PCRE2_ERROR_JIT_STACKLIMIT. Without a JIT the interpreter resolves them either way, so
+// only the concurrency half of this case carries over to such a build.
+//
+// Every round starts at a barrier, so all eight threads are alive for the whole case and
+// each round's matches start together on every thread. The test is deterministic when the
+// stacks are distinct, but it still does not prove that they are: one stack shared by all
+// eight corrupts a result only when two deep matches touch the same frames at once, which
+// the barrier invites but cannot force. Measured on arm64 macOS with the callback returning
+// one shared stack, about 70 of 100 runs fail, with or without the barrier. An oracle
+// would need a thread paused inside a match, which the public API does not offer.
+// ThreadSanitizer does not see those writes either, because they come from sljit code.
+TEST_CASE("Regex matches concurrently on one instance", "[libts][Regex][threads]")
+{
+ Regex re_captures;
+ REQUIRE(re_captures.compile(R"(^/([a-z]+)/([0-9]+)/(.*)$)"));
+
+ Regex re_deep;
+ REQUIRE(re_deep.compile(R"(^(?:(a)|(b))+$)"));
+
+ constexpr int THREADS = 8;
+ constexpr int ITERATIONS = 2000;
Review Comment:
Measured on this branch (local arm64 macOS build), the test case takes 0.07 to 0.17 seconds across three runs, so I've left `ITERATIONS` at 2000. The count matters for what the test can catch: as the comment above it says, a shared stack corrupts a result only when two deep matches overlap, and fewer rounds would make that less likely to show up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Regex: hold the JIT stack in a pthread key and inherit the shared match context [trafficserver]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #13683:
URL: https://github.com/apache/trafficserver/pull/13683#discussion_r4012025417
##########
tests/gold_tests/pluginTest/regex_remap/regex_remap.test.py:
##########
@@ -123,12 +125,29 @@
tr.Processes.Default.Streams.stdout = "gold/regex_remap_simple.gold"
tr.StillRunningAfter = ts
-# 3 Test - Preserve the original crash guard from #5762. This request must
+# 3 Test - A 3 KB query redirects. This rule backtracks once per subject
+# character, so it used to exhaust the 32 KB stack PCRE2 falls back to when a
+# match context carries none, and the rule was skipped. The plugin's context now
+# inherits the shared 1 MB stack, so the rule matches and the redirect fires.
+tr = Test.AddTestRun("long query redirects rather than exhausting the JIT stack")
+creq = replay_txns[1]['client-request']
+tr.MakeCurlCommand(
+ curl_and_args + f"--header 'uuid: {creq['headers']['fields'][1][1]}' '{creq['url']}'" + " | grep -e '^HTTP/' -e '^Location'",
+ ts=ts)
+tr.Processes.Default.ReturnCode = 0
+tr.Processes.Default.Streams.stdout = "gold/regex_remap_redirect.gold"
+tr.StillRunningAfter = ts
+
+# 3b Test - Preserve the original crash guard from #5762. This request must
# survive resource exhaustion without redirecting, regardless of which matching
-# resource limit is reached (JIT stack, match work, depth, or heap).
+# resource limit is reached (JIT stack, match work, depth, or heap). Against the
+# shared 1 MB stack this rule needs a subject past 43 KB to exhaust it, which is
+# why the request header limit is raised above. Shortening this query silently
+# turns the run into a plain redirect test.
+crash_guard_query = 'x' * 64000
tr = Test.AddTestRun("resource exhaustion does not crash ATS")
-creq = replay_txns[1]['client-request']
-tr.MakeCurlCommand(curl_and_args + f"--header 'uuid: {creq['headers']['fields'][1][1]}' '{creq['url']}'", ts=ts)
+tr.MakeCurlCommand(
+ curl_and_args + "--header 'uuid: 180' " + f"'http://example.one/alpha/bravo/?action=newsfed;{crash_guard_query}'", ts=ts)
Review Comment:
Please gate this crash-guard run on a JIT-capable PCRE2 build (or make its expected result conditional). `Regex::compile()` ignores a failed `pcre2_jit_compile`, and the new unit tests explicitly skip when `pattern_has_jit()` is false (src/tsutil/unit_tests/test_Regex.cc:1120-1122). Without JIT, this 64 KiB subject is interpreted instead of exhausting the 1 MiB JIT stack, so it follows the redirect and cannot match `regex_remap_crash.gold` or produce the resource-limit diagnostic, causing the AuTest to fail.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Regex: hold the JIT stack in a pthread key and inherit the shared match context [trafficserver]
Posted by "JosiahWI (via GitHub)" <gi...@apache.org>.
JosiahWI commented on code in PR #13683:
URL: https://github.com/apache/trafficserver/pull/13683#discussion_r4083500390
##########
src/tsutil/Regex.cc:
##########
@@ -278,22 +351,61 @@ RegexMatchContext::operator=(RegexMatchContext const &other)
{
if (&other != this) {
auto ptr = _MatchContext::get(other._match_context);
- if (nullptr != ptr) {
- pcre2_match_context *const ctx = pcre2_match_context_copy(ptr);
- _MatchContext::set(_match_context, ctx);
- } else {
- _MatchContext::set(_match_context, nullptr);
+
+ // Take the copy before releasing what this object already holds, so a failing
+ // copy leaves it holding its old context rather than a freed one. Releasing it
+ // is what this operator used to omit, and every assignment leaked one context.
+ pcre2_match_context *const ctx = nullptr != ptr ? pcre2_match_context_copy(ptr) : nullptr;
+
+ // pcre2_match_context_copy() returns nullptr when it cannot allocate. Assigning it
+ // anyway would free the old context and leave this object empty, which is the very
+ // thing the ordering above exists to prevent, so leave the object untouched instead.
+ if (nullptr != ptr && nullptr == ctx) {
+ return *this;
}
+
+ pcre2_match_context *const old = _MatchContext::get(_match_context);
+
+ _MatchContext::set(_match_context, ctx);
+ if (old != nullptr) {
+ pcre2_match_context_free(old);
+ }
+ }
+ return *this;
+}
+
+//----------------------------------------------------------------------------
+RegexMatchContext::RegexMatchContext(RegexMatchContext &&that) noexcept
+{
+ // Through the typed accessors rather than std::exchange on the raw member: _ptr is
+ // void *, and void * does not implicitly convert to pcre2_match_context *, which is
+ // what set() takes.
+ _MatchContext::set(_match_context, _MatchContext::get(that._match_context));
+ _MatchContext::set(that._match_context, nullptr);
+}
+
+//----------------------------------------------------------------------------
+RegexMatchContext &
+RegexMatchContext::operator=(RegexMatchContext &&that) noexcept
+{
+ if (this != &that) {
+ if (auto *const old = _MatchContext::get(_match_context); old != nullptr) {
+ pcre2_match_context_free(old);
+ }
+ _MatchContext::set(_match_context, _MatchContext::get(that._match_context));
+ _MatchContext::set(that._match_context, nullptr);
}
return *this;
}
//----------------------------------------------------------------------------
RegexMatchContext::~RegexMatchContext()
{
- auto ptr = _MatchContext::get(_match_context);
- debug_assert_message(ptr, "Failed to get the match context");
- if (ptr != nullptr) {
+ // No assert that the pointer is set. Null is now a legitimate state: a moved-from
+ // object holds nothing, and asserting here would abort a debug build on the first
+ // destruction of one. Before the move operations existed the only way to reach this
+ // with null was a failed construction, which is why the assert was reasonable then.
Review Comment:
This comment is talking about old code which is not relevant to this revision.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Regex: test concurrent matching and JIT stack exhaustion [trafficserver]
Posted by "bryancall (via GitHub) via github" <gi...@trafficserver.apache.org>.
bryancall commented on code in PR #13683:
URL: https://github.com/apache/trafficserver/pull/13683#discussion_r4532235394
##########
src/tsutil/unit_tests/test_Regex.cc:
##########
@@ -1646,3 +1651,358 @@ TEST_CASE("Regex copies answer the same as their original", "[libts][Regex][copy
CHECK(original.exec(ordinary, original_ordinary) == copy.exec(ordinary, copy_ordinary));
}
}
+
+namespace
+{
+/** Does PCRE2 have JIT code for this pattern?
+ *
+ * The two tests below are about the JIT stack, and PCRE2 consults it only when it
+ * has JIT code to run. Without it both a blank context and the shared one take the
+ * interpreter and return the same answer, so the tests would pass whether or not
+ * the behaviour they describe is present. Ask PCRE2 rather than assume.
+ *
+ * This asks about one pattern, deliberately: PCRE2 declines some pattern items and
+ * leaves PCRE2_INFO_JITSIZE at zero on a build whose JIT is otherwise fine, so the
+ * library-wide probes (pcre2_config, PCRE2_JIT_TEST_ALLOC) answer a different
+ * question. Call it from the test thread only; Catch2 assertions are not thread safe.
+ */
Review Comment:
Done in 494c2c3617c5, it's a plain comment now:
https://github.com/apache/trafficserver/blob/f7ec6bc9058fcb051ddf0117ca10523a0fdc9d36/src/tsutil/unit_tests/test_Regex.cc#L1691-L1713
##########
src/tsutil/unit_tests/test_Regex.cc:
##########
@@ -1646,3 +1651,358 @@ TEST_CASE("Regex copies answer the same as their original", "[libts][Regex][copy
CHECK(original.exec(ordinary, original_ordinary) == copy.exec(ordinary, copy_ordinary));
}
}
+
+namespace
+{
+/** Does PCRE2 have JIT code for this pattern?
+ *
+ * The two tests below are about the JIT stack, and PCRE2 consults it only when it
+ * has JIT code to run. Without it both a blank context and the shared one take the
+ * interpreter and return the same answer, so the tests would pass whether or not
+ * the behaviour they describe is present. Ask PCRE2 rather than assume.
+ *
+ * This asks about one pattern, deliberately: PCRE2 declines some pattern items and
+ * leaves PCRE2_INFO_JITSIZE at zero on a build whose JIT is otherwise fine, so the
+ * library-wide probes (pcre2_config, PCRE2_JIT_TEST_ALLOC) answer a different
+ * question. Call it from the test thread only; Catch2 assertions are not thread safe.
+ */
+bool
+pattern_has_jit(char const *pattern)
+{
+ int errnum = 0;
+ PCRE2_SIZE erroffset = 0;
+ pcre2_code *code = pcre2_compile(reinterpret_cast<PCRE2_SPTR>(pattern), PCRE2_ZERO_TERMINATED, 0, &errnum, &erroffset, nullptr);
+
+ // A pattern that will not compile is a broken test, not a build without a JIT.
+ // Reporting it as "no JIT" would turn a typo into a silent skip.
Review Comment:
Agreed. Reworded in 494c2c3617c5 so it only says why the REQUIRE is there: a typo in a pattern should fail the test, not turn into a SKIP.
https://github.com/apache/trafficserver/blob/f7ec6bc9058fcb051ddf0117ca10523a0fdc9d36/src/tsutil/unit_tests/test_Regex.cc#L1706-L1707
##########
src/tsutil/unit_tests/test_Regex.cc:
##########
@@ -1646,3 +1651,358 @@ TEST_CASE("Regex copies answer the same as their original", "[libts][Regex][copy
CHECK(original.exec(ordinary, original_ordinary) == copy.exec(ordinary, copy_ordinary));
}
}
+
+namespace
+{
+/** Does PCRE2 have JIT code for this pattern?
+ *
+ * The two tests below are about the JIT stack, and PCRE2 consults it only when it
+ * has JIT code to run. Without it both a blank context and the shared one take the
+ * interpreter and return the same answer, so the tests would pass whether or not
+ * the behaviour they describe is present. Ask PCRE2 rather than assume.
+ *
+ * This asks about one pattern, deliberately: PCRE2 declines some pattern items and
+ * leaves PCRE2_INFO_JITSIZE at zero on a build whose JIT is otherwise fine, so the
+ * library-wide probes (pcre2_config, PCRE2_JIT_TEST_ALLOC) answer a different
+ * question. Call it from the test thread only; Catch2 assertions are not thread safe.
+ */
+bool
+pattern_has_jit(char const *pattern)
+{
+ int errnum = 0;
+ PCRE2_SIZE erroffset = 0;
+ pcre2_code *code = pcre2_compile(reinterpret_cast<PCRE2_SPTR>(pattern), PCRE2_ZERO_TERMINATED, 0, &errnum, &erroffset, nullptr);
+
+ // A pattern that will not compile is a broken test, not a build without a JIT.
+ // Reporting it as "no JIT" would turn a typo into a silent skip.
+ REQUIRE(code != nullptr);
+
+ pcre2_jit_compile(code, PCRE2_JIT_COMPLETE);
+ size_t jit_size = 0;
+
+ pcre2_pattern_info(code, PCRE2_INFO_JITSIZE, &jit_size);
Review Comment:
Yes, it is. I checked on PCRE2 10.49 and on 10.32, which Rocky 8 ships. In every case where PCRE2_INFO_JITSIZE comes back zero, pcre2_jit_compile() also returns an error: -48 when executable memory is refused, and -68 on 10.49 (-48 on 10.32) for the two unsupported items, \C in UTF mode and a callout before an assertion condition. The only mismatch I found is a pattern that starts with (*NO_JIT). It returns 0 and gets no JIT code. None of ours do, and the comment says not to pass one.
So 494c2c3617c5 checks the return value and drops the JITSIZE query:
https://github.com/apache/trafficserver/blob/f7ec6bc9058fcb051ddf0117ca10523a0fdc9d36/src/tsutil/unit_tests/test_Regex.cc#L1709-L1712
I made the same change to the pre-10.45 fallback in `traffic_layout info` in b889988b0ea2 so the two agree. With executable mappings refused, the exhaustion case now SKIPs and the rest of `[Regex]` passes.
##########
src/tsutil/unit_tests/test_Regex.cc:
##########
@@ -1646,3 +1651,358 @@ TEST_CASE("Regex copies answer the same as their original", "[libts][Regex][copy
CHECK(original.exec(ordinary, original_ordinary) == copy.exec(ordinary, copy_ordinary));
}
}
+
+namespace
+{
+/** Does PCRE2 have JIT code for this pattern?
+ *
+ * The two tests below are about the JIT stack, and PCRE2 consults it only when it
+ * has JIT code to run. Without it both a blank context and the shared one take the
+ * interpreter and return the same answer, so the tests would pass whether or not
+ * the behaviour they describe is present. Ask PCRE2 rather than assume.
+ *
+ * This asks about one pattern, deliberately: PCRE2 declines some pattern items and
+ * leaves PCRE2_INFO_JITSIZE at zero on a build whose JIT is otherwise fine, so the
+ * library-wide probes (pcre2_config, PCRE2_JIT_TEST_ALLOC) answer a different
+ * question. Call it from the test thread only; Catch2 assertions are not thread safe.
+ */
+bool
+pattern_has_jit(char const *pattern)
+{
+ int errnum = 0;
+ PCRE2_SIZE erroffset = 0;
+ pcre2_code *code = pcre2_compile(reinterpret_cast<PCRE2_SPTR>(pattern), PCRE2_ZERO_TERMINATED, 0, &errnum, &erroffset, nullptr);
+
+ // A pattern that will not compile is a broken test, not a build without a JIT.
+ // Reporting it as "no JIT" would turn a typo into a silent skip.
+ REQUIRE(code != nullptr);
+
+ pcre2_jit_compile(code, PCRE2_JIT_COMPLETE);
+ size_t jit_size = 0;
+
+ pcre2_pattern_info(code, PCRE2_INFO_JITSIZE, &jit_size);
+ pcre2_code_free(code);
+ return jit_size > 0;
+}
+} // namespace
+
+// A caller-supplied RegexMatchContext must behave like the shared context that
+// Regex::exec uses when none is supplied. A context built from scratch drops
+// everything the shared one configures, the JIT stack callback included, and a
+// match through it then runs on PCRE2's fallback 32KiB JIT stack instead of the
+// 1MiB one.
+TEST_CASE("RegexMatchContext matches the shared context", "[libts][Regex][RegexMatchContext]")
+{
+ // Quantified alternation of capture groups: every subject character pushes a
+ // backtracking frame, so the JIT stack size is what bounds this.
+ char const *const pattern = R"(^(?:(a)|(b))+$)";
+ if (!pattern_has_jit(pattern)) {
+ SKIP("PCRE2 has no JIT for this pattern, so the JIT stack is never consulted");
+ }
+
+ Regex re;
+ REQUIRE(re.compile(pattern));
+
+ // 40 bytes of JIT stack per subject character, so PCRE2's 32KiB fallback stops at
+ // 818 characters and the shared context's 1MiB stack at 26,213. 5,000 sits six
+ // times above the first and five times below the second, so a blank context and
+ // the shared one give different answers with margin either way.
+ std::string const subject(5000, 'a');
+
+ RegexMatches shared_matches;
+ RegexMatchContext match_context;
+ RegexMatches own_matches;
+
+ int const shared_rc = re.exec(subject, shared_matches);
+ int const own_rc = re.exec(subject, own_matches, 0, &match_context);
+ CAPTURE(shared_rc, own_rc);
+
+ REQUIRE(shared_rc > 0);
+ REQUIRE(own_rc == shared_rc);
+
+ // The copy paths must carry the inherited configuration too. Building a blank
+ // context in either of them is the same defect, and nothing else here notices.
+ RegexMatchContext const copied{match_context};
+ RegexMatchContext assigned;
+
+ assigned = match_context;
+
+ RegexMatches copied_matches;
+ RegexMatches assigned_matches;
+
+ REQUIRE(re.exec(subject, copied_matches, 0, &copied) == shared_rc);
+ REQUIRE(re.exec(subject, assigned_matches, 0, &assigned) == shared_rc);
+}
+
+// Ownership only, and deliberately not inside the case above.
+//
+// That case SKIPs when pattern_has_jit() is false, which is right for what it tests and
+// wrong for this: move construction and move assignment have nothing to do with the JIT,
+// and a double free matters as much on a PCRE2 without JIT, or on a hardened runtime
+// where JIT allocation fails, as anywhere else. Inside that case this coverage would
+// skip on exactly those builds and still report green.
+TEST_CASE("RegexMatchContext move operations transfer ownership", "[libts][Regex][RegexMatchContext]")
+{
+ Regex re;
+
+ REQUIRE(re.compile("^(a+)(b+)$"));
+
+ std::string_view const subject{"aaabbb"};
+
+ auto matches_through = [&](RegexMatchContext const *ctx) {
+ RegexMatches m;
+ return re.exec(subject, m, 0, ctx);
+ };
+
+ // Baseline through a plain context, so a later mismatch means the move broke something
+ // rather than the pattern never having matched.
+ RegexMatchContext plain;
+ int const expected = matches_through(&plain);
+
+ REQUIRE(expected > 0);
+
+ SECTION("move construction")
+ {
+ RegexMatchContext donor;
+ RegexMatchContext const moved{std::move(donor)};
+
+ REQUIRE(matches_through(&moved) == expected);
+ // A double free aborts when donor and moved are both destroyed, so leaving this
+ // section is itself the second assertion.
+ }
+
+ SECTION("move assignment onto a destination that already holds a context")
+ {
+ // On an ordinary build the assertions below do not detect a dropped
+ // pcre2_match_context_free(old): removing that call leaves this section passing,
+ // because a leak has no observer here. The oracle for the leak is LeakSanitizer,
+ // which AddressSanitizer turns on by default on Linux but not on macOS.
+ //
+ // The section still earns its place: it is what gives LSan a move assignment onto a
+ // live context to observe, which no other case in this file produces. Without it
+ // the leak is unreachable and therefore undetectable anywhere.
+ RegexMatchContext destination;
+
+ REQUIRE(matches_through(&destination) == expected);
+
+ RegexMatchContext donor;
+
+ destination = std::move(donor);
+
+ REQUIRE(matches_through(&destination) == expected);
+ }
+
+ SECTION("self move assignment")
+ {
+ RegexMatchContext ctx;
+
+ // A match limit of one fails this match, so the result after the move shows whether
+ // ctx still holds its own context. Without the this != &that guard the operator frees
+ // the context and then nulls itself, and the match runs on PCRE2's defaults instead.
+ ctx.set_match_limit(1);
+ REQUIRE(matches_through(&ctx) == PCRE2_ERROR_MATCHLIMIT);
+
+ auto &alias = ctx;
+
+ ctx = std::move(alias);
+
+ REQUIRE(matches_through(&ctx) == PCRE2_ERROR_MATCHLIMIT);
+ }
+}
+
+// A pattern that backtracks once per character must fail cleanly rather than run
+// the thread out of stack. A regex engine that recurses on the machine stack crashes
+// the server on a long enough subject; PCRE2 must report an error instead.
+TEST_CASE("Regex reports resource exhaustion rather than crashing", "[libts][Regex][limits]")
+{
+ // Only the JIT path has a bound to exhaust here. PCRE2's interpreter keeps its
+ // backtracking frames on the heap, so it matches this subject rather than running
+ // out of anything, and there is no resource error to assert.
+ char const *const pattern = R"(^/alpha/bravo/[?]((?!action=(newsfeed|calendar|contacts|notepad)).)*$)";
+ if (!pattern_has_jit(pattern)) {
+ SKIP("PCRE2 has no JIT for this pattern, so there is no stack bound to exhaust");
+ }
+
+ Regex re;
+ REQUIRE(re.compile(pattern));
+
+ // This pattern starts failing at roughly 43KiB of subject against a 1MiB JIT
+ // stack, measured identically on x86_64 and arm64. Smaller JIT frames get more
+ // subject out of the same stack and so push that threshold up; 256KiB keeps a
+ // six times margin against that. Do not trim this to just above 43KiB.
+ std::string subject{"/alpha/bravo/?"};
+ subject.append(256 * 1024, 'x');
+
+ RegexMatches matches;
+ int const rc = re.exec(subject, matches);
+ CAPTURE(rc);
+
+ // Reaching this line at all is the crash assertion.
Review Comment:
Agreed, removed in fc186eeb9f05. The case name and the comment above it already say what it guards against.
##########
src/tsutil/unit_tests/test_Regex.cc:
##########
@@ -1646,3 +1651,358 @@ TEST_CASE("Regex copies answer the same as their original", "[libts][Regex][copy
CHECK(original.exec(ordinary, original_ordinary) == copy.exec(ordinary, copy_ordinary));
}
}
+
+namespace
+{
+/** Does PCRE2 have JIT code for this pattern?
+ *
+ * The two tests below are about the JIT stack, and PCRE2 consults it only when it
+ * has JIT code to run. Without it both a blank context and the shared one take the
+ * interpreter and return the same answer, so the tests would pass whether or not
+ * the behaviour they describe is present. Ask PCRE2 rather than assume.
+ *
+ * This asks about one pattern, deliberately: PCRE2 declines some pattern items and
+ * leaves PCRE2_INFO_JITSIZE at zero on a build whose JIT is otherwise fine, so the
+ * library-wide probes (pcre2_config, PCRE2_JIT_TEST_ALLOC) answer a different
+ * question. Call it from the test thread only; Catch2 assertions are not thread safe.
+ */
+bool
+pattern_has_jit(char const *pattern)
+{
+ int errnum = 0;
+ PCRE2_SIZE erroffset = 0;
+ pcre2_code *code = pcre2_compile(reinterpret_cast<PCRE2_SPTR>(pattern), PCRE2_ZERO_TERMINATED, 0, &errnum, &erroffset, nullptr);
+
+ // A pattern that will not compile is a broken test, not a build without a JIT.
+ // Reporting it as "no JIT" would turn a typo into a silent skip.
+ REQUIRE(code != nullptr);
+
+ pcre2_jit_compile(code, PCRE2_JIT_COMPLETE);
+ size_t jit_size = 0;
+
+ pcre2_pattern_info(code, PCRE2_INFO_JITSIZE, &jit_size);
+ pcre2_code_free(code);
+ return jit_size > 0;
+}
+} // namespace
+
+// A caller-supplied RegexMatchContext must behave like the shared context that
+// Regex::exec uses when none is supplied. A context built from scratch drops
+// everything the shared one configures, the JIT stack callback included, and a
+// match through it then runs on PCRE2's fallback 32KiB JIT stack instead of the
+// 1MiB one.
+TEST_CASE("RegexMatchContext matches the shared context", "[libts][Regex][RegexMatchContext]")
+{
+ // Quantified alternation of capture groups: every subject character pushes a
+ // backtracking frame, so the JIT stack size is what bounds this.
+ char const *const pattern = R"(^(?:(a)|(b))+$)";
+ if (!pattern_has_jit(pattern)) {
+ SKIP("PCRE2 has no JIT for this pattern, so the JIT stack is never consulted");
+ }
+
+ Regex re;
+ REQUIRE(re.compile(pattern));
+
+ // 40 bytes of JIT stack per subject character, so PCRE2's 32KiB fallback stops at
+ // 818 characters and the shared context's 1MiB stack at 26,213. 5,000 sits six
+ // times above the first and five times below the second, so a blank context and
+ // the shared one give different answers with margin either way.
+ std::string const subject(5000, 'a');
+
+ RegexMatches shared_matches;
+ RegexMatchContext match_context;
+ RegexMatches own_matches;
+
+ int const shared_rc = re.exec(subject, shared_matches);
+ int const own_rc = re.exec(subject, own_matches, 0, &match_context);
+ CAPTURE(shared_rc, own_rc);
+
+ REQUIRE(shared_rc > 0);
+ REQUIRE(own_rc == shared_rc);
+
+ // The copy paths must carry the inherited configuration too. Building a blank
+ // context in either of them is the same defect, and nothing else here notices.
+ RegexMatchContext const copied{match_context};
+ RegexMatchContext assigned;
+
+ assigned = match_context;
+
+ RegexMatches copied_matches;
+ RegexMatches assigned_matches;
+
+ REQUIRE(re.exec(subject, copied_matches, 0, &copied) == shared_rc);
+ REQUIRE(re.exec(subject, assigned_matches, 0, &assigned) == shared_rc);
+}
+
+// Ownership only, and deliberately not inside the case above.
+//
+// That case SKIPs when pattern_has_jit() is false, which is right for what it tests and
+// wrong for this: move construction and move assignment have nothing to do with the JIT,
+// and a double free matters as much on a PCRE2 without JIT, or on a hardened runtime
+// where JIT allocation fails, as anywhere else. Inside that case this coverage would
+// skip on exactly those builds and still report green.
+TEST_CASE("RegexMatchContext move operations transfer ownership", "[libts][Regex][RegexMatchContext]")
+{
+ Regex re;
+
+ REQUIRE(re.compile("^(a+)(b+)$"));
+
+ std::string_view const subject{"aaabbb"};
+
+ auto matches_through = [&](RegexMatchContext const *ctx) {
+ RegexMatches m;
+ return re.exec(subject, m, 0, ctx);
+ };
+
+ // Baseline through a plain context, so a later mismatch means the move broke something
+ // rather than the pattern never having matched.
+ RegexMatchContext plain;
+ int const expected = matches_through(&plain);
+
+ REQUIRE(expected > 0);
+
+ SECTION("move construction")
+ {
+ RegexMatchContext donor;
+ RegexMatchContext const moved{std::move(donor)};
+
+ REQUIRE(matches_through(&moved) == expected);
+ // A double free aborts when donor and moved are both destroyed, so leaving this
+ // section is itself the second assertion.
+ }
+
+ SECTION("move assignment onto a destination that already holds a context")
+ {
+ // On an ordinary build the assertions below do not detect a dropped
+ // pcre2_match_context_free(old): removing that call leaves this section passing,
+ // because a leak has no observer here. The oracle for the leak is LeakSanitizer,
+ // which AddressSanitizer turns on by default on Linux but not on macOS.
+ //
+ // The section still earns its place: it is what gives LSan a move assignment onto a
+ // live context to observe, which no other case in this file produces. Without it
+ // the leak is unreachable and therefore undetectable anywhere.
+ RegexMatchContext destination;
+
+ REQUIRE(matches_through(&destination) == expected);
+
+ RegexMatchContext donor;
+
+ destination = std::move(donor);
+
+ REQUIRE(matches_through(&destination) == expected);
+ }
+
+ SECTION("self move assignment")
+ {
+ RegexMatchContext ctx;
+
+ // A match limit of one fails this match, so the result after the move shows whether
+ // ctx still holds its own context. Without the this != &that guard the operator frees
+ // the context and then nulls itself, and the match runs on PCRE2's defaults instead.
+ ctx.set_match_limit(1);
+ REQUIRE(matches_through(&ctx) == PCRE2_ERROR_MATCHLIMIT);
+
+ auto &alias = ctx;
+
+ ctx = std::move(alias);
+
+ REQUIRE(matches_through(&ctx) == PCRE2_ERROR_MATCHLIMIT);
+ }
+}
+
+// A pattern that backtracks once per character must fail cleanly rather than run
+// the thread out of stack. A regex engine that recurses on the machine stack crashes
+// the server on a long enough subject; PCRE2 must report an error instead.
+TEST_CASE("Regex reports resource exhaustion rather than crashing", "[libts][Regex][limits]")
+{
+ // Only the JIT path has a bound to exhaust here. PCRE2's interpreter keeps its
+ // backtracking frames on the heap, so it matches this subject rather than running
+ // out of anything, and there is no resource error to assert.
+ char const *const pattern = R"(^/alpha/bravo/[?]((?!action=(newsfeed|calendar|contacts|notepad)).)*$)";
+ if (!pattern_has_jit(pattern)) {
+ SKIP("PCRE2 has no JIT for this pattern, so there is no stack bound to exhaust");
+ }
+
+ Regex re;
+ REQUIRE(re.compile(pattern));
+
+ // This pattern starts failing at roughly 43KiB of subject against a 1MiB JIT
+ // stack, measured identically on x86_64 and arm64. Smaller JIT frames get more
+ // subject out of the same stack and so push that threshold up; 256KiB keeps a
+ // six times margin against that. Do not trim this to just above 43KiB.
+ std::string subject{"/alpha/bravo/?"};
+ subject.append(256 * 1024, 'x');
+
+ RegexMatches matches;
+ int const rc = re.exec(subject, matches);
+ CAPTURE(rc);
+
+ // Reaching this line at all is the crash assertion.
+ REQUIRE(rc < 0);
+ REQUIRE(rc != RE_ERROR_NOMATCH);
+
+ // The paired positive control, and the only part of this case that can tell a
+ // 1MiB stack from PCRE2's 32KiB fallback: 256KiB overruns both bounds, so every
+ // assertion above also holds with the JIT stack callback deleted. 10,000 is seven
+ // times over the fallback's bound and four times under the 1MiB stack's. Neither
+ // bound pins 1MiB itself; the suite only requires a maximum in roughly
+ // [240KB, 6MB].
+ std::string smaller{"/alpha/bravo/?"};
+ smaller.append(10000, 'x');
+
+ RegexMatches small_matches;
+ int const small_rc = re.exec(smaller, small_matches);
+ CAPTURE(small_rc);
+
+ REQUIRE(small_rc > 0);
+}
+
+namespace
+{
+/** A reusable barrier, so every round starts with all threads alive and inside it. The
Review Comment:
Done in 4cadf1629693. The barrier class is gone (see the next thread), and the subject helper's comment is plain now:
https://github.com/apache/trafficserver/blob/f7ec6bc9058fcb051ddf0117ca10523a0fdc9d36/src/tsutil/unit_tests/test_Regex.cc#L1763-L1768
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Regex: test concurrent matching and JIT stack exhaustion [trafficserver]
Posted by "bryancall (via GitHub) via github" <gi...@trafficserver.apache.org>.
bryancall commented on code in PR #13683:
URL: https://github.com/apache/trafficserver/pull/13683#discussion_r4532236716
##########
src/tsutil/unit_tests/test_Regex.cc:
##########
@@ -1646,3 +1651,358 @@ TEST_CASE("Regex copies answer the same as their original", "[libts][Regex][copy
CHECK(original.exec(ordinary, original_ordinary) == copy.exec(ordinary, copy_ordinary));
}
}
+
+namespace
+{
+/** Does PCRE2 have JIT code for this pattern?
+ *
+ * The two tests below are about the JIT stack, and PCRE2 consults it only when it
+ * has JIT code to run. Without it both a blank context and the shared one take the
+ * interpreter and return the same answer, so the tests would pass whether or not
+ * the behaviour they describe is present. Ask PCRE2 rather than assume.
+ *
+ * This asks about one pattern, deliberately: PCRE2 declines some pattern items and
+ * leaves PCRE2_INFO_JITSIZE at zero on a build whose JIT is otherwise fine, so the
+ * library-wide probes (pcre2_config, PCRE2_JIT_TEST_ALLOC) answer a different
+ * question. Call it from the test thread only; Catch2 assertions are not thread safe.
+ */
+bool
+pattern_has_jit(char const *pattern)
+{
+ int errnum = 0;
+ PCRE2_SIZE erroffset = 0;
+ pcre2_code *code = pcre2_compile(reinterpret_cast<PCRE2_SPTR>(pattern), PCRE2_ZERO_TERMINATED, 0, &errnum, &erroffset, nullptr);
+
+ // A pattern that will not compile is a broken test, not a build without a JIT.
+ // Reporting it as "no JIT" would turn a typo into a silent skip.
+ REQUIRE(code != nullptr);
+
+ pcre2_jit_compile(code, PCRE2_JIT_COMPLETE);
+ size_t jit_size = 0;
+
+ pcre2_pattern_info(code, PCRE2_INFO_JITSIZE, &jit_size);
+ pcre2_code_free(code);
+ return jit_size > 0;
+}
+} // namespace
+
+// A caller-supplied RegexMatchContext must behave like the shared context that
+// Regex::exec uses when none is supplied. A context built from scratch drops
+// everything the shared one configures, the JIT stack callback included, and a
+// match through it then runs on PCRE2's fallback 32KiB JIT stack instead of the
+// 1MiB one.
+TEST_CASE("RegexMatchContext matches the shared context", "[libts][Regex][RegexMatchContext]")
+{
+ // Quantified alternation of capture groups: every subject character pushes a
+ // backtracking frame, so the JIT stack size is what bounds this.
+ char const *const pattern = R"(^(?:(a)|(b))+$)";
+ if (!pattern_has_jit(pattern)) {
+ SKIP("PCRE2 has no JIT for this pattern, so the JIT stack is never consulted");
+ }
+
+ Regex re;
+ REQUIRE(re.compile(pattern));
+
+ // 40 bytes of JIT stack per subject character, so PCRE2's 32KiB fallback stops at
+ // 818 characters and the shared context's 1MiB stack at 26,213. 5,000 sits six
+ // times above the first and five times below the second, so a blank context and
+ // the shared one give different answers with margin either way.
+ std::string const subject(5000, 'a');
+
+ RegexMatches shared_matches;
+ RegexMatchContext match_context;
+ RegexMatches own_matches;
+
+ int const shared_rc = re.exec(subject, shared_matches);
+ int const own_rc = re.exec(subject, own_matches, 0, &match_context);
+ CAPTURE(shared_rc, own_rc);
+
+ REQUIRE(shared_rc > 0);
+ REQUIRE(own_rc == shared_rc);
+
+ // The copy paths must carry the inherited configuration too. Building a blank
+ // context in either of them is the same defect, and nothing else here notices.
+ RegexMatchContext const copied{match_context};
+ RegexMatchContext assigned;
+
+ assigned = match_context;
+
+ RegexMatches copied_matches;
+ RegexMatches assigned_matches;
+
+ REQUIRE(re.exec(subject, copied_matches, 0, &copied) == shared_rc);
+ REQUIRE(re.exec(subject, assigned_matches, 0, &assigned) == shared_rc);
+}
+
+// Ownership only, and deliberately not inside the case above.
+//
+// That case SKIPs when pattern_has_jit() is false, which is right for what it tests and
+// wrong for this: move construction and move assignment have nothing to do with the JIT,
+// and a double free matters as much on a PCRE2 without JIT, or on a hardened runtime
+// where JIT allocation fails, as anywhere else. Inside that case this coverage would
+// skip on exactly those builds and still report green.
+TEST_CASE("RegexMatchContext move operations transfer ownership", "[libts][Regex][RegexMatchContext]")
+{
+ Regex re;
+
+ REQUIRE(re.compile("^(a+)(b+)$"));
+
+ std::string_view const subject{"aaabbb"};
+
+ auto matches_through = [&](RegexMatchContext const *ctx) {
+ RegexMatches m;
+ return re.exec(subject, m, 0, ctx);
+ };
+
+ // Baseline through a plain context, so a later mismatch means the move broke something
+ // rather than the pattern never having matched.
+ RegexMatchContext plain;
+ int const expected = matches_through(&plain);
+
+ REQUIRE(expected > 0);
+
+ SECTION("move construction")
+ {
+ RegexMatchContext donor;
+ RegexMatchContext const moved{std::move(donor)};
+
+ REQUIRE(matches_through(&moved) == expected);
+ // A double free aborts when donor and moved are both destroyed, so leaving this
+ // section is itself the second assertion.
+ }
+
+ SECTION("move assignment onto a destination that already holds a context")
+ {
+ // On an ordinary build the assertions below do not detect a dropped
+ // pcre2_match_context_free(old): removing that call leaves this section passing,
+ // because a leak has no observer here. The oracle for the leak is LeakSanitizer,
+ // which AddressSanitizer turns on by default on Linux but not on macOS.
+ //
+ // The section still earns its place: it is what gives LSan a move assignment onto a
+ // live context to observe, which no other case in this file produces. Without it
+ // the leak is unreachable and therefore undetectable anywhere.
+ RegexMatchContext destination;
+
+ REQUIRE(matches_through(&destination) == expected);
+
+ RegexMatchContext donor;
+
+ destination = std::move(donor);
+
+ REQUIRE(matches_through(&destination) == expected);
+ }
+
+ SECTION("self move assignment")
+ {
+ RegexMatchContext ctx;
+
+ // A match limit of one fails this match, so the result after the move shows whether
+ // ctx still holds its own context. Without the this != &that guard the operator frees
+ // the context and then nulls itself, and the match runs on PCRE2's defaults instead.
+ ctx.set_match_limit(1);
+ REQUIRE(matches_through(&ctx) == PCRE2_ERROR_MATCHLIMIT);
+
+ auto &alias = ctx;
+
+ ctx = std::move(alias);
+
+ REQUIRE(matches_through(&ctx) == PCRE2_ERROR_MATCHLIMIT);
+ }
+}
+
+// A pattern that backtracks once per character must fail cleanly rather than run
+// the thread out of stack. A regex engine that recurses on the machine stack crashes
+// the server on a long enough subject; PCRE2 must report an error instead.
+TEST_CASE("Regex reports resource exhaustion rather than crashing", "[libts][Regex][limits]")
+{
+ // Only the JIT path has a bound to exhaust here. PCRE2's interpreter keeps its
+ // backtracking frames on the heap, so it matches this subject rather than running
+ // out of anything, and there is no resource error to assert.
+ char const *const pattern = R"(^/alpha/bravo/[?]((?!action=(newsfeed|calendar|contacts|notepad)).)*$)";
+ if (!pattern_has_jit(pattern)) {
+ SKIP("PCRE2 has no JIT for this pattern, so there is no stack bound to exhaust");
+ }
+
+ Regex re;
+ REQUIRE(re.compile(pattern));
+
+ // This pattern starts failing at roughly 43KiB of subject against a 1MiB JIT
+ // stack, measured identically on x86_64 and arm64. Smaller JIT frames get more
+ // subject out of the same stack and so push that threshold up; 256KiB keeps a
+ // six times margin against that. Do not trim this to just above 43KiB.
+ std::string subject{"/alpha/bravo/?"};
+ subject.append(256 * 1024, 'x');
+
+ RegexMatches matches;
+ int const rc = re.exec(subject, matches);
+ CAPTURE(rc);
+
+ // Reaching this line at all is the crash assertion.
+ REQUIRE(rc < 0);
+ REQUIRE(rc != RE_ERROR_NOMATCH);
+
+ // The paired positive control, and the only part of this case that can tell a
+ // 1MiB stack from PCRE2's 32KiB fallback: 256KiB overruns both bounds, so every
+ // assertion above also holds with the JIT stack callback deleted. 10,000 is seven
+ // times over the fallback's bound and four times under the 1MiB stack's. Neither
+ // bound pins 1MiB itself; the suite only requires a maximum in roughly
+ // [240KB, 6MB].
+ std::string smaller{"/alpha/bravo/?"};
+ smaller.append(10000, 'x');
+
+ RegexMatches small_matches;
+ int const small_rc = re.exec(smaller, small_matches);
+ CAPTURE(small_rc);
+
+ REQUIRE(small_rc > 0);
+}
+
+namespace
+{
+/** A reusable barrier, so every round starts with all threads alive and inside it. The
+ * matches of one round then start together on every thread. std::barrier would say this
+ * directly; it is hand-rolled here to match notstd::barrier in benchmark_LogObject.cc.
Review Comment:
std::barrier is reusable from one phase to the next, so it's the right tool here. I switched to it in 4cadf1629693 and deleted the hand-rolled class:
https://github.com/apache/trafficserver/blob/f7ec6bc9058fcb051ddf0117ca10523a0fdc9d36/src/tsutil/unit_tests/test_Regex.cc#L1858
The log benchmark's notstd::barrier came in with #9096 in September 2022, when the tree still built with -std=c++17, so std::barrier wasn't available to it:
https://github.com/apache/trafficserver/blob/6c600a3f6432864a0bec463de6183db32f887085/configure.ac#L677
It's also one-shot, so it couldn't be reused here. Its count goes to zero once and never resets, and a second crossing leaves both threads blocked for good. std::barrier builds and runs with GCC 11 from the Rocky 8 CI image, GCC 16, and Apple clang 21.
The barrier does earn its place. I broke the code on purpose so every thread shares one JIT stack: the test failed 300 of 300 runs with the barrier and 276 of 300 without it. Those numbers are in the 4cadf1629693 commit message.
##########
src/tsutil/unit_tests/test_Regex.cc:
##########
@@ -1646,3 +1651,358 @@ TEST_CASE("Regex copies answer the same as their original", "[libts][Regex][copy
CHECK(original.exec(ordinary, original_ordinary) == copy.exec(ordinary, copy_ordinary));
}
}
+
+namespace
+{
+/** Does PCRE2 have JIT code for this pattern?
+ *
+ * The two tests below are about the JIT stack, and PCRE2 consults it only when it
+ * has JIT code to run. Without it both a blank context and the shared one take the
+ * interpreter and return the same answer, so the tests would pass whether or not
+ * the behaviour they describe is present. Ask PCRE2 rather than assume.
+ *
+ * This asks about one pattern, deliberately: PCRE2 declines some pattern items and
+ * leaves PCRE2_INFO_JITSIZE at zero on a build whose JIT is otherwise fine, so the
+ * library-wide probes (pcre2_config, PCRE2_JIT_TEST_ALLOC) answer a different
+ * question. Call it from the test thread only; Catch2 assertions are not thread safe.
+ */
+bool
+pattern_has_jit(char const *pattern)
+{
+ int errnum = 0;
+ PCRE2_SIZE erroffset = 0;
+ pcre2_code *code = pcre2_compile(reinterpret_cast<PCRE2_SPTR>(pattern), PCRE2_ZERO_TERMINATED, 0, &errnum, &erroffset, nullptr);
+
+ // A pattern that will not compile is a broken test, not a build without a JIT.
+ // Reporting it as "no JIT" would turn a typo into a silent skip.
+ REQUIRE(code != nullptr);
+
+ pcre2_jit_compile(code, PCRE2_JIT_COMPLETE);
+ size_t jit_size = 0;
+
+ pcre2_pattern_info(code, PCRE2_INFO_JITSIZE, &jit_size);
+ pcre2_code_free(code);
+ return jit_size > 0;
+}
+} // namespace
+
+// A caller-supplied RegexMatchContext must behave like the shared context that
+// Regex::exec uses when none is supplied. A context built from scratch drops
+// everything the shared one configures, the JIT stack callback included, and a
+// match through it then runs on PCRE2's fallback 32KiB JIT stack instead of the
+// 1MiB one.
+TEST_CASE("RegexMatchContext matches the shared context", "[libts][Regex][RegexMatchContext]")
+{
+ // Quantified alternation of capture groups: every subject character pushes a
+ // backtracking frame, so the JIT stack size is what bounds this.
+ char const *const pattern = R"(^(?:(a)|(b))+$)";
+ if (!pattern_has_jit(pattern)) {
+ SKIP("PCRE2 has no JIT for this pattern, so the JIT stack is never consulted");
+ }
+
+ Regex re;
+ REQUIRE(re.compile(pattern));
+
+ // 40 bytes of JIT stack per subject character, so PCRE2's 32KiB fallback stops at
+ // 818 characters and the shared context's 1MiB stack at 26,213. 5,000 sits six
+ // times above the first and five times below the second, so a blank context and
+ // the shared one give different answers with margin either way.
+ std::string const subject(5000, 'a');
+
+ RegexMatches shared_matches;
+ RegexMatchContext match_context;
+ RegexMatches own_matches;
+
+ int const shared_rc = re.exec(subject, shared_matches);
+ int const own_rc = re.exec(subject, own_matches, 0, &match_context);
+ CAPTURE(shared_rc, own_rc);
+
+ REQUIRE(shared_rc > 0);
+ REQUIRE(own_rc == shared_rc);
+
+ // The copy paths must carry the inherited configuration too. Building a blank
+ // context in either of them is the same defect, and nothing else here notices.
+ RegexMatchContext const copied{match_context};
+ RegexMatchContext assigned;
+
+ assigned = match_context;
+
+ RegexMatches copied_matches;
+ RegexMatches assigned_matches;
+
+ REQUIRE(re.exec(subject, copied_matches, 0, &copied) == shared_rc);
+ REQUIRE(re.exec(subject, assigned_matches, 0, &assigned) == shared_rc);
+}
+
+// Ownership only, and deliberately not inside the case above.
+//
+// That case SKIPs when pattern_has_jit() is false, which is right for what it tests and
+// wrong for this: move construction and move assignment have nothing to do with the JIT,
+// and a double free matters as much on a PCRE2 without JIT, or on a hardened runtime
+// where JIT allocation fails, as anywhere else. Inside that case this coverage would
+// skip on exactly those builds and still report green.
+TEST_CASE("RegexMatchContext move operations transfer ownership", "[libts][Regex][RegexMatchContext]")
+{
+ Regex re;
+
+ REQUIRE(re.compile("^(a+)(b+)$"));
+
+ std::string_view const subject{"aaabbb"};
+
+ auto matches_through = [&](RegexMatchContext const *ctx) {
+ RegexMatches m;
+ return re.exec(subject, m, 0, ctx);
+ };
+
+ // Baseline through a plain context, so a later mismatch means the move broke something
+ // rather than the pattern never having matched.
+ RegexMatchContext plain;
+ int const expected = matches_through(&plain);
+
+ REQUIRE(expected > 0);
+
+ SECTION("move construction")
+ {
+ RegexMatchContext donor;
+ RegexMatchContext const moved{std::move(donor)};
+
+ REQUIRE(matches_through(&moved) == expected);
+ // A double free aborts when donor and moved are both destroyed, so leaving this
+ // section is itself the second assertion.
+ }
+
+ SECTION("move assignment onto a destination that already holds a context")
+ {
+ // On an ordinary build the assertions below do not detect a dropped
+ // pcre2_match_context_free(old): removing that call leaves this section passing,
+ // because a leak has no observer here. The oracle for the leak is LeakSanitizer,
+ // which AddressSanitizer turns on by default on Linux but not on macOS.
+ //
+ // The section still earns its place: it is what gives LSan a move assignment onto a
+ // live context to observe, which no other case in this file produces. Without it
+ // the leak is unreachable and therefore undetectable anywhere.
+ RegexMatchContext destination;
+
+ REQUIRE(matches_through(&destination) == expected);
+
+ RegexMatchContext donor;
+
+ destination = std::move(donor);
+
+ REQUIRE(matches_through(&destination) == expected);
+ }
+
+ SECTION("self move assignment")
+ {
+ RegexMatchContext ctx;
+
+ // A match limit of one fails this match, so the result after the move shows whether
+ // ctx still holds its own context. Without the this != &that guard the operator frees
+ // the context and then nulls itself, and the match runs on PCRE2's defaults instead.
+ ctx.set_match_limit(1);
+ REQUIRE(matches_through(&ctx) == PCRE2_ERROR_MATCHLIMIT);
+
+ auto &alias = ctx;
+
+ ctx = std::move(alias);
+
+ REQUIRE(matches_through(&ctx) == PCRE2_ERROR_MATCHLIMIT);
+ }
+}
+
+// A pattern that backtracks once per character must fail cleanly rather than run
+// the thread out of stack. A regex engine that recurses on the machine stack crashes
+// the server on a long enough subject; PCRE2 must report an error instead.
+TEST_CASE("Regex reports resource exhaustion rather than crashing", "[libts][Regex][limits]")
+{
+ // Only the JIT path has a bound to exhaust here. PCRE2's interpreter keeps its
+ // backtracking frames on the heap, so it matches this subject rather than running
+ // out of anything, and there is no resource error to assert.
+ char const *const pattern = R"(^/alpha/bravo/[?]((?!action=(newsfeed|calendar|contacts|notepad)).)*$)";
+ if (!pattern_has_jit(pattern)) {
+ SKIP("PCRE2 has no JIT for this pattern, so there is no stack bound to exhaust");
+ }
+
+ Regex re;
+ REQUIRE(re.compile(pattern));
+
+ // This pattern starts failing at roughly 43KiB of subject against a 1MiB JIT
+ // stack, measured identically on x86_64 and arm64. Smaller JIT frames get more
+ // subject out of the same stack and so push that threshold up; 256KiB keeps a
+ // six times margin against that. Do not trim this to just above 43KiB.
+ std::string subject{"/alpha/bravo/?"};
+ subject.append(256 * 1024, 'x');
+
+ RegexMatches matches;
+ int const rc = re.exec(subject, matches);
+ CAPTURE(rc);
+
+ // Reaching this line at all is the crash assertion.
+ REQUIRE(rc < 0);
+ REQUIRE(rc != RE_ERROR_NOMATCH);
+
+ // The paired positive control, and the only part of this case that can tell a
+ // 1MiB stack from PCRE2's 32KiB fallback: 256KiB overruns both bounds, so every
+ // assertion above also holds with the JIT stack callback deleted. 10,000 is seven
+ // times over the fallback's bound and four times under the 1MiB stack's. Neither
+ // bound pins 1MiB itself; the suite only requires a maximum in roughly
+ // [240KB, 6MB].
+ std::string smaller{"/alpha/bravo/?"};
+ smaller.append(10000, 'x');
+
+ RegexMatches small_matches;
+ int const small_rc = re.exec(smaller, small_matches);
+ CAPTURE(small_rc);
+
+ REQUIRE(small_rc > 0);
+}
+
+namespace
+{
+/** A reusable barrier, so every round starts with all threads alive and inside it. The
+ * matches of one round then start together on every thread. std::barrier would say this
+ * directly; it is hand-rolled here to match notstd::barrier in benchmark_LogObject.cc.
+ *
+ * Each completed round advances the generation, which is what releases the waiters, so
+ * the barrier can be crossed any number of times. Exactly `expected` threads must call
+ * arrive_and_wait() every round or the rest block forever, and the barrier must outlive
+ * all of them, since destroying a condition_variable with waiters on it is undefined.
+ */
+class RoundBarrier
Review Comment:
Switching to std::barrier in 4cadf1629693 makes this moot for the test, but I did try it. The obvious semaphore version counts arrivals, and the last thread releases N-1 permits. Nothing ties a permit to a round, so a thread released early can loop around and take a permit meant for a thread still waiting in the previous round. At 8 threads and 2000 rounds the first run hung. With a timeout on the acquire, every run had 7 to 9 waits that never got a permit, and each of those is a hang without the timeout. A correct version needs a second turnstile, which is more code than the mutex version was.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Regex: hold the JIT stack in a pthread key and inherit the shared match context [trafficserver]
Posted by "bryancall (via GitHub)" <gi...@apache.org>.
bryancall commented on code in PR #13683:
URL: https://github.com/apache/trafficserver/pull/13683#discussion_r4087655608
##########
src/tsutil/Regex.cc:
##########
@@ -278,22 +351,61 @@ RegexMatchContext::operator=(RegexMatchContext const &other)
{
if (&other != this) {
auto ptr = _MatchContext::get(other._match_context);
- if (nullptr != ptr) {
- pcre2_match_context *const ctx = pcre2_match_context_copy(ptr);
- _MatchContext::set(_match_context, ctx);
- } else {
- _MatchContext::set(_match_context, nullptr);
+
+ // Take the copy before releasing what this object already holds, so a failing
+ // copy leaves it holding its old context rather than a freed one. Releasing it
+ // is what this operator used to omit, and every assignment leaked one context.
+ pcre2_match_context *const ctx = nullptr != ptr ? pcre2_match_context_copy(ptr) : nullptr;
+
+ // pcre2_match_context_copy() returns nullptr when it cannot allocate. Assigning it
+ // anyway would free the old context and leave this object empty, which is the very
+ // thing the ordering above exists to prevent, so leave the object untouched instead.
+ if (nullptr != ptr && nullptr == ctx) {
+ return *this;
}
+
+ pcre2_match_context *const old = _MatchContext::get(_match_context);
+
+ _MatchContext::set(_match_context, ctx);
+ if (old != nullptr) {
+ pcre2_match_context_free(old);
+ }
+ }
+ return *this;
+}
+
+//----------------------------------------------------------------------------
+RegexMatchContext::RegexMatchContext(RegexMatchContext &&that) noexcept
+{
+ // Through the typed accessors rather than std::exchange on the raw member: _ptr is
+ // void *, and void * does not implicitly convert to pcre2_match_context *, which is
+ // what set() takes.
+ _MatchContext::set(_match_context, _MatchContext::get(that._match_context));
+ _MatchContext::set(that._match_context, nullptr);
+}
+
+//----------------------------------------------------------------------------
+RegexMatchContext &
+RegexMatchContext::operator=(RegexMatchContext &&that) noexcept
+{
+ if (this != &that) {
+ if (auto *const old = _MatchContext::get(_match_context); old != nullptr) {
+ pcre2_match_context_free(old);
+ }
+ _MatchContext::set(_match_context, _MatchContext::get(that._match_context));
+ _MatchContext::set(that._match_context, nullptr);
}
return *this;
}
//----------------------------------------------------------------------------
RegexMatchContext::~RegexMatchContext()
{
- auto ptr = _MatchContext::get(_match_context);
- debug_assert_message(ptr, "Failed to get the match context");
- if (ptr != nullptr) {
+ // No assert that the pointer is set. Null is now a legitimate state: a moved-from
+ // object holds nothing, and asserting here would abort a debug build on the first
+ // destruction of one. Before the move operations existed the only way to reach this
+ // with null was a failed construction, which is why the assert was reasonable then.
Review Comment:
Agreed. Cut to one line that describes the current state, "A moved-from object holds no context," in 584075033f27.
##########
src/tsutil/Regex.cc:
##########
@@ -79,6 +80,67 @@ my_free(void *ptr, void * /*caller*/)
free(ptr);
}
+//----------------------------------------------------------------------------
+// One match context is shared by every thread that matches through it, and PCRE2
+// requires a distinct JIT stack per thread, so the stack comes from a callback
+// invoked at match time rather than a pointer baked in when the context is built.
+//
+// The per thread stack is held in a pthread key rather than a thread_local. A
+// thread_local with a destructor registers it through __cxa_thread_atexit, which
+// takes the dynamic loader lock; doing that from a match would invert lock order
+// against a dlopen caller running a plugin's static initialization. See the same
+// hazard described at Diags::tag_activated. A pthread key registers its destructor
+// once, at key creation, and never from the matching path.
+pthread_key_t jit_stack_key;
+bool jit_stack_key_valid = false;
+pthread_once_t jit_stack_key_once = PTHREAD_ONCE_INIT;
Review Comment:
They already do. They sit inside the unnamed namespace that opens at line 68 and closes at line 198, which is too far above this hunk to show in the diff. `nm` on the built object lists `jit_stack_key`, `jit_stack_key_valid` and `jit_stack_key_once` as local symbols under `(anonymous namespace)`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Regex: hold the JIT stack in a pthread key and inherit the shared match context [trafficserver]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #13683:
URL: https://github.com/apache/trafficserver/pull/13683#discussion_r4042118960
##########
src/tsutil/Regex.cc:
##########
@@ -257,8 +310,20 @@ struct RegexMatchContext::_MatchContext {
//----------------------------------------------------------------------------
RegexMatchContext::RegexMatchContext()
{
- auto ctx = pcre2_match_context_create(nullptr);
- debug_assert_message(ctx, "Failed to allocate custom pcre2 match context");
+ // Copy the shared context rather than building a blank one. A blank context
+ // silently drops everything the shared context configures, which is how this
+ // type came to run with PCRE2's fallback 32KiB JIT stack instead of the 1MiB
+ // one. Callers override only the fields they mean to.
+ //
+ // pcre2_match_context_copy dereferences its argument rather than returning null
+ // for one, and RegexContext's constructor does not check its allocations, so a
+ // shared context that failed to allocate would crash here. Fall back to a blank
+ // context, which is what this constructor built before and which no reader of
+ // _match_context dereferences unchecked.
+ auto *shared = RegexContext::get_instance()->get_match_context();
+ auto *ctx = shared != nullptr ? pcre2_match_context_copy(shared) : pcre2_match_context_create(nullptr);
Review Comment:
Calling `RegexContext::get_instance()` here initializes the `thread_local RegexContext`, whose non-trivial destructor registers via `__cxa_thread_atexit`. That is the same loader-lock path this change is trying to avoid, and it now occurs merely by constructing a caller context (including from plugin/static-initialization code), whereas the old constructor did not need it. Please copy from a process/shared context without initializing this thread-local, or otherwise defer the TLS construction, before relying on this standalone change.
##########
src/tsutil/unit_tests/test_Regex.cc:
##########
@@ -1147,3 +1152,260 @@ TEST_CASE("Regex copies answer the same as their original", "[libts][Regex][copy
CHECK(original.exec(ordinary, original_ordinary) == copy.exec(ordinary, copy_ordinary));
}
}
+
+namespace
+{
+/** Does PCRE2 have JIT code for this pattern?
+ *
+ * The two tests below are about the JIT stack, and PCRE2 consults it only when it
+ * has JIT code to run. Without it both a blank context and the shared one take the
+ * interpreter and return the same answer, so the tests would pass whether or not
+ * the behaviour they describe is present. Ask PCRE2 rather than assume.
+ *
+ * This asks about one pattern, deliberately: PCRE2 declines some pattern items and
+ * leaves PCRE2_INFO_JITSIZE at zero on a build whose JIT is otherwise fine, so the
+ * library-wide probes (pcre2_config, PCRE2_JIT_TEST_ALLOC) answer a different
+ * question. Call it from the test thread only; Catch2 assertions are not thread safe.
+ */
+bool
+pattern_has_jit(char const *pattern)
+{
+ int errnum = 0;
+ PCRE2_SIZE erroffset = 0;
+ pcre2_code *code = pcre2_compile(reinterpret_cast<PCRE2_SPTR>(pattern), PCRE2_ZERO_TERMINATED, 0, &errnum, &erroffset, nullptr);
+
+ // A pattern that will not compile is a broken test, not a build without a JIT.
+ // Reporting it as "no JIT" would turn a typo into a silent skip.
+ REQUIRE(code != nullptr);
+
+ pcre2_jit_compile(code, PCRE2_JIT_COMPLETE);
+ size_t jit_size = 0;
+
+ pcre2_pattern_info(code, PCRE2_INFO_JITSIZE, &jit_size);
+ pcre2_code_free(code);
+ return jit_size > 0;
+}
+} // namespace
+
+// A caller-supplied RegexMatchContext must behave like the shared context that
+// Regex::exec uses when none is supplied. A context built from scratch silently
+// drops everything the shared one configures, which is how regex_remap came to
+// run with PCRE2's fallback 32KiB JIT stack instead of the 1MiB one.
+TEST_CASE("RegexMatchContext matches the shared context", "[libts][Regex][RegexMatchContext]")
+{
+ // Quantified alternation of capture groups: every subject character pushes a
+ // backtracking frame, so the JIT stack size is what bounds this.
+ char const *const pattern = R"(^(?:(a)|(b))+$)";
+ if (!pattern_has_jit(pattern)) {
+ SKIP("PCRE2 has no JIT for this pattern, so the JIT stack is never consulted");
+ }
+
+ Regex re;
+ REQUIRE(re.compile(pattern));
+
+ // 40 bytes of JIT stack per subject character, so PCRE2's 32KiB fallback stops at
+ // 818 characters and the shared context's 1MiB stack at 26,213. 5,000 sits six
+ // times above the first and five times below the second, so a blank context and
+ // the shared one give different answers with margin either way.
+ std::string const subject(5000, 'a');
+
+ RegexMatches shared_matches;
+ RegexMatchContext match_context;
+ RegexMatches own_matches;
+
+ int const shared_rc = re.exec(subject, shared_matches);
+ int const own_rc = re.exec(subject, own_matches, 0, &match_context);
+ CAPTURE(shared_rc, own_rc);
+
+ REQUIRE(shared_rc > 0);
+ REQUIRE(own_rc == shared_rc);
+
+ // The copy paths must carry the inherited configuration too. Building a blank
+ // context in either of them is the same defect, and nothing else here notices.
+ RegexMatchContext const copied{match_context};
+ RegexMatchContext assigned;
+
+ assigned = match_context;
Review Comment:
Please fix `RegexMatchContext::operator=` before exercising this path: `assigned` already owns a PCRE2 match context, but the assignment overwrites `_match_context` without freeing the old one (src/tsutil/Regex.cc:342-351). This added test therefore leaks one context on every run and can fail LeakSanitizer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org