You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@maven.apache.org by "dfa1 (via GitHub)" <gi...@apache.org> on 2026/09/07 05:15:12 UTC
[I] 3.6.0 regression (bisected to #3374/commit 90529e9): forked JVM silently vanishes on Windows for tests using FFM upcalls from background threads [maven-surefire]
dfa1 opened a new issue, #3459:
URL: https://github.com/apache/maven-surefire/issues/3459
## Affected version
3.6.0 (regression). Bisected to commit
[90529e9](https://github.com/apache/maven-surefire/commit/90529e9fab1a2df720d9a88c1abb6c34a4ab5336)
("Use StackWalker in StackTraceProvider when available (Java 9+)", #3374).
Last known good: 3.6.0-M1 and everything before that commit.
## Environment
- OS: windows-latest GitHub Actions runner (Windows Server 2022/2025)
- JDK: Zulu 25.0.4-7, x64
- Build: Maven, `reuseForks=false`, `argLine` includes `--enable-native-access=ALL-UNNAMED`
- Project: https://github.com/dfa1/rocksdb-ffm (Java FFM bindings for RocksDB's C API)
## Bug description
Bumping `maven-surefire-plugin` from 3.5.6 to 3.6.0 makes three specific test
classes deterministically kill their forked JVM on `windows-latest` only.
macOS and Linux (x86_64 and aarch64) are unaffected. The same commit set with
surefire pinned at 3.5.6 (or at 3.6.0-M1) passes on Windows every time.
```
[ERROR] ExecutionException The forked VM terminated without properly saying goodbye. VM crash or System.exit called?
[ERROR] Crashed tests:
io.github.dfa1.rocksdbffm.CompactionFilterFactoryTest
io.github.dfa1.rocksdbffm.CompactionFilterTest
io.github.dfa1.rocksdbffm.EventNotifierTest
```
This reproduced identically across many independent CI runs — always exactly
these three classes, never a different subset.
### What we ruled out
- **Not a classic native crash**: no `hs_err_pid*.log` is ever produced,
despite surefire's own message suggesting a VM crash. If HotSpot's signal
handler had caught a fault normally, that file would exist.
- **Not the "Boot Manifest-JAR" classpath warning**: every single fork
(150+, since `reuseForks=false`) logs
`Boot Manifest-JAR contains absolute paths in classpath '...', 'other' has
different root` to the `.dumpstream` file (a `Path.relativize()` failure
across Windows drive roots, falling back to absolute paths in the manifest
jar). This happens for both passing and crashing forks alike — harmless
noise, unrelated to the real bug.
- **Not simply "any test with a native upcall crashes"**: other tests in the
same suite that register FFM upcalls but only ever invoke them
synchronously on the JVM's own calling thread (e.g. a custom merge
operator callback, or a test that calls `System.exit` from a JVM-owned
thread) pass without issue.
### Bisection
We built surefire from source at each candidate commit between
`surefire-3.6.0-M1` and `surefire-3.6.0` (54 commits) and ran our suite
against each, narrowing by binary search:
| Round | Commit (index in the 54) | Result |
|---|---|---|
| baseline | 3.6.0-M1 | pass |
| 1 | `37f0725` (26) | crash |
| 2 | `381c404` (12) | crash |
| 3 | `c633042` (5) | pass |
| 4 | `90529e9` (8) | **crash** |
| 5 | `e759c64` (6) | pass |
| 6 (final) | `523e1ce` (7) | pass |
Index 7 passing and index 8 crashing pins the boundary exactly: commit
`90529e9` is the first bad commit in the range.
The regression is isolated to that single commit,
[90529e9](https://github.com/apache/maven-surefire/commit/90529e9fab1a2df720d9a88c1abb6c34a4ab5336)
/ PR [#3374](https://github.com/apache/maven-surefire/pull/3374), which
switches `StackTraceProvider.getStack()` from `Thread.currentThread().getStackTrace()`
to `java.lang.StackWalker.getInstance(options, depth).walk(...)` (falling back
to the old path only if the `StackWalker` call throws).
### Suspected mechanism
`CompactionFilter`/`CompactionFilterFactory`/`EventNotifier` register a
native → Java callback that RocksDB invokes from its own background threads
(the compaction thread pool, and flush/compaction event-listener threads) —
not threads the JVM created. The callback path is a JDK 25
`Linker.upcallStub()` entry point, which the JVM implements as a
generated hidden class/`MethodHandle` trampoline. The first time such a
background thread fires the upcall, the JVM attaches that foreign thread,
and from the JVM's point of view its call stack starts at that synthetic
hidden-class frame rather than a normal `Thread.run()`-rooted chain.
`Thread.getStackTrace()` (still used as the fallback in the patched code)
tolerates that stack shape fine — this project's whole suite passed under
every surefire version before this commit. `StackWalker`, by contrast,
does extra classification work on each frame specifically to hide
reflection/lambda frames (per the PR's own code comment), which means it's
doing more invasive introspection of frame metadata for exactly the kind of
synthetic/hidden frame an FFM upcall trampoline produces. We suspect that
introspection is hitting a rough edge in HotSpot's stack-walking internals
for hidden-class/MethodHandle frames belonging to a JVM-attached (not
JVM-created) thread, causing a hard, in-process JVM fault severe enough that
the normal crash-dump path (`hs_err_pid*.log`) never runs.
We have not attached a native debugger to confirm this at the HotSpot
source level — it's our best explanation grounded in the diff, not a
proven root cause.
### Workaround
Setting `-Dsurefire.stackTraceMaxFrames=0` (or
`<stackTraceMaxFrames>0</stackTraceMaxFrames>` in the plugin
`<configuration>`) should avoid the crash entirely, based on reading the
source: `StackTraceProvider.getStack()` returns `Collections.emptyList()`
before ever reaching `StackWalkerStrategy`, so `StackWalker` is never
invoked. This would let a consumer stay on 3.6.0 rather than pinning back
to 3.5.6, at the documented cost of losing the console-output-to-test-class
association this option controls — not a real fix, but a plausible interim
mitigation.
## Reproduction
Not yet minimized to a surefire-only reproducer (would require a small
native library exercising a JDK 25 `Linker.upcallStub` invoked from a
non-JVM-created thread, run under `StackTraceProvider.getStack()`'s new
path). Happy to attempt one if useful — our current reproduction is the
full `dfa1/rocksdb-ffm` CI matrix.
## What we'd like from surefire maintainers
- Whether `StackWalker.getInstance(...).walk(...)` in `StackWalkerStrategy`
is known to interact badly with frames belonging to a foreign/attached
(not JVM-created) thread, particularly hidden-class or `MethodHandle`
frames such as an FFM upcall trampoline.
- Whether a dedicated opt-out (independent of `stackTraceMaxFrames`, which
also disables the underlying feature) is worth adding, so a consumer can
keep stack-trace-to-console-output association while forcing the legacy
`Thread.getStackTrace()` path. As it stands, `StackWalkerStrategy.walk`'s
`catch (ReflectiveOperationException | RuntimeException)` can't help here
since a native JVM fault isn't a Java exception at all.
Happy to run additional diagnostics on our CI if that helps narrow this
down further.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@maven.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] 3.6.0 regression: forked JVM silently vanishes on Windows for tests using FFM upcalls from background threads [maven-surefire]
Posted by "dfa1 (via GitHub)" <gi...@apache.org>.
dfa1 commented on issue #3459:
URL: https://github.com/apache/maven-surefire/issues/3459#issuecomment-5969156104
Thanks @olamy ! In the meantime, I notified of the JDK devs of the bug (but it is not confirmed yet)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@maven.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] 3.6.0 regression: forked JVM silently vanishes on Windows for tests using FFM upcalls from background threads [maven-surefire]
Posted by "dfa1 (via GitHub)" <gi...@apache.org>.
dfa1 commented on issue #3459:
URL: https://github.com/apache/maven-surefire/issues/3459#issuecomment-6412966878
@andrei-ivanov https://bugs.java/bugdatabase/JDK-8392019
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@maven.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] 3.6.0 regression: forked JVM silently vanishes on Windows for tests using FFM upcalls from background threads [maven-surefire]
Posted by "dfa1 (via GitHub)" <gi...@apache.org>.
dfa1 commented on issue #3459:
URL: https://github.com/apache/maven-surefire/issues/3459#issuecomment-5565564394
Confirmed empirically: applying `stackTraceMaxFrames=0` to the real, published 3.6.0 release (not just the source-built bisection commits) makes all 4 platforms in our CI matrix pass, including `windows-latest` with the three previously-crashing test classes (`CompactionFilterFactoryTest`, `CompactionFilterTest`, `EventNotifierTest`) all green.
```xml
<plugin>
<artifactId>maven-surefire-plugin</artifactId>
<version>3.6.0</version>
<configuration>
<stackTraceMaxFrames>0</stackTraceMaxFrames>
</configuration>
</plugin>
```
We've merged this into our project (dfa1/rocksdb-ffm#163) as the interim mitigation rather than pinning back to 3.5.6.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@maven.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] 3.6.0 regression: forked JVM silently vanishes on Windows for tests using FFM upcalls from background threads [maven-surefire]
Posted by "olamy (via GitHub)" <gi...@apache.org>.
olamy commented on issue #3459:
URL: https://github.com/apache/maven-surefire/issues/3459#issuecomment-5565724269
Sounds more like an issue with the JDK not supporting a similar feature with `StackWalker` than a standard stack trace mechanism.
Well, for the standard use case, `StackWalker` is just so so so faster.
Maybe a way to disable `StackWalker` depending on a system property you could activate via `argLine`?
What do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@maven.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] 3.6.0 regression: forked JVM silently vanishes on Windows for tests using FFM upcalls from background threads [maven-surefire]
Posted by "dfa1 (via GitHub)" <gi...@apache.org>.
dfa1 commented on issue #3459:
URL: https://github.com/apache/maven-surefire/issues/3459#issuecomment-5565939705
yes, looks like a JDK issue as it is not working only on Windows. There is a clear workaround in the meantime.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@maven.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] 3.6.0 regression: forked JVM silently vanishes on Windows for tests using FFM upcalls from background threads [maven-surefire]
Posted by "andrei-ivanov (via GitHub)" <gi...@apache.org>.
andrei-ivanov commented on issue #3459:
URL: https://github.com/apache/maven-surefire/issues/3459#issuecomment-6410235407
@dfa1 do you have a link?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@maven.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] 3.6.0 regression: forked JVM silently vanishes on Windows for tests using FFM upcalls from background threads [maven-surefire]
Posted by "olamy (via GitHub)" <gi...@apache.org>.
olamy closed issue #3459: 3.6.0 regression: forked JVM silently vanishes on Windows for tests using FFM upcalls from background threads
URL: https://github.com/apache/maven-surefire/issues/3459
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@maven.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] 3.6.0 regression: forked JVM silently vanishes on Windows for tests using FFM upcalls from background threads [maven-surefire]
Posted by "dfa1 (via GitHub) via issues" <is...@maven.apache.org>.
dfa1 commented on issue #3459:
URL: https://github.com/apache/maven-surefire/issues/3459#issuecomment-6625577225
@andrei-ivanov so after another check, the upstream is closed again as not an issue. I think the current workaround in surefire is fine but I'm really wondering why the same setup is not causing issues on Linux/MacOS
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@maven.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] 3.6.0 regression: forked JVM silently vanishes on Windows for tests using FFM upcalls from background threads [maven-surefire]
Posted by "andrei-ivanov (via GitHub)" <gi...@apache.org>.
andrei-ivanov commented on issue #3459:
URL: https://github.com/apache/maven-surefire/issues/3459#issuecomment-6482373893
@dfa1 Thanks, I see it was rejected due to the lack of requested details 😟
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@maven.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org