You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@maven.apache.org by "dfa1 (via GitHub)" <gi...@apache.org> on 2026/09/07 05:15:12 UTC

[I] 3.6.0 regression (bisected to #3374/commit 90529e9): forked JVM silently vanishes on Windows for tests using FFM upcalls from background threads [maven-surefire]

dfa1 opened a new issue, #3459:
URL: https://github.com/apache/maven-surefire/issues/3459

   
   ## Affected version
   3.6.0 (regression). Bisected to commit
   [90529e9](https://github.com/apache/maven-surefire/commit/90529e9fab1a2df720d9a88c1abb6c34a4ab5336)
   ("Use StackWalker in StackTraceProvider when available (Java 9+)", #3374).
   Last known good: 3.6.0-M1 and everything before that commit.
   
   ## Environment
   - OS: windows-latest GitHub Actions runner (Windows Server 2022/2025)
   - JDK: Zulu 25.0.4-7, x64
   - Build: Maven, `reuseForks=false`, `argLine` includes `--enable-native-access=ALL-UNNAMED`
   - Project: https://github.com/dfa1/rocksdb-ffm (Java FFM bindings for RocksDB's C API)
   
   ## Bug description
   
   Bumping `maven-surefire-plugin` from 3.5.6 to 3.6.0 makes three specific test
   classes deterministically kill their forked JVM on `windows-latest` only.
   macOS and Linux (x86_64 and aarch64) are unaffected. The same commit set with
   surefire pinned at 3.5.6 (or at 3.6.0-M1) passes on Windows every time.
   
   ```
   [ERROR] ExecutionException The forked VM terminated without properly saying goodbye. VM crash or System.exit called?
   [ERROR] Crashed tests:
     io.github.dfa1.rocksdbffm.CompactionFilterFactoryTest
     io.github.dfa1.rocksdbffm.CompactionFilterTest
     io.github.dfa1.rocksdbffm.EventNotifierTest
   ```
   
   This reproduced identically across many independent CI runs — always exactly
   these three classes, never a different subset.
   
   ### What we ruled out
   
   - **Not a classic native crash**: no `hs_err_pid*.log` is ever produced,
     despite surefire's own message suggesting a VM crash. If HotSpot's signal
     handler had caught a fault normally, that file would exist.
   - **Not the "Boot Manifest-JAR" classpath warning**: every single fork
     (150+, since `reuseForks=false`) logs
     `Boot Manifest-JAR contains absolute paths in classpath '...', 'other' has
     different root` to the `.dumpstream` file (a `Path.relativize()` failure
     across Windows drive roots, falling back to absolute paths in the manifest
     jar). This happens for both passing and crashing forks alike — harmless
     noise, unrelated to the real bug.
   - **Not simply "any test with a native upcall crashes"**: other tests in the
     same suite that register FFM upcalls but only ever invoke them
     synchronously on the JVM's own calling thread (e.g. a custom merge
     operator callback, or a test that calls `System.exit` from a JVM-owned
     thread) pass without issue.
   
   ### Bisection
   
   We built surefire from source at each candidate commit between
   `surefire-3.6.0-M1` and `surefire-3.6.0` (54 commits) and ran our suite
   against each, narrowing by binary search:
   
   | Round | Commit (index in the 54) | Result |
   |---|---|---|
   | baseline | 3.6.0-M1 | pass |
   | 1 | `37f0725` (26) | crash |
   | 2 | `381c404` (12) | crash |
   | 3 | `c633042` (5) | pass |
   | 4 | `90529e9` (8) | **crash** |
   | 5 | `e759c64` (6) | pass |
   | 6 (final) | `523e1ce` (7) | pass |
   
   Index 7 passing and index 8 crashing pins the boundary exactly: commit
   `90529e9` is the first bad commit in the range.
   
   The regression is isolated to that single commit,
   [90529e9](https://github.com/apache/maven-surefire/commit/90529e9fab1a2df720d9a88c1abb6c34a4ab5336)
   / PR [#3374](https://github.com/apache/maven-surefire/pull/3374), which
   switches `StackTraceProvider.getStack()` from `Thread.currentThread().getStackTrace()`
   to `java.lang.StackWalker.getInstance(options, depth).walk(...)` (falling back
   to the old path only if the `StackWalker` call throws).
   
   ### Suspected mechanism
   
   `CompactionFilter`/`CompactionFilterFactory`/`EventNotifier` register a
   native → Java callback that RocksDB invokes from its own background threads
   (the compaction thread pool, and flush/compaction event-listener threads) —
   not threads the JVM created. The callback path is a JDK 25
   `Linker.upcallStub()` entry point, which the JVM implements as a
   generated hidden class/`MethodHandle` trampoline. The first time such a
   background thread fires the upcall, the JVM attaches that foreign thread,
   and from the JVM's point of view its call stack starts at that synthetic
   hidden-class frame rather than a normal `Thread.run()`-rooted chain.
   
   `Thread.getStackTrace()` (still used as the fallback in the patched code)
   tolerates that stack shape fine — this project's whole suite passed under
   every surefire version before this commit. `StackWalker`, by contrast,
   does extra classification work on each frame specifically to hide
   reflection/lambda frames (per the PR's own code comment), which means it's
   doing more invasive introspection of frame metadata for exactly the kind of
   synthetic/hidden frame an FFM upcall trampoline produces. We suspect that
   introspection is hitting a rough edge in HotSpot's stack-walking internals
   for hidden-class/MethodHandle frames belonging to a JVM-attached (not
   JVM-created) thread, causing a hard, in-process JVM fault severe enough that
   the normal crash-dump path (`hs_err_pid*.log`) never runs.
   
   We have not attached a native debugger to confirm this at the HotSpot
   source level — it's our best explanation grounded in the diff, not a
   proven root cause.
   
   ### Workaround
   
   Setting `-Dsurefire.stackTraceMaxFrames=0` (or
   `<stackTraceMaxFrames>0</stackTraceMaxFrames>` in the plugin
   `<configuration>`) should avoid the crash entirely, based on reading the
   source: `StackTraceProvider.getStack()` returns `Collections.emptyList()`
   before ever reaching `StackWalkerStrategy`, so `StackWalker` is never
   invoked. This would let a consumer stay on 3.6.0 rather than pinning back
   to 3.5.6, at the documented cost of losing the console-output-to-test-class
   association this option controls — not a real fix, but a plausible interim
   mitigation.
   
   ## Reproduction
   
   Not yet minimized to a surefire-only reproducer (would require a small
   native library exercising a JDK 25 `Linker.upcallStub` invoked from a
   non-JVM-created thread, run under `StackTraceProvider.getStack()`'s new
   path). Happy to attempt one if useful — our current reproduction is the
   full `dfa1/rocksdb-ffm` CI matrix.
   
   ## What we'd like from surefire maintainers
   
   - Whether `StackWalker.getInstance(...).walk(...)` in `StackWalkerStrategy`
     is known to interact badly with frames belonging to a foreign/attached
     (not JVM-created) thread, particularly hidden-class or `MethodHandle`
     frames such as an FFM upcall trampoline.
   - Whether a dedicated opt-out (independent of `stackTraceMaxFrames`, which
     also disables the underlying feature) is worth adding, so a consumer can
     keep stack-trace-to-console-output association while forcing the legacy
     `Thread.getStackTrace()` path. As it stands, `StackWalkerStrategy.walk`'s
     `catch (ReflectiveOperationException | RuntimeException)` can't help here
     since a native JVM fault isn't a Java exception at all.
   
   Happy to run additional diagnostics on our CI if that helps narrow this
   down further.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@maven.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] 3.6.0 regression: forked JVM silently vanishes on Windows for tests using FFM upcalls from background threads [maven-surefire]

Posted by "dfa1 (via GitHub)" <gi...@apache.org>.
dfa1 commented on issue #3459:
URL: https://github.com/apache/maven-surefire/issues/3459#issuecomment-5969156104

   Thanks @olamy ! In the meantime, I notified of the JDK devs of the bug (but it is not confirmed yet)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@maven.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] 3.6.0 regression: forked JVM silently vanishes on Windows for tests using FFM upcalls from background threads [maven-surefire]

Posted by "dfa1 (via GitHub)" <gi...@apache.org>.
dfa1 commented on issue #3459:
URL: https://github.com/apache/maven-surefire/issues/3459#issuecomment-6412966878

   @andrei-ivanov https://bugs.java/bugdatabase/JDK-8392019


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@maven.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] 3.6.0 regression: forked JVM silently vanishes on Windows for tests using FFM upcalls from background threads [maven-surefire]

Posted by "dfa1 (via GitHub)" <gi...@apache.org>.
dfa1 commented on issue #3459:
URL: https://github.com/apache/maven-surefire/issues/3459#issuecomment-5565564394

   Confirmed empirically: applying `stackTraceMaxFrames=0` to the real, published 3.6.0 release (not just the source-built bisection commits) makes all 4 platforms in our CI matrix pass, including `windows-latest` with the three previously-crashing test classes (`CompactionFilterFactoryTest`, `CompactionFilterTest`, `EventNotifierTest`) all green.
   
   ```xml
   <plugin>
     <artifactId>maven-surefire-plugin</artifactId>
     <version>3.6.0</version>
     <configuration>
       <stackTraceMaxFrames>0</stackTraceMaxFrames>
     </configuration>
   </plugin>
   ```
   
   We've merged this into our project (dfa1/rocksdb-ffm#163) as the interim mitigation rather than pinning back to 3.5.6.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@maven.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] 3.6.0 regression: forked JVM silently vanishes on Windows for tests using FFM upcalls from background threads [maven-surefire]

Posted by "olamy (via GitHub)" <gi...@apache.org>.
olamy commented on issue #3459:
URL: https://github.com/apache/maven-surefire/issues/3459#issuecomment-5565724269

   Sounds more like an issue with the JDK not supporting a similar feature with `StackWalker` than a standard stack trace mechanism.
   Well, for the standard use case, `StackWalker` is just so so so faster.
   Maybe a way to disable `StackWalker` depending on a system property you could activate via `argLine`?
   What do you think?
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@maven.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] 3.6.0 regression: forked JVM silently vanishes on Windows for tests using FFM upcalls from background threads [maven-surefire]

Posted by "dfa1 (via GitHub)" <gi...@apache.org>.
dfa1 commented on issue #3459:
URL: https://github.com/apache/maven-surefire/issues/3459#issuecomment-5565939705

   yes, looks like a JDK issue as it is not working only on Windows. There is a clear workaround in the meantime.  


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@maven.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] 3.6.0 regression: forked JVM silently vanishes on Windows for tests using FFM upcalls from background threads [maven-surefire]

Posted by "andrei-ivanov (via GitHub)" <gi...@apache.org>.
andrei-ivanov commented on issue #3459:
URL: https://github.com/apache/maven-surefire/issues/3459#issuecomment-6410235407

   @dfa1 do you have a link?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@maven.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] 3.6.0 regression: forked JVM silently vanishes on Windows for tests using FFM upcalls from background threads [maven-surefire]

Posted by "olamy (via GitHub)" <gi...@apache.org>.
olamy closed issue #3459: 3.6.0 regression: forked JVM silently vanishes on Windows for tests using FFM upcalls from background threads
URL: https://github.com/apache/maven-surefire/issues/3459


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@maven.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] 3.6.0 regression: forked JVM silently vanishes on Windows for tests using FFM upcalls from background threads [maven-surefire]

Posted by "dfa1 (via GitHub) via issues" <is...@maven.apache.org>.
dfa1 commented on issue #3459:
URL: https://github.com/apache/maven-surefire/issues/3459#issuecomment-6625577225

   @andrei-ivanov so after another check, the upstream is closed again as not an issue. I think the current workaround in surefire is fine but I'm really wondering why the same setup is not causing issues on Linux/MacOS


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@maven.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] 3.6.0 regression: forked JVM silently vanishes on Windows for tests using FFM upcalls from background threads [maven-surefire]

Posted by "andrei-ivanov (via GitHub)" <gi...@apache.org>.
andrei-ivanov commented on issue #3459:
URL: https://github.com/apache/maven-surefire/issues/3459#issuecomment-6482373893

   @dfa1 Thanks, I see it was rejected due to the lack of requested details 😟


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@maven.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org