You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by "gianm (via GitHub)" <gi...@apache.org> on 2024/10/28 18:46:14 UTC
[I] Segfault in JDK 21 processing tests (druid)
gianm opened a new issue, #17429:
URL: https://github.com/apache/druid/issues/17429
We've been seeing a high degree of segfaults recently in the `processing` unit tests under JDK 21. Here's an example: https://github.com/apache/druid/actions/runs/11528594002/job/32166241444?pr=17414. The common thread is an error like this:
```
Current CompileTask:
C2:1780380 144967 4 org.apache.druid.query.filter.InDimFilter::optimizeLookup (744 bytes)
Stack: [0x00007f9c045fb000,0x00007f9c046fb000], sp=0x00007f9c046f6660, free space=1005k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V [libjvm.so+0xf15629] BoolNode::Ideal(PhaseGVN*, bool)+0x19
V [libjvm.so+0xd6258e] PhaseIterGVN::transform_old(Node*)+0x9e
V [libjvm.so+0x6ba360] Conv2BNode::Ideal(PhaseGVN*, bool)+0x180
V [libjvm.so+0xd6258e] PhaseIterGVN::transform_old(Node*)+0x9e
V [libjvm.so+0xd5e4a9] PhaseIterGVN::optimize()+0xf9
V [libjvm.so+0x6712cf] Compile::Optimize()+0xf9f
V [libjvm.so+0x672a06] Compile::Compile(ciEnv*, ciMethod*, int, Options, DirectiveSet*)+0xf26
V [libjvm.so+0x598acb] C2Compiler::compile_method(ciEnv*, ciMethod*, int, bool, DirectiveSet*)+0x18b
V [libjvm.so+0x678ee4] CompileBroker::invoke_compiler_on_method(CompileTask*)+0xca4
V [libjvm.so+0x67c0d8] CompileBroker::compiler_thread_loop()+0x6a8
V [libjvm.so+0x931f40] JavaThread::thread_main_inner()+0x1e0
V [libjvm.so+0xf86b88] Thread::call_run()+0xa8
V [libjvm.so+0xd103ba] thread_native_entry(Thread*)+0xda
```
Things that did _not_ help include:
- Using Corretto instead of Zulu (#17426)
- Switching off `jfr_profiler` (#17418).
- Updating various test/build dependencies (#17414).
Since this is in the C2 compiler, and only happens on JDK 21, it seems likely to be a JDK bug rather than something we're doing wrong.
@pranavbhole has raised a support case with Azul at https://support.azul.com/hc/en-us/requests/65354. We don't have any special support contract with them, so hopefully they respond out of the goodness of their hearts. Since this issue happens on Corretto too, it's possibly a broader OpenJDK issue, so we could also consider raising a bug with OpenJDK.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [I] Segfault in JDK 21 processing tests (druid)
Posted by "gianm (via GitHub)" <gi...@apache.org>.
gianm commented on issue #17429:
URL: https://github.com/apache/druid/issues/17429#issuecomment-2461347867
> Switching the JDK distribution to `temurin` in [#17449](https://github.com/apache/druid/pull/17449) seems to not have the segfaults seen in the JDK 21 tests, at least in the four repeated CI runs. Any thoughts on switching the distribution from `zulu` to `temurin`?
The logs in #17449 say the JDK version is `21.0.4+7`. Given we only started seeing this problem with Zulu in 21.0.5 (21.0.4 was fine), it's probably just working because the `temurin` JDK happens to be older right now. But that might change. I guess it will work for some time though, until the `temurin` build upgrades.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [I] Segfault in JDK 21 processing tests (druid)
Posted by "gianm (via GitHub)" <gi...@apache.org>.
gianm commented on issue #17429:
URL: https://github.com/apache/druid/issues/17429#issuecomment-2458285448
@kgyrtkirk shared a core dump and libs archive with me offline. I've attached them to the Azul support case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [I] Segfault in JDK 21 processing tests (druid)
Posted by "abhishekrb19 (via GitHub)" <gi...@apache.org>.
abhishekrb19 commented on issue #17429:
URL: https://github.com/apache/druid/issues/17429#issuecomment-2461323507
Switching the JDK distribution to `temurin` in [#17449](https://github.com/apache/druid/pull/17449) seems to not have the segfaults seen in the JDK 21 tests, at least in the four repeated CI runs. Any thoughts on switching the distribution from `zulu` to `temurin`? This is similar to [#17426](https://github.com/apache/druid/pull/17426), but the `temurin` build appears to help.
cc: @gianm @kgyrtkirk
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [I] Segfault in JDK 21 processing tests (druid)
Posted by "asdf2014 (via GitHub)" <gi...@apache.org>.
asdf2014 commented on issue #17429:
URL: https://github.com/apache/druid/issues/17429#issuecomment-2443049421
Hi @gianm , it appears that a similar issue has already been reported and resolved in [JDK-8323190](https://bugs.openjdk.org/browse/JDK-8323190). You may want to confirm this by checking the details in the issue report :smile:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [I] Segfault in JDK 21 processing tests (druid)
Posted by "cryptoe (via GitHub)" <gi...@apache.org>.
cryptoe commented on issue #17429:
URL: https://github.com/apache/druid/issues/17429#issuecomment-2662424700
@kgyrtkirk Could you please log the bug with different jdk vendors as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [I] Segfault in JDK 21 processing tests (druid)
Posted by "gianm (via GitHub)" <gi...@apache.org>.
gianm commented on issue #17429:
URL: https://github.com/apache/druid/issues/17429#issuecomment-2461353516
Trying to pin to JDK 21.0.4: #17458
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [I] Segfault in JDK 21 processing tests (druid)
Posted by "abhishekrb19 (via GitHub)" <gi...@apache.org>.
abhishekrb19 commented on issue #17429:
URL: https://github.com/apache/druid/issues/17429#issuecomment-2461416766
Ah got it, I didn't notice the difference in JDK update version with `temurin`. Pinning the version to JDK 21.0.4 makes sense.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [I] Segfault in JDK 21 processing tests (druid)
Posted by "gianm (via GitHub)" <gi...@apache.org>.
gianm commented on issue #17429:
URL: https://github.com/apache/druid/issues/17429#issuecomment-2461010050
Now a quidem test is failing with:
```
*** java.lang.instrument ASSERTION FAILED ***: "!errorOutstanding" with message transform method call failed at src/java.instrument/share/native/libinstrument/JPLISAgent.c line: 884
```
Maybe JDK 21 is too buggy and we should switch off its unit tests for now 😛
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [I] Segfault in JDK 21 processing tests (druid)
Posted by "gianm (via GitHub)" <gi...@apache.org>.
gianm commented on issue #17429:
URL: https://github.com/apache/druid/issues/17429#issuecomment-2460369664
> disabling C2 optimization for tests on 21 might also be a possible way out
This seems like the best temporary approach. I raised #17456 to attempt it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [I] Segfault in JDK 21 processing tests (druid)
Posted by "capistrant (via GitHub)" <gi...@apache.org>.
capistrant closed issue #17429: Segfault in JDK 21 processing tests
URL: https://github.com/apache/druid/issues/17429
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [I] Segfault in JDK 21 processing tests (druid)
Posted by "kgyrtkirk (via GitHub)" <gi...@apache.org>.
kgyrtkirk commented on issue #17429:
URL: https://github.com/apache/druid/issues/17429#issuecomment-2461587343
fyi: I've run locally temurin `21.0.5` - and it did produced the issue; so we won't be out of the water with it...I should have closed the PR around that time....
but I've encountered a `21.0.5` which did passed these tests: using the `openjdk-21-jdk` package shipped with `debian/trixie` I was not able to reproduce the issue (this doesn't necessary mean its not there; but there is a possibility that it was patched)
I think we should go to `21.0.4` until it gets fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [I] Segfault in JDK 21 processing tests (druid)
Posted by "pranavbhole (via GitHub)" <gi...@apache.org>.
pranavbhole commented on issue #17429:
URL: https://github.com/apache/druid/issues/17429#issuecomment-2445939102
Looks like we have setting in place to check if core.pid file dumped but unfortunately core dumps are not being generated and aborted due to memory limitations.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [I] Segfault in JDK 21 processing tests (druid)
Posted by "pranavbhole (via GitHub)" <gi...@apache.org>.
pranavbhole commented on issue #17429:
URL: https://github.com/apache/druid/issues/17429#issuecomment-2445901341
I received a response from Azul regarding the Zulu JDK crash. I informed them that the issue is consistently reproducible (still occurring with the latest master commit). Azul has requested a core dump as well as list of shared libraries to analyze the crash.
```
There is
# Core dump will be written. Default location: Core dumps may be processed with "/usr/share/apport/apport -p%p -s%s -c%c -d%d -P%P -u%u -g%g -- %E" (or dumping to /home/runner/work/druid/druid/processing/core.3533)
and
rlimit (soft/hard): STACK 16384k/infinity , CORE 0k/infinity , NPROC 63899/63899 , NOFILE 65536/65536 , AS infinity/infinity , CPU infinity/infinity , DATA infinity/infinity , FSIZE infinity/infinity , MEMLOCK 8192k/8192k
in hs_err file, that tells there might be a core dump in the system generated after crash happened.
In case you manage to locate core file please collect the shared libs as well.
You can use this command
grep '/.*\.so' hs_err_pid1913.log | awk '{ print $NF }' | uniq > libs.txt
Then run the following command on the resulting libs.txt file from above
tar -czf libs.tgz -T libs.txt
Upload core and libs.tgz to the file sharing resource and your convenience and share the link to download.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [I] Segfault in JDK 21 processing tests (druid)
Posted by "gianm (via GitHub)" <gi...@apache.org>.
gianm commented on issue #17429:
URL: https://github.com/apache/druid/issues/17429#issuecomment-2632894153
Reverted #17458 in #17694. It's possible this issue was fixed in 21.0.6 🤞
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [I] Segfault in JDK 21 processing tests (druid)
Posted by "abhishekrb19 (via GitHub)" <gi...@apache.org>.
abhishekrb19 commented on issue #17429:
URL: https://github.com/apache/druid/issues/17429#issuecomment-2967826998
With JDK 21.0.7, an indexing-service unit test crashed with a Segfault: https://github.com/apache/druid/pull/18128#issuecomment-2967825145
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [I] Segfault in JDK 21 processing tests (druid)
Posted by "kgyrtkirk (via GitHub)" <gi...@apache.org>.
kgyrtkirk commented on issue #17429:
URL: https://github.com/apache/druid/issues/17429#issuecomment-2456639199
this issue was blocking me for a few days now - so I was forced to take a look ;)
so far my observations are:
* the issue did happened even earlier with different GHA ubuntu images
* its nondeterministic; I've seen it happen with 2-3 tests; the most common victim is `AndFilterTest` ; but it could be `SelectorFilterTest` as well - so its not related to just one testclass
* I've blind-tuned the `jit` to be more agressive...right now I'm not sure if that even mattered
* I was able to repro it locally after I've switched to `OpenJDK Runtime Environment Zulu21.38+21-CA (build 21.0.5+11-LTS)`
* I was running `21.0.4` earlier which was unaffected with several attempts
* switching to `21.0.5` right away showed the issue; so it must be introduced in `21.0.5`
* the issue consistently happens with `org.apache.druid.query.filter.InDimFilter::optimizeLookup`
* my attempts to get a disassembly so far was unsuccessfull; I was bumping into that I'm not prepared for it as `hsdis-amd64.so` is missing
* all tests run in a single jvm -> jvm retains method invocation counts from earlier tests
* surefire test execution order is unspecified - which makes surefire run them in `filesystem` order (which becomes essentially a random in CI runs)
* right now the shortest running command which is still able to repro the issue is:
```
time mvn install -pl processing/ -Dtest=AndFilterTest,SuperSorterTest*,InFilterTests,*FilterTest -Pskip-static-checks
```
* even thru the test execution order is the same I've seen crashes in 3 different tests (`NotFilterTest` ; `AndFilterTest` ; `SelectorFilterTest`)
I think right now we have the following options:
* debug it further
* even if we uncover the underlying issue - a new jdk release will be needed to get rid of the problem ; which might take some time...so we will need an alternate fix until that will happen
* disable the affected tests: not really an option since the C2 optimization of `InDimFilter::optimizeLookup` may trigger the issue
* from the call stack it seem like its connected to the C2 compiler - so disabling C2 optimization for tests on 21 might also be a possible way out
* possibly force `21.0.4` to be used
* try to not reuse the jvm for subsequent test runs (`reuseforks`)
* other options?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [I] Segfault in JDK 21 processing tests (druid)
Posted by "gianm (via GitHub)" <gi...@apache.org>.
gianm commented on issue #17429:
URL: https://github.com/apache/druid/issues/17429#issuecomment-2462992828
"Fixed" for now in #17458. It's not really fixed for good, this is just a workaround. Hopefully the Azul folks come through for us. If not, it sounds like we'd be able to raise this issue with various other JDK suppliers, as we've seen it in others too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org