You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ignite.apache.org by "Denis Chudov (Jira)" <ji...@apache.org> on 2025/12/31 11:20:00 UTC
[jira] [Created] (IGNITE-27493) Test timeout may not work due to FJP await in cluster shutdown
Denis Chudov created IGNITE-27493:
-------------------------------------
Summary: Test timeout may not work due to FJP await in cluster shutdown
Key: IGNITE-27493
URL: https://issues.apache.org/jira/browse/IGNITE-27493
Project: Ignite
Issue Type: Bug
Reporter: Denis Chudov
Sometimes we see test suite timeouts like this:
[https://ci.ignite.apache.org/buildConfiguration/ApacheIgnite3x_Test_IntegrationTests_SystemDisasterRecovery/9876671]
Full log is attached.
There is stack trace in thread dump:
{code:java}
"Test worker" #1 prio=5 os_prio=0 cpu=19887.94ms elapsed=3540.59s tid=0x00007fab3002f000 nid=0x4e547 in Object.wait() [0x00007fab363fa000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(java.base@11.0.28/Native Method)
- waiting on <no object reference available>
at java.util.concurrent.ForkJoinTask.externalAwaitDone(java.base@11.0.28/ForkJoinTask.java:330)
- waiting to re-lock in wait() <0x000000072881e770> (a java.util.stream.ForEachOps$ForEachTask)
at java.util.concurrent.ForkJoinTask.doInvoke(java.base@11.0.28/ForkJoinTask.java:412)
at java.util.concurrent.ForkJoinTask.invoke(java.base@11.0.28/ForkJoinTask.java:736)
at java.util.stream.ForEachOps$ForEachOp.evaluateParallel(java.base@11.0.28/ForEachOps.java:159)
at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(java.base@11.0.28/ForEachOps.java:173)
at java.util.stream.AbstractPipeline.evaluate(java.base@11.0.28/AbstractPipeline.java:233)
at java.util.stream.ReferencePipeline.forEach(java.base@11.0.28/ReferencePipeline.java:497)
at org.apache.ignite.internal.Cluster.stopNodes(Cluster.java:690)
at org.apache.ignite.internal.Cluster.shutdown(Cluster.java:672)
at org.apache.ignite.internal.ClusterPerTestIntegrationTest.stopCluster(ClusterPerTestIntegrationTest.java:89)
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(java.base@11.0.28/Native Method)
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(java.base@11.0.28/NativeMethodAccessorImpl.java:62)
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@11.0.28/DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(java.base@11.0.28/Method.java:566)
at org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:786)
at org.junit.platform.commons.support.ReflectionSupport.invokeMethod(ReflectionSupport.java:514)
at org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
at org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
at org.junit.jupiter.engine.extension.SameThreadTimeoutInvocation.proceed(SameThreadTimeoutInvocation.java:49)
at org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:161)
at org.junit.jupiter.engine.extension.TimeoutExtension.interceptLifecycleMethod(TimeoutExtension.java:133)
at org.junit.jupiter.engine.extension.TimeoutExtension.interceptAfterEachMethod(TimeoutExtension.java:115)
at org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor$$Lambda$5077/0x0000000800d93440.apply(Unknown Source)
at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(InterceptingExecutableInvoker.java:112)
at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker$ReflectiveInterceptorCall$$Lambda$277/0x00000008001da840.apply(Unknown Source)
at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.lambda$invoke$0(InterceptingExecutableInvoker.java:94)
at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker$$Lambda$505/0x000000080022dc40.apply(Unknown Source)
at org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
at org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64)
at org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45)
at org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37)
at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.invoke(InterceptingExecutableInvoker.java:93)
at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.invoke(InterceptingExecutableInvoker.java:87)
at org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor.invokeMethodInExtensionContext(ClassBasedTestDescriptor.java:547)
at org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor.lambda$synthesizeAfterEachMethodAdapter$21(ClassBasedTestDescriptor.java:537)
at org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor$$Lambda$453/0x0000000800220840.invokeAfterEachMethod(Unknown Source)
at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeAfterEachMethods$6(TestMethodTestDescriptor.java:246)
at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor$$Lambda$5075/0x0000000800d86440.invoke(Unknown Source)
at org.junit.jupiter.engine.descriptor.CallbackSupport.lambda$invokeAfterCallbacks$1(CallbackSupport.java:49)
at org.junit.jupiter.engine.descriptor.CallbackSupport$$Lambda$5076/0x0000000800d93040.execute(Unknown Source)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.jupiter.engine.descriptor.CallbackSupport.lambda$invokeAfterCallbacks$2(CallbackSupport.java:49)
at org.junit.jupiter.engine.descriptor.CallbackSupport$$Lambda$5074/0x0000000800d86040.accept(Unknown Source)
at org.junit.platform.commons.util.CollectionUtils.forEachInReverseOrder(CollectionUtils.java:262)
at org.junit.jupiter.engine.descriptor.CallbackSupport.invokeAfterCallbacks(CallbackSupport.java:48)
at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeAfterEachMethods(TestMethodTestDescriptor.java:244)
at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:163)
at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:70)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:157)
at org.junit.platform.engine.support.hierarchical.NodeTestTask$$Lambda$414/0x000000080020b040.execute(Unknown Source)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:147)
at org.junit.platform.engine.support.hierarchical.NodeTestTask$$Lambda$413/0x000000080020bc40.invoke(Unknown Source)
at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:145)
at org.junit.platform.engine.support.hierarchical.NodeTestTask$$Lambda$412/0x000000080020b840.execute(Unknown Source)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:144)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:101)
at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService$$Lambda$418/0x000000080020a040.accept(Unknown Source)
at java.util.ArrayList.forEach(java.base@11.0.28/ArrayList.java:1541)
at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:41)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:161)
at org.junit.platform.engine.support.hierarchical.NodeTestTask$$Lambda$414/0x000000080020b040.execute(Unknown Source)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:147)
at org.junit.platform.engine.support.hierarchical.NodeTestTask$$Lambda$413/0x000000080020bc40.invoke(Unknown Source)
at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:145)
at org.junit.platform.engine.support.hierarchical.NodeTestTask$$Lambda$412/0x000000080020b840.execute(Unknown Source)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:144)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:101)
at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService$$Lambda$418/0x000000080020a040.accept(Unknown Source)
at java.util.ArrayList.forEach(java.base@11.0.28/ArrayList.java:1541)
at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:41)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:161)
at org.junit.platform.engine.support.hierarchical.NodeTestTask$$Lambda$414/0x000000080020b040.execute(Unknown Source)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:147)
at org.junit.platform.engine.support.hierarchical.NodeTestTask$$Lambda$413/0x000000080020bc40.invoke(Unknown Source)
at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:145)
at org.junit.platform.engine.support.hierarchical.NodeTestTask$$Lambda$412/0x000000080020b840.execute(Unknown Source)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:144)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:101)
at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.submit(SameThreadHierarchicalTestExecutorService.java:35)
at org.junit.platform.engine.support.hierarchical.HierarchicalTestExecutor.execute(HierarchicalTestExecutor.java:57)
at org.junit.platform.engine.support.hierarchical.HierarchicalTestEngine.execute(HierarchicalTestEngine.java:54)
at org.junit.platform.launcher.core.EngineExecutionOrchestrator.executeEngine(EngineExecutionOrchestrator.java:230)
at org.junit.platform.launcher.core.EngineExecutionOrchestrator.failOrExecuteEngine(EngineExecutionOrchestrator.java:204)
at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:172)
at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:101)
at org.junit.platform.launcher.core.EngineExecutionOrchestrator.lambda$execute$0(EngineExecutionOrchestrator.java:64)
at org.junit.platform.launcher.core.EngineExecutionOrchestrator$$Lambda$349/0x00000008001ed040.accept(Unknown Source)
at org.junit.platform.launcher.core.EngineExecutionOrchestrator.withInterceptedStreams(EngineExecutionOrchestrator.java:150)
at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:63)
at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:109)
at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:91)
at org.junit.platform.launcher.core.DelegatingLauncher.execute(DelegatingLauncher.java:47)
at org.junit.platform.launcher.core.InterceptingLauncher.lambda$execute$1(InterceptingLauncher.java:39)
at org.junit.platform.launcher.core.InterceptingLauncher$$Lambda$88/0x000000080016e040.proceed(Unknown Source)
at org.junit.platform.launcher.core.ClasspathAlignmentCheckingLauncherInterceptor.intercept(ClasspathAlignmentCheckingLauncherInterceptor.java:25)
at org.junit.platform.launcher.core.InterceptingLauncher.execute(InterceptingLauncher.java:38)
at org.junit.platform.launcher.core.DelegatingLauncher.execute(DelegatingLauncher.java:47)
at org.gradle.api.internal.tasks.testing.junitplatform.JUnitPlatformTestClassProcessor$CollectAllTestClassesExecutor.processAllTestClasses(JUnitPlatformTestClassProcessor.java:119)
at org.gradle.api.internal.tasks.testing.junitplatform.JUnitPlatformTestClassProcessor$CollectAllTestClassesExecutor.access$000(JUnitPlatformTestClassProcessor.java:94)
at org.gradle.api.internal.tasks.testing.junitplatform.JUnitPlatformTestClassProcessor.stop(JUnitPlatformTestClassProcessor.java:89)
at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.stop(SuiteTestClassProcessor.java:62)
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(java.base@11.0.28/Native Method)
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(java.base@11.0.28/NativeMethodAccessorImpl.java:62)
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@11.0.28/DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(java.base@11.0.28/Method.java:566)
at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:33)
at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:94)
at com.sun.proxy.$Proxy2.stop(Unknown Source)
at org.gradle.api.internal.tasks.testing.worker.TestWorker$3.run(TestWorker.java:193)
at org.gradle.api.internal.tasks.testing.worker.TestWorker.executeAndMaintainThreadName(TestWorker.java:129)
at org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:100)
at org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:60)
at org.gradle.process.internal.worker.child.ActionExecutionWorker.execute(ActionExecutionWorker.java:56)
at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:113)
at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:65)
at worker.org.gradle.process.internal.worker.GradleWorkerMain.run(GradleWorkerMain.java:69)
at worker.org.gradle.process.internal.worker.GradleWorkerMain.main(GradleWorkerMain.java:74)
{code}
ClusterPerTestIntegrationTest#stopCluster is annotated with @Timeout which should interrupt the thread after timeout, see [https://docs.junit.org/5.12.0/user-guide/index.html]
But in stack trace we can see ForkJoinTask#externalAwaitDone which swallows InterruptedException and waits forever even if cluster shutdown can't be completed for some reason.
The actual reason of cluster shutdown failure is unclear in this case: thread dump of test jvm has few threads, probably some of them were interrupted for some reason, and no thread may fire an event that shutdown worker may wait for.
Proposal: make the test cluster shutdown limited by time using another way, or avoid using FJP in the test cluster shutdown code.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)