You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@uniffle.apache.org by "xianjingfeng (via GitHub)" <gi...@apache.org> on 2025/04/10 09:50:03 UTC
[I] [Flaky Test] org.apache.uniffle.test.ShuffleWithRssClientTest#testRetryAssgin [uniffle]
xianjingfeng opened a new issue, #2439:
URL: https://github.com/apache/uniffle/issues/2439
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
### Search before asking
- [x] I have searched in the [issues](https://github.com/apache/incubator-uniffle/issues?q=is%3Aissue) and found no similar issues.
### Describe the flaky test
```
Error: Tests run: 6, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 14.61 s <<< FAILURE! - in org.apache.uniffle.test.ShuffleWithRssClientTest
Error: testRetryAssgin Time elapsed: 3.067 s <<< ERROR!
io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
at io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:268)
at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:249)
at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:167)
at org.apache.uniffle.proto.ShuffleServerGrpc$ShuffleServerBlockingStub.registerShuffle(ShuffleServerGrpc.java:850)
at org.apache.uniffle.client.impl.grpc.ShuffleServerGrpcClient.doRegisterShuffle(ShuffleServerGrpcClient.java:244)
at org.apache.uniffle.client.impl.grpc.ShuffleServerGrpcClient.registerShuffle(ShuffleServerGrpcClient.java:500)
at org.apache.uniffle.client.impl.ShuffleWriteClientImpl.registerShuffle(ShuffleWriteClientImpl.java:604)
at org.apache.uniffle.client.api.ShuffleWriteClient.registerShuffle(ShuffleWriteClient.java:67)
at org.apache.uniffle.test.ShuffleWithRssClientTest.lambda$null$4(ShuffleWithRssClientTest.java:484)
at java.util.HashMap$EntrySet.forEach(HashMap.java:1045)
at org.apache.uniffle.test.ShuffleWithRssClientTest.lambda$testRetryAssgin$5(ShuffleWithRssClientTest.java:470)
at org.apache.uniffle.common.util.RetryUtils.retryWithCondition(RetryUtils.java:81)
at org.apache.uniffle.common.util.RetryUtils.retry(RetryUtils.java:61)
at org.apache.uniffle.common.util.RetryUtils.retry(RetryUtils.java:32)
at org.apache.uniffle.test.ShuffleWithRssClientTest.testRetryAssgin(ShuffleWithRssClientTest.java:458)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:725)
at org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
at org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
at org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:149)
at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:140)
at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:84)
at org.junit.jupiter.engine.execution.ExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(ExecutableInvoker.java:115)
at org.junit.jupiter.engine.execution.ExecutableInvoker.lambda$invoke$0(ExecutableInvoker.java:105)
at org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
at org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64)
at org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45)
at org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37)
at org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:104)
at org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:98)
at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$7(TestMethodTestDescriptor.java:214)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeTestMethod(TestMethodTestDescriptor.java:210)
at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:135)
at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:66)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:151)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
at java.util.ArrayList.forEach(ArrayList.java:1259)
at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:41)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
at java.util.ArrayList.forEach(ArrayList.java:1259)
at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:41)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.submit(SameThreadHierarchicalTestExecutorService.java:35)
at org.junit.platform.engine.support.hierarchical.HierarchicalTestExecutor.execute(HierarchicalTestExecutor.java:57)
at org.junit.platform.engine.support.hierarchical.HierarchicalTestEngine.execute(HierarchicalTestEngine.java:54)
at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:107)
at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:88)
at org.junit.platform.launcher.core.EngineExecutionOrchestrator.lambda$execute$0(EngineExecutionOrchestrator.java:54)
at org.junit.platform.launcher.core.EngineExecutionOrchestrator.withInterceptedStreams(EngineExecutionOrchestrator.java:67)
at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:52)
at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:114)
at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:86)
at org.junit.platform.launcher.core.DefaultLauncherSession$DelegatingLauncher.execute(DefaultLauncherSession.java:86)
at org.junit.platform.launcher.core.SessionPerRequestLauncher.execute(SessionPerRequestLauncher.java:53)
at org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.invokeAllTests(JUnitPlatformProvider.java:150)
at org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.invoke(JUnitPlatformProvider.java:124)
at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
Caused by: io.grpc.netty.shaded.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: /10.1.0.25:59373
Caused by: java.net.ConnectException: finishConnect(..) failed: Connection refused
at io.grpc.netty.shaded.io.netty.channel.unix.Errors.newConnectException0(Errors.java:166)
at io.grpc.netty.shaded.io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:131)
at io.grpc.netty.shaded.io.netty.channel.unix.Socket.finishConnect(Socket.java:359)
at io.grpc.netty.shaded.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:710)
at io.grpc.netty.shaded.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:687)
at io.grpc.netty.shaded.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567)
at io.grpc.netty.shaded.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:499)
at io.grpc.netty.shaded.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:407)
at io.grpc.netty.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
at io.grpc.netty.shaded.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.grpc.netty.shaded.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:750)
```
### Actions URL
https://github.com/apache/uniffle/actions/runs/14359740714/job/40257871501
https://github.com/apache/uniffle/actions/runs/14371412975/job/40295045200?pr=2112
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [I] [Flaky Test] org.apache.uniffle.test.ShuffleWithRssClientTest#testRetryAssgin [uniffle]
Posted by "xianjingfeng (via GitHub)" <gi...@apache.org>.
xianjingfeng closed issue #2439: [Flaky Test] org.apache.uniffle.test.ShuffleWithRssClientTest#testRetryAssgin
URL: https://github.com/apache/uniffle/issues/2439
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [I] [Flaky Test] org.apache.uniffle.test.ShuffleWithRssClientTest#testRetryAssgin [uniffle]
Posted by "xianjingfeng (via GitHub)" <gi...@apache.org>.
xianjingfeng commented on issue #2439:
URL: https://github.com/apache/uniffle/issues/2439#issuecomment-2804442833
cc @jerqi
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [I] [Flaky Test] org.apache.uniffle.test.ShuffleWithRssClientTest#testRetryAssgin [uniffle]
Posted by "xianjingfeng (via GitHub)" <gi...@apache.org>.
xianjingfeng closed issue #2439: [Flaky Test] org.apache.uniffle.test.ShuffleWithRssClientTest#testRetryAssgin
URL: https://github.com/apache/uniffle/issues/2439
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [I] [Flaky Test] org.apache.uniffle.test.ShuffleWithRssClientTest#testRetryAssgin [uniffle]
Posted by "jerqi (via GitHub)" <gi...@apache.org>.
jerqi commented on issue #2439:
URL: https://github.com/apache/uniffle/issues/2439#issuecomment-2804467533
> This problem still exists. https://github.com/apache/uniffle/actions/runs/14464695097?pr=2444 I checked the logs [integration-tests-log-spark3.3](https://github.com/apache/uniffle/actions/runs/14464695097/artifacts/2946182876)
>
> And i found one of the shuffle server(Netty port: 51612) hasn't register to the coordinator. as blew
>
> ```
> xxx@xxx tmp % grep 'bind localAddress is' integration-test/common/target/unit-tests.log| grep '08:22'
> [2025-04-15 08:22:09.055] [main] [INFO] StreamServer.startOnPort - bind localAddress is /0:0:0:0:0:0:0:0:51612
> [2025-04-15 08:22:09.063] [main] [INFO] StreamServer.startOnPort - bind localAddress is /0:0:0:0:0:0:0:0:61179
> [2025-04-15 08:22:20.527] [main] [INFO] StreamServer.startOnPort - bind localAddress is /0:0:0:0:0:0:0:0:43591
> [2025-04-15 08:22:51.282] [main] [INFO] StreamServer.startOnPort - bind localAddress is /0:0:0:0:0:0:0:0:35559
> [2025-04-15 08:22:51.290] [main] [INFO] StreamServer.startOnPort - bind localAddress is /0:0:0:0:0:0:0:0:37787
> [2025-04-15 08:22:58.353] [main] [INFO] StreamServer.startOnPort - bind localAddress is /0:0:0:0:0:0:0:0:51852
> xxx@xxx tmp % grep 'Newly registering node' integration-test/common/target/unit-tests.log| grep '08:22'
> [2025-04-15 08:22:10.088] [Grpc-1] [INFO] SimpleClusterManager.add - Newly registering node: 10.1.0.18-62088-61179
> [2025-04-15 08:22:28.956] [Grpc-4] [INFO] SimpleClusterManager.add - Newly registering node: 10.1.0.18-55112
> [2025-04-15 08:22:28.965] [Grpc-7] [INFO] SimpleClusterManager.add - Newly registering node: 10.1.0.18-57785
> [2025-04-15 08:22:36.858] [Grpc-7] [INFO] SimpleClusterManager.add - Newly registering node: 10.1.0.18-60986
> [2025-04-15 08:22:41.988] [Grpc-9] [INFO] SimpleClusterManager.add - Newly registering node: 10.1.0.18-57161
> ```
>
> I haven't found the reason yet and I think we can disable this test case.
Maybe we should add some logs first.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [I] [Flaky Test] org.apache.uniffle.test.ShuffleWithRssClientTest#testRetryAssgin [uniffle]
Posted by "xianjingfeng (via GitHub)" <gi...@apache.org>.
xianjingfeng commented on issue #2439:
URL: https://github.com/apache/uniffle/issues/2439#issuecomment-2804441903
This problem still exists. https://github.com/apache/uniffle/actions/runs/14464695097?pr=2444
I checked the logs [integration-tests-log-spark3.3](https://github.com/apache/uniffle/actions/runs/14464695097/artifacts/2946182876)
And i found one of the shuffle server(Netty port: 51612) hasn't register to the coordinator. as blew
```
xxx@xxx tmp % grep 'bind localAddress is' integration-test/common/target/unit-tests.log| grep '08:22'
[2025-04-15 08:22:09.055] [main] [INFO] StreamServer.startOnPort - bind localAddress is /0:0:0:0:0:0:0:0:51612
[2025-04-15 08:22:09.063] [main] [INFO] StreamServer.startOnPort - bind localAddress is /0:0:0:0:0:0:0:0:61179
[2025-04-15 08:22:20.527] [main] [INFO] StreamServer.startOnPort - bind localAddress is /0:0:0:0:0:0:0:0:43591
[2025-04-15 08:22:51.282] [main] [INFO] StreamServer.startOnPort - bind localAddress is /0:0:0:0:0:0:0:0:35559
[2025-04-15 08:22:51.290] [main] [INFO] StreamServer.startOnPort - bind localAddress is /0:0:0:0:0:0:0:0:37787
[2025-04-15 08:22:58.353] [main] [INFO] StreamServer.startOnPort - bind localAddress is /0:0:0:0:0:0:0:0:51852
xxx@xxx tmp % grep 'Newly registering node' integration-test/common/target/unit-tests.log| grep '08:22'
[2025-04-15 08:22:10.088] [Grpc-1] [INFO] SimpleClusterManager.add - Newly registering node: 10.1.0.18-62088-61179
[2025-04-15 08:22:28.956] [Grpc-4] [INFO] SimpleClusterManager.add - Newly registering node: 10.1.0.18-55112
[2025-04-15 08:22:28.965] [Grpc-7] [INFO] SimpleClusterManager.add - Newly registering node: 10.1.0.18-57785
[2025-04-15 08:22:36.858] [Grpc-7] [INFO] SimpleClusterManager.add - Newly registering node: 10.1.0.18-60986
[2025-04-15 08:22:41.988] [Grpc-9] [INFO] SimpleClusterManager.add - Newly registering node: 10.1.0.18-57161
```
I haven't found the reason yet and I think we can disable this test case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org