You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@uniffle.apache.org by "maobaolong (via GitHub)" <gi...@apache.org> on 2024/12/02 14:47:09 UTC

[I] [Bug] Client do not honor CANCEL interrupt exception while reportShuffleResult cause duplicated blockIds [incubator-uniffle]

maobaolong opened a new issue, #2269:
URL: https://github.com/apache/incubator-uniffle/issues/2269

   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
   
   
   ### Search before asking
   
   - [X] I have searched in the [issues](https://github.com/apache/incubator-uniffle/issues?q=is%3Aissue) and found no similar issues.
   
   
   ### Describe the bug
   
   Client continue to reportShuffleResult even interrupted by caller. this can cause server received duplicated reportShufflerResult rpc which harm for server cpu.
   
   It should better to stop retrying and throw RssException.
   
   ### Affects Version(s)
   
   master
   
   ### Uniffle Server Log Output
   
   ```logtalk
   - Server.log
   
   [2024-11-29 02:43:02.906] [Grpc-311] [WARN] ShuffleServerGrpcService - Existing 300 duplicated blockIds on blockId report for appId: application_1729849261128_3823937_1732816526781, shuffleId: 6
   ```
   
   - Server Rpc audit log
   ```
   [2024-11-29 02:43:02.893] appId application_1729849261128_3823937_1732816526781 cmd=reportShuffleResult argstaskAttemptId=16897, bitmapNum=10, partitionToBlockIdsSize=300
   ```
   
   
   <img width="1601" alt="image" src="https://github.com/user-attachments/assets/2e2a774e-64bf-4178-9e12-93ee433384e7">
   ```
   
   
   ### Uniffle Engine Log Output
   
   ```logtalk
   spark.log
   
   
   [02:42:51:164] [Executor task launch worker for task 2112.1 in stage 9.0 (TID 54359)] INFO  org.apache.spark.shuffle.writer.RssShuffleWriter.<init>:193 - RssShuffle start write taskAttemptId[16897] data with RssHandle[appId application_1729849261128_3823937_1732816526781, shuffleId 6].
   
   [02:42:56:731] [Executor task launch worker for task 2112.1 in stage 9.0 (TID 54359)] INFO  org.apache.spark.shuffle.writer.WriteBufferManager.clear:365 - Flush total buffer for shuffleId[6] with allocated[134217728], dataSize[113121456], memoryUsed[124038144], number of blocks[6000], flush ratio[1.0]
   [02:42:59:230] [dispatcher-Executor] INFO  org.apache.spark.executor.Executor.logInfo:61 - Executor is trying to kill task 2112.1 in stage 9.0 (TID 54359), reason: another attempt succeeded
   
   [02:43:03:205] [Executor task launch worker for task 2112.1 in stage 9.0 (TID 54359)] WARN  org.apache.uniffle.client.impl.grpc.ShuffleServerGrpcClient.doReportShuffleResult:797 - Report shuffle result to host[192.168.1.100], port[19977] failed, try again, retryNum[100]
   org.apache.uniffle.shaded.io.grpc.StatusRuntimeException: CANCELLED: Thread interrupted
   	at org.apache.uniffle.shaded.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:268) ~[rss-client-spark3-shaded.jar:?]
   	at org.apache.uniffle.shaded.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:249) ~[rss-client-spark3-shaded.jar:?]
   	at org.apache.uniffle.shaded.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:167) ~[rss-client-spark3-shaded.jar:?]
   	at org.apache.uniffle.proto.ShuffleServerGrpc$ShuffleServerBlockingStub.reportShuffleResult(ShuffleServerGrpc.java:814) ~[rss-client-spark3-shaded.jar:?]
   	at org.apache.uniffle.client.impl.grpc.ShuffleServerGrpcClient.doReportShuffleResult(ShuffleServerGrpcClient.java:793) ~[rss-client-spark3-shaded.jar:?]
   	at org.apache.uniffle.client.impl.grpc.ShuffleServerGrpcClient.reportShuffleResult(ShuffleServerGrpcClient.java:762) ~[rss-client-spark3-shaded.jar:?]
   	at org.apache.uniffle.client.impl.ShuffleWriteClientImpl.reportShuffleResult(ShuffleWriteClientImpl.java:724) ~[rss-client-spark3-shaded.jar:?]
   	at org.apache.spark.shuffle.writer.RssShuffleWriter.stop(RssShuffleWriter.java:778) ~[rss-client-spark3-shaded.jar:?]
   	at org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:60) ~[spark-core_2.12-3.3.1.jar:3.3.1]
   	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99) ~[spark-core_2.12-3.3.1.jar:3.3.1]
   	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:52) ~[spark-core_2.12-3.3.1.jar:3.3.1]
   	at org.apache.spark.scheduler.Task.run(Task.scala:136) ~[spark-core_2.12-3.3.1.jar:3.3.1]
   	at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:548) ~[spark-core_2.12-3.3.1.jar:3.3.1]
   	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1556) ~[spark-core_2.12-3.3.1.jar:3.3.1]
   	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:551) ~[spark-core_2.12-3.3.1.jar:3.3.1]
   	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_392]
   	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_392]
   	at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_392]
   Caused by: java.lang.InterruptedException
   	at org.apache.uniffle.shaded.io.grpc.stub.ClientCalls$ThreadlessExecutor.throwIfInterrupted(ClientCalls.java:750) ~[rss-client-spark3-shaded.jar:?]
   	at org.apache.uniffle.shaded.io.grpc.stub.ClientCalls$ThreadlessExecutor.waitAndDrain(ClientCalls.java:711) ~[rss-client-spark3-shaded.jar:?]
   	at org.apache.uniffle.shaded.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:159) ~[rss-client-spark3-shaded.jar:?]
   	... 15 more
   [02:43:03:206] [Executor task launch worker for task 2112.1 in stage 9.0 (TID 54359)] WARN  org.apache.uniffle.client.impl.ShuffleWriteClientImpl.reportShuffleResult:747 - Report shuffle result is failed to ShuffleServerInfo{host[192.168.1.100], grpc port[19977], netty port[17000]} for appId[application_1729849261128_3823937_1732816526781], shuffleId[6]
   [02:43:03:208] [Executor task launch worker for task 2112.1 in stage 9.0 (TID 54359)] WARN  org.apache.spark.memory.ExecutionMemoryPool.logWarning:73 - Internal error: release called on 10179584 bytes but task only has 0 bytes of memory from the on-heap execution pool
   [02:43:03:208] [Executor task launch worker for task 2112.1 in stage 9.0 (TID 54359)] INFO  org.apache.spark.executor.Executor.logInfo:61 - Executor interrupted and killed task 2112.1 in stage 9.0 (TID 54359), reason: another attempt succeeded
   ```
   ```
   
   
   ### Uniffle Server Configurations
   
   _No response_
   
   ### Uniffle Engine Configurations
   
   _No response_
   
   ### Additional context
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [X] Yes I am willing to submit a PR!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


Re: [I] [Bug] Client do not honor CANCEL interrupt exception while reportShuffleResult cause duplicated blockIds [incubator-uniffle]

Posted by "maobaolong (via GitHub)" <gi...@apache.org>.
maobaolong closed issue #2269: [Bug] Client do not honor CANCEL interrupt exception while reportShuffleResult cause duplicated blockIds
URL: https://github.com/apache/incubator-uniffle/issues/2269


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


Re: [I] [Bug] Client do not honor CANCEL interrupt exception while reportShuffleResult cause duplicated blockIds [incubator-uniffle]

Posted by "maobaolong (via GitHub)" <gi...@apache.org>.
maobaolong closed issue #2269: [Bug] Client do not honor CANCEL interrupt exception while reportShuffleResult cause duplicated blockIds
URL: https://github.com/apache/incubator-uniffle/issues/2269


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org