[GitHub] [incubator-celeborn] waitinfuture commented on a diff in pull request #1613: [CELEBORN-703][WORKER][PERF] Avoid calling `CelebornConf#get` multi-time when `PushDataHandler` handle `PushData`/`PushMergedData`

["waitinfuture (via GitHub)" <gi...@apache.org>]

waitinfuture commented on code in PR #1613:
URL: https://github.com/apache/incubator-celeborn/pull/1613#discussion_r1236264704


##########
worker/src/main/scala/org/apache/celeborn/service/deploy/worker/PushDataHandler.scala:
##########
@@ -1068,6 +1074,6 @@ class PushDataHandler extends BaseMessageHandler with Logging {
 }
 
 object PushDataHandler {
-  @volatile var pushMasterDataTimeoutTested = false
-  @volatile var pushSlaveDataTimeoutTested = false
+  @volatile private var pushMasterMergeDataTimeoutTested = false

Review Comment:
   I dived into it and think the reason is that, if put the variables in the object, it's process-level singleton, as a result, although we have 3 workers in the MiniCluster, only one timeout will be triggered, and the worker will be put in the blacklist. However, when put the variables into Handler, all three workers will trigger timeout and all will be blacklist, as a result it will cause Revive Exception for no slots available. Following are related logs:
   ```
   Reporting Worker Failure:
   Host:30.221.101.208:RpcPort:57565:PushPort:57568:FetchPort:57570:ReplicatePort:57569   PUSH_DATA_TIMEOUT_MASTER   1687321430258
   Current blacklist:
   Host:30.221.101.208:RpcPort:57560:PushPort:57562:FetchPort:57564:ReplicatePort:57563   PUSH_DATA_TIMEOUT_MASTER   1687321430254
   Host:30.221.101.208:RpcPort:57572:PushPort:57576:FetchPort:57578:ReplicatePort:57577   PUSH_DATA_TIMEOUT_MASTER   1687321430248  
   ```
   ```
   Revive failed, status SLOT_NOT_AVAILABLE
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org