[GitHub] [incubator-celeborn] AngersZhuuuu commented on a diff in pull request #1395: [CELEBORN-484][PERF] Master trigger LifecycleManager commit shutdown worker's partition location.

["AngersZhuuuu (via GitHub)" <gi...@apache.org>]

AngersZhuuuu commented on code in PR #1395:
URL: https://github.com/apache/incubator-celeborn/pull/1395#discussion_r1151357017


##########
master/src/main/java/org/apache/celeborn/service/deploy/master/clustermeta/AbstractMetaManager.java:
##########
@@ -244,6 +247,7 @@ public void updateRegisterWorkerMeta(
     synchronized (workers) {
       if (!workers.contains(workerInfo)) {
         workers.add(workerInfo);
+        shutdownWorkers.remove(workerInfo);

Review Comment:
   Only remove shutdown worker when re-register



##########
master/src/main/java/org/apache/celeborn/service/deploy/master/clustermeta/AbstractMetaManager.java:
##########
@@ -344,6 +354,7 @@ public void restoreMetaFromFile(File file) throws IOException {
 
   public void updateBlacklistByReportWorkerUnavailable(List<WorkerInfo> failedWorkers) {
     synchronized (this.workers) {
+      shutdownWorkers.addAll(failedWorkers);

Review Comment:
   Only add shutdownWorkers when ReportWorkerUnavailable



##########
client/src/main/scala/org/apache/celeborn/client/CommitManager.scala:
##########
@@ -291,4 +292,27 @@ class CommitManager(appId: String, val conf: CelebornConf, lifecycleManager: Lif
     }
     (totalWritten, totalFileCount)
   }
+
+  def handleShutdownWorker(shutdownWorker: WorkerInfo): Unit = {

Review Comment:
   During a worker shutdown, it won't revive new reserve slot request and push data request.
   So here in the ShufflePartitionLocationInfo, there won't contains new partition location of this worker.
   Since if we don't commit, it won't restart.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org