(incubator-celeborn) branch main updated: [CELEBORN-1222] Fix Celeborn worker won't record HDFS writer

[zh...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

zhouky pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/incubator-celeborn.git


The following commit(s) were added to refs/heads/main by this push:
     new 413e4f3b8 [CELEBORN-1222] Fix Celeborn worker won't record HDFS writer
413e4f3b8 is described below

commit 413e4f3b8015b6e0c121162c8a60154934e23d4f
Author: mingji <fe...@alibaba-inc.com>
AuthorDate: Tue Jan 16 10:08:03 2024 +0800

    [CELEBORN-1222] Fix Celeborn worker won't record HDFS writer
    
    ### What changes were proposed in this pull request?
    To record hdfs writer in worker.
    To fix a bug introduced by https://github.com/apache/incubator-celeborn/pull/2130.
    
    ### Why are the changes needed?
    If the hdfs writer won't be recorded, the worker won't clean the HDFS shuffle file if a partition is broken until the master cleans the HDFS.
    
    ### Does this PR introduce _any_ user-facing change?
    NO.
    
    ### How was this patch tested?
    GA and cluster.
    
    Closes #2227 from FMX/b1222.
    
    Authored-by: mingji <fe...@alibaba-inc.com>
    Signed-off-by: zky.zhoukeyong <zk...@alibaba-inc.com>
---
 .../apache/celeborn/service/deploy/worker/storage/StorageManager.scala  | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/worker/src/main/scala/org/apache/celeborn/service/deploy/worker/storage/StorageManager.scala b/worker/src/main/scala/org/apache/celeborn/service/deploy/worker/storage/StorageManager.scala
index 6fcc98050..f2171b7f5 100644
--- a/worker/src/main/scala/org/apache/celeborn/service/deploy/worker/storage/StorageManager.scala
+++ b/worker/src/main/scala/org/apache/celeborn/service/deploy/worker/storage/StorageManager.scala
@@ -399,6 +399,8 @@ final private[worker] class StorageManager(conf: CelebornConf, workerSource: Abs
       workingDirWriters.computeIfAbsent(workingDir, workingDirWriterListFunc).put(
         diskFileInfo.getFilePath,
         writer)
+    } else {
+      hdfsWriters.put(diskFileInfo.getFilePath, writer)
     }
     writer
   }