(incubator-celeborn) branch branch-0.3 updated: [CELEBORN-1103][BUG] only clean up expire data for good disks

[zh...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

zhouky pushed a commit to branch branch-0.3
in repository https://gitbox.apache.org/repos/asf/incubator-celeborn.git


The following commit(s) were added to refs/heads/branch-0.3 by this push:
     new 0ebd9d53c [CELEBORN-1103][BUG] only clean up expire data for good disks
0ebd9d53c is described below

commit 0ebd9d53c64b1ce0c7019163b2c790900031b777
Author: qinrui <qr...@gmail.com>
AuthorDate: Tue Oct 31 21:52:46 2023 +0800

    [CELEBORN-1103][BUG] only clean up expire data for good disks
    
    ### What changes were proposed in this pull request?
    **When a bad disk occurs, cleaning up expired shuffle keys can cause NullPointerException appearing in the thread pool obtained from `diskOperators` in `StorageManager`.
    Therefore, only cleaning up expired shuffle keys from good disks will not cause the above problems.**
    
    https://issues.apache.org/jira/browse/CELEBORN-1103
    
    ### Why are the changes needed?
    bugfix
    
    ### Does this PR introduce _any_ user-facing change?
    
    ### How was this patch tested?
    
    Closes #2060 from suizhe007/CELEBORN-1103.
    
    Lead-authored-by: qinrui <qr...@gmail.com>
    Co-authored-by: qinrui <51...@users.noreply.github.com>
    Signed-off-by: zky.zhoukeyong <zk...@alibaba-inc.com>
    (cherry picked from commit 232a44b9ea78ef74f9b3ac535e6e61401b8667b9)
    Signed-off-by: zky.zhoukeyong <zk...@alibaba-inc.com>
---
 .../celeborn/service/deploy/worker/storage/StorageManager.scala   | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)

diff --git a/worker/src/main/scala/org/apache/celeborn/service/deploy/worker/storage/StorageManager.scala b/worker/src/main/scala/org/apache/celeborn/service/deploy/worker/storage/StorageManager.scala
index af49cae60..8da9db9b6 100644
--- a/worker/src/main/scala/org/apache/celeborn/service/deploy/worker/storage/StorageManager.scala
+++ b/worker/src/main/scala/org/apache/celeborn/service/deploy/worker/storage/StorageManager.scala
@@ -557,7 +557,9 @@ final private[worker] class StorageManager(conf: CelebornConf, workerSource: Abs
           }
         }
         val (appId, shuffleId) = Utils.splitShuffleKey(shuffleKey)
-        disksSnapshot().filter(_.status != DiskStatus.IO_HANG).foreach { diskInfo =>
+        disksSnapshot().filter(diskInfo =>
+          diskInfo.status == DiskStatus.HEALTHY
+            || diskInfo.status == DiskStatus.HIGH_DISK_USAGE).foreach { diskInfo =>
           diskInfo.dirs.foreach { dir =>
             val file = new File(dir, s"$appId/$shuffleId")
             deleteDirectory(file, diskOperators.get(diskInfo.mountPoint))
@@ -603,7 +605,9 @@ final private[worker] class StorageManager(conf: CelebornConf, workerSource: Abs
 
   private def cleanupExpiredAppDirs(expireDuration: Long): Unit = {
     val diskInfoAndAppDirs = disksSnapshot()
-      .filter(_.status != DiskStatus.IO_HANG)
+      .filter(diskInfo =>
+        diskInfo.status == DiskStatus.HEALTHY
+          || diskInfo.status == DiskStatus.HIGH_DISK_USAGE)
       .map { case diskInfo =>
         (diskInfo, diskInfo.dirs.filter(_.exists).flatMap(_.listFiles()))
       }