You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@celeborn.apache.org by "RexXiong (via GitHub)" <gi...@apache.org> on 2023/03/07 09:37:27 UTC

[GitHub] [incubator-celeborn] RexXiong opened a new pull request, #1314: [CELEBORN-356] [Flink] support release single partition resource

RexXiong opened a new pull request, #1314:
URL: https://github.com/apache/incubator-celeborn/pull/1314

   <!--
   Thanks for sending a pull request!  Here are some tips for you:
     - Make sure the PR title start w/ a JIRA ticket, e.g. '[CELEBORN-XXXX] Your PR title ...'.
     - Be sure to keep the PR description updated to reflect all changes.
     - Please write your PR title to summarize what this PR proposes.
     - If possible, provide a concise example to reproduce the issue for a faster review.
   -->
   
   ### What changes were proposed in this pull request?
   1.use destroy rpc to support release single partition resource for flink
   2.support batch mode to reduce the number of total rpcs
   
   Here would try to release resources for later using,  anything which not released successfully will be released when the shuffle /job is expired.
   
   ### Why are the changes needed?
   The unit of Flink resource management(or Map Partition mode) is partition,and if a partition is no longer needed, it can be immediately released, this will reduce the resources used by the Flink job, and the released partition resources can be reused later even in the same job. 
   
   ### Does this PR introduce _any_ user-facing change?
   No
   
   
   ### How was this patch tested?
   UT & manual TPCDS 1T
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-celeborn] waitinfuture merged pull request #1314: [CELEBORN-356] [FLINK] Support release single partition resource

Posted by "waitinfuture (via GitHub)" <gi...@apache.org>.
waitinfuture merged PR #1314:
URL: https://github.com/apache/incubator-celeborn/pull/1314


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-celeborn] waitinfuture commented on pull request #1314: [CELEBORN-356] [FLINK] Support release single partition resource

Posted by "waitinfuture (via GitHub)" <gi...@apache.org>.
waitinfuture commented on PR #1314:
URL: https://github.com/apache/incubator-celeborn/pull/1314#issuecomment-1458426307

   > This patch is mainly for map partition[Flink], but it must be pointed out that due to concurrent problem, I have to change partition location structure to concurrentHashSet(or use synchronized on that object instead), Would you have time to take a look @FMX @waitinfuture @AngersZhuuuu
   
   I will review late today, thanks!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-celeborn] zhongqiangczq commented on a diff in pull request #1314: [CELEBORN-356] [FLINK] Support release single partition resource

Posted by "zhongqiangczq (via GitHub)" <gi...@apache.org>.
zhongqiangczq commented on code in PR #1314:
URL: https://github.com/apache/incubator-celeborn/pull/1314#discussion_r1147075253


##########
worker/src/main/scala/org/apache/celeborn/service/deploy/worker/storage/StorageManager.scala:
##########
@@ -424,30 +458,10 @@ final private[worker] class StorageManager(conf: CelebornConf, workerSource: Abs
         var isHdfsExpired = false
         if (removedFileInfos != null) {
           removedFileInfos.asScala.foreach {
-            case (_, fileInfo) => {
-              if (fileInfo.isHdfs) {
+            case (_, fileInfo) =>
+              if (cleanFileInternal(shuffleKey, fileInfo)) {
                 isHdfsExpired = true

Review Comment:
   "isHdfsExpired = true" should be removed. when fileInfo.isHdfs , isHdfsExpired can be setted to true.
   
   " if (cleanFileInternal(shuffleKey, fileInfo)) {
                   isHdfsExpired = true" => cleanFileInternal(shuffleKey, fileInfo)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-celeborn] RexXiong commented on a diff in pull request #1314: [CELEBORN-356] [FLINK] Support release single partition resource

Posted by "RexXiong (via GitHub)" <gi...@apache.org>.
RexXiong commented on code in PR #1314:
URL: https://github.com/apache/incubator-celeborn/pull/1314#discussion_r1147136647


##########
worker/src/main/scala/org/apache/celeborn/service/deploy/worker/storage/StorageManager.scala:
##########
@@ -416,6 +416,40 @@ final private[worker] class StorageManager(conf: CelebornConf, workerSource: Abs
     }.toSeq.sortBy(_._2).reverse.take(conf.metricsAppTopDiskUsageCount * 2).toMap.asJava
   }
 
+  def cleanFile(shuffleKey: String, fileName: String): Unit = {
+    val fileInfo = getFileInfo(shuffleKey, fileName)
+    if (fileInfo != null) {
+      cleanFileInternal(shuffleKey, fileInfo)
+    }
+  }
+
+  def cleanFileInternal(shuffleKey: String, fileInfo: FileInfo): Boolean = {
+    var isHdfsExpired = false
+    if (fileInfo.isHdfs) {
+      isHdfsExpired = true
+      val hdfsFileWriter = hdfsWriters.get(fileInfo.getFilePath)
+      if (hdfsFileWriter != null) {
+        hdfsFileWriter.destroy(new IOException(
+          s"Destroy FileWriter ${hdfsFileWriter} caused by shuffle ${shuffleKey} expired."))
+        hdfsWriters.remove(fileInfo.getFilePath)

Review Comment:
   > @RexXiong If it happens, will it cause problem?
   
   As cleanFile method is idempotent & support concurrent, So it would works fine even that scenario.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-celeborn] RexXiong commented on pull request #1314: [CELEBORN-356] [FLINK] Support release single partition resource

Posted by "RexXiong (via GitHub)" <gi...@apache.org>.
RexXiong commented on PR #1314:
URL: https://github.com/apache/incubator-celeborn/pull/1314#issuecomment-1458202633

   This patch is mainly for map partition[Flink], but it must be pointed out that due to concurrent problem, I have to change partition location structure to concurrentHashSet(or use synchronized on that object instead), Would you have time to take a look @FMX @waitinfuture @AngersZhuuuu


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-celeborn] RexXiong commented on a diff in pull request #1314: [CELEBORN-356] [FLINK] Support release single partition resource

Posted by "RexXiong (via GitHub)" <gi...@apache.org>.
RexXiong commented on code in PR #1314:
URL: https://github.com/apache/incubator-celeborn/pull/1314#discussion_r1147109734


##########
client/src/test/scala/org/apache/celeborn/client/WithShuffleClientSuite.scala:
##########
@@ -80,6 +80,22 @@ trait WithShuffleClientSuite extends CelebornFunSuite {
     count =
       partitionLocationInfos.map(r => r.getMasterPartitions().size()).sum
     Assert.assertEquals(count, numMappers + 1)
+
+    celebornConf.set(CelebornConf.BATCH_HANDLE_RELEASE_PARTITION_ENABLED.key, "false")

Review Comment:
   done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-celeborn] RexXiong commented on a diff in pull request #1314: [CELEBORN-356] [FLINK] Support release single partition resource

Posted by "RexXiong (via GitHub)" <gi...@apache.org>.
RexXiong commented on code in PR #1314:
URL: https://github.com/apache/incubator-celeborn/pull/1314#discussion_r1147110417


##########
worker/src/main/scala/org/apache/celeborn/service/deploy/worker/storage/StorageManager.scala:
##########
@@ -416,6 +416,40 @@ final private[worker] class StorageManager(conf: CelebornConf, workerSource: Abs
     }.toSeq.sortBy(_._2).reverse.take(conf.metricsAppTopDiskUsageCount * 2).toMap.asJava
   }
 
+  def cleanFile(shuffleKey: String, fileName: String): Unit = {
+    val fileInfo = getFileInfo(shuffleKey, fileName)
+    if (fileInfo != null) {
+      cleanFileInternal(shuffleKey, fileInfo)
+    }
+  }
+
+  def cleanFileInternal(shuffleKey: String, fileInfo: FileInfo): Boolean = {
+    var isHdfsExpired = false
+    if (fileInfo.isHdfs) {
+      isHdfsExpired = true
+      val hdfsFileWriter = hdfsWriters.get(fileInfo.getFilePath)
+      if (hdfsFileWriter != null) {
+        hdfsFileWriter.destroy(new IOException(
+          s"Destroy FileWriter ${hdfsFileWriter} caused by shuffle ${shuffleKey} expired."))
+        hdfsWriters.remove(fileInfo.getFilePath)

Review Comment:
   > Will it happen that hdfsFileWriter is null but the directory is not empty? cc @zhongqiangczq Why is old code written that way?
   
   Currently batch release thread may release partition while shuffle expired, and this can be very rare case.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-celeborn] waitinfuture commented on a diff in pull request #1314: [CELEBORN-356] [FLINK] Support release single partition resource

Posted by "waitinfuture (via GitHub)" <gi...@apache.org>.
waitinfuture commented on code in PR #1314:
URL: https://github.com/apache/incubator-celeborn/pull/1314#discussion_r1147132607


##########
worker/src/main/scala/org/apache/celeborn/service/deploy/worker/storage/StorageManager.scala:
##########
@@ -416,6 +416,40 @@ final private[worker] class StorageManager(conf: CelebornConf, workerSource: Abs
     }.toSeq.sortBy(_._2).reverse.take(conf.metricsAppTopDiskUsageCount * 2).toMap.asJava
   }
 
+  def cleanFile(shuffleKey: String, fileName: String): Unit = {
+    val fileInfo = getFileInfo(shuffleKey, fileName)
+    if (fileInfo != null) {
+      cleanFileInternal(shuffleKey, fileInfo)
+    }
+  }
+
+  def cleanFileInternal(shuffleKey: String, fileInfo: FileInfo): Boolean = {
+    var isHdfsExpired = false
+    if (fileInfo.isHdfs) {
+      isHdfsExpired = true
+      val hdfsFileWriter = hdfsWriters.get(fileInfo.getFilePath)
+      if (hdfsFileWriter != null) {
+        hdfsFileWriter.destroy(new IOException(
+          s"Destroy FileWriter ${hdfsFileWriter} caused by shuffle ${shuffleKey} expired."))
+        hdfsWriters.remove(fileInfo.getFilePath)

Review Comment:
   @RexXiong If it happens, will it cause problem?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-celeborn] zhongqiangczq commented on a diff in pull request #1314: [CELEBORN-356] [FLINK] Support release single partition resource

Posted by "zhongqiangczq (via GitHub)" <gi...@apache.org>.
zhongqiangczq commented on code in PR #1314:
URL: https://github.com/apache/incubator-celeborn/pull/1314#discussion_r1147073692


##########
worker/src/main/scala/org/apache/celeborn/service/deploy/worker/storage/StorageManager.scala:
##########
@@ -416,6 +416,40 @@ final private[worker] class StorageManager(conf: CelebornConf, workerSource: Abs
     }.toSeq.sortBy(_._2).reverse.take(conf.metricsAppTopDiskUsageCount * 2).toMap.asJava
   }
 
+  def cleanFile(shuffleKey: String, fileName: String): Unit = {
+    val fileInfo = getFileInfo(shuffleKey, fileName)
+    if (fileInfo != null) {
+      cleanFileInternal(shuffleKey, fileInfo)
+    }
+  }
+
+  def cleanFileInternal(shuffleKey: String, fileInfo: FileInfo): Boolean = {
+    var isHdfsExpired = false
+    if (fileInfo.isHdfs) {
+      isHdfsExpired = true
+      val hdfsFileWriter = hdfsWriters.get(fileInfo.getFilePath)
+      if (hdfsFileWriter != null) {
+        hdfsFileWriter.destroy(new IOException(
+          s"Destroy FileWriter ${hdfsFileWriter} caused by shuffle ${shuffleKey} expired."))
+        hdfsWriters.remove(fileInfo.getFilePath)

Review Comment:
   conncurrent case, multiple thread will call cleanFileInternal. After one thread delete the file and then another thread gets the null filewriter. so it's necessary to  check whether hdfsFileWriter is null 



##########
worker/src/main/scala/org/apache/celeborn/service/deploy/worker/storage/StorageManager.scala:
##########
@@ -416,6 +416,40 @@ final private[worker] class StorageManager(conf: CelebornConf, workerSource: Abs
     }.toSeq.sortBy(_._2).reverse.take(conf.metricsAppTopDiskUsageCount * 2).toMap.asJava
   }
 
+  def cleanFile(shuffleKey: String, fileName: String): Unit = {
+    val fileInfo = getFileInfo(shuffleKey, fileName)
+    if (fileInfo != null) {
+      cleanFileInternal(shuffleKey, fileInfo)
+    }
+  }
+
+  def cleanFileInternal(shuffleKey: String, fileInfo: FileInfo): Boolean = {
+    var isHdfsExpired = false
+    if (fileInfo.isHdfs) {
+      isHdfsExpired = true
+      val hdfsFileWriter = hdfsWriters.get(fileInfo.getFilePath)
+      if (hdfsFileWriter != null) {
+        hdfsFileWriter.destroy(new IOException(
+          s"Destroy FileWriter ${hdfsFileWriter} caused by shuffle ${shuffleKey} expired."))
+        hdfsWriters.remove(fileInfo.getFilePath)

Review Comment:
   conncurrent case, multiple threads will call cleanFileInternal. After one thread delete the file and then another thread gets the null filewriter. so it's necessary to  check whether hdfsFileWriter is null 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-celeborn] zhongqiangczq commented on a diff in pull request #1314: [CELEBORN-356] [FLINK] Support release single partition resource

Posted by "zhongqiangczq (via GitHub)" <gi...@apache.org>.
zhongqiangczq commented on code in PR #1314:
URL: https://github.com/apache/incubator-celeborn/pull/1314#discussion_r1147077142


##########
worker/src/main/scala/org/apache/celeborn/service/deploy/worker/storage/StorageManager.scala:
##########
@@ -424,30 +458,10 @@ final private[worker] class StorageManager(conf: CelebornConf, workerSource: Abs
         var isHdfsExpired = false
         if (removedFileInfos != null) {
           removedFileInfos.asScala.foreach {
-            case (_, fileInfo) => {
-              if (fileInfo.isHdfs) {
+            case (_, fileInfo) =>
+              if (cleanFileInternal(shuffleKey, fileInfo)) {
                 isHdfsExpired = true

Review Comment:
   " if (cleanFileInternal(shuffleKey, fileInfo)) {
                   isHdfsExpired = true" => isHdfsExpired = cleanFileInternal(shuffleKey, fileInfo)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-celeborn] zhongqiangczq commented on a diff in pull request #1314: [CELEBORN-356] [FLINK] Support release single partition resource

Posted by "zhongqiangczq (via GitHub)" <gi...@apache.org>.
zhongqiangczq commented on code in PR #1314:
URL: https://github.com/apache/incubator-celeborn/pull/1314#discussion_r1147073692


##########
worker/src/main/scala/org/apache/celeborn/service/deploy/worker/storage/StorageManager.scala:
##########
@@ -416,6 +416,40 @@ final private[worker] class StorageManager(conf: CelebornConf, workerSource: Abs
     }.toSeq.sortBy(_._2).reverse.take(conf.metricsAppTopDiskUsageCount * 2).toMap.asJava
   }
 
+  def cleanFile(shuffleKey: String, fileName: String): Unit = {
+    val fileInfo = getFileInfo(shuffleKey, fileName)
+    if (fileInfo != null) {
+      cleanFileInternal(shuffleKey, fileInfo)
+    }
+  }
+
+  def cleanFileInternal(shuffleKey: String, fileInfo: FileInfo): Boolean = {
+    var isHdfsExpired = false
+    if (fileInfo.isHdfs) {
+      isHdfsExpired = true
+      val hdfsFileWriter = hdfsWriters.get(fileInfo.getFilePath)
+      if (hdfsFileWriter != null) {
+        hdfsFileWriter.destroy(new IOException(
+          s"Destroy FileWriter ${hdfsFileWriter} caused by shuffle ${shuffleKey} expired."))
+        hdfsWriters.remove(fileInfo.getFilePath)

Review Comment:
   "hdfsFileWriter is null but the directory is not empty" can't happen, because delete files first and then remove. "if (hdfsFileWriter != null)" can be removed



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-celeborn] zhongqiangczq commented on a diff in pull request #1314: [CELEBORN-356] [FLINK] Support release single partition resource

Posted by "zhongqiangczq (via GitHub)" <gi...@apache.org>.
zhongqiangczq commented on code in PR #1314:
URL: https://github.com/apache/incubator-celeborn/pull/1314#discussion_r1147073692


##########
worker/src/main/scala/org/apache/celeborn/service/deploy/worker/storage/StorageManager.scala:
##########
@@ -416,6 +416,40 @@ final private[worker] class StorageManager(conf: CelebornConf, workerSource: Abs
     }.toSeq.sortBy(_._2).reverse.take(conf.metricsAppTopDiskUsageCount * 2).toMap.asJava
   }
 
+  def cleanFile(shuffleKey: String, fileName: String): Unit = {
+    val fileInfo = getFileInfo(shuffleKey, fileName)
+    if (fileInfo != null) {
+      cleanFileInternal(shuffleKey, fileInfo)
+    }
+  }
+
+  def cleanFileInternal(shuffleKey: String, fileInfo: FileInfo): Boolean = {
+    var isHdfsExpired = false
+    if (fileInfo.isHdfs) {
+      isHdfsExpired = true
+      val hdfsFileWriter = hdfsWriters.get(fileInfo.getFilePath)
+      if (hdfsFileWriter != null) {
+        hdfsFileWriter.destroy(new IOException(
+          s"Destroy FileWriter ${hdfsFileWriter} caused by shuffle ${shuffleKey} expired."))
+        hdfsWriters.remove(fileInfo.getFilePath)

Review Comment:
   conncurrent case, multiple threads will call cleanFileInternal. After one thread deletes the file and then another thread gets the null filewriter. so it's necessary to  check whether hdfsFileWriter is null 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-celeborn] codecov[bot] commented on pull request #1314: [CELEBORN-356] [Flink] support release single partition resource

Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #1314:
URL: https://github.com/apache/incubator-celeborn/pull/1314#issuecomment-1457909351

   # [Codecov](https://codecov.io/gh/apache/incubator-celeborn/pull/1314?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#1314](https://codecov.io/gh/apache/incubator-celeborn/pull/1314?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (c04cd6f) into [main](https://codecov.io/gh/apache/incubator-celeborn/commit/786fcd6744cc755eeb884b82b2186fb1b29e03f7?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (786fcd6) will **increase** coverage by `0.75%`.
   > The diff coverage is `11.57%`.
   
   ```diff
   @@             Coverage Diff              @@
   ##               main    #1314      +/-   ##
   ============================================
   + Coverage     26.97%   27.72%   +0.75%     
   - Complexity      821      843      +22     
   ============================================
     Files           218      222       +4     
     Lines         18540    18562      +22     
     Branches       1998     1994       -4     
   ============================================
   + Hits           5000     5144     +144     
   + Misses        13212    13065     -147     
   - Partials        328      353      +25     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/incubator-celeborn/pull/1314?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [...ala/org/apache/celeborn/client/CommitManager.scala](https://codecov.io/gh/apache/incubator-celeborn/pull/1314?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y2xpZW50L3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvY2VsZWJvcm4vY2xpZW50L0NvbW1pdE1hbmFnZXIuc2NhbGE=) | `0.00% <0.00%> (ø)` | |
   | [.../org/apache/celeborn/client/LifecycleManager.scala](https://codecov.io/gh/apache/incubator-celeborn/pull/1314?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y2xpZW50L3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvY2VsZWJvcm4vY2xpZW50L0xpZmVjeWNsZU1hbmFnZXIuc2NhbGE=) | `0.00% <0.00%> (ø)` | |
   | [...ache/celeborn/client/ReleasePartitionManager.scala](https://codecov.io/gh/apache/incubator-celeborn/pull/1314?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y2xpZW50L3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvY2VsZWJvcm4vY2xpZW50L1JlbGVhc2VQYXJ0aXRpb25NYW5hZ2VyLnNjYWxh) | `0.00% <0.00%> (ø)` | |
   | [.../apache/celeborn/client/commit/CommitHandler.scala](https://codecov.io/gh/apache/incubator-celeborn/pull/1314?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y2xpZW50L3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvY2VsZWJvcm4vY2xpZW50L2NvbW1pdC9Db21taXRIYW5kbGVyLnNjYWxh) | `0.00% <0.00%> (ø)` | |
   | [...born/client/commit/MapPartitionCommitHandler.scala](https://codecov.io/gh/apache/incubator-celeborn/pull/1314?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y2xpZW50L3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvY2VsZWJvcm4vY2xpZW50L2NvbW1pdC9NYXBQYXJ0aXRpb25Db21taXRIYW5kbGVyLnNjYWxh) | `0.00% <0.00%> (ø)` | |
   | [...he/celeborn/common/protocol/PartitionLocation.java](https://codecov.io/gh/apache/incubator-celeborn/pull/1314?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9jZWxlYm9ybi9jb21tb24vcHJvdG9jb2wvUGFydGl0aW9uTG9jYXRpb24uamF2YQ==) | `38.55% <0.00%> (-0.40%)` | :arrow_down: |
   | [...orn/common/meta/ShufflePartitionLocationInfo.scala](https://codecov.io/gh/apache/incubator-celeborn/pull/1314?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvY2VsZWJvcm4vY29tbW9uL21ldGEvU2h1ZmZsZVBhcnRpdGlvbkxvY2F0aW9uSW5mby5zY2FsYQ==) | `0.00% <0.00%> (ø)` | |
   | [...he/celeborn/service/deploy/worker/Controller.scala](https://codecov.io/gh/apache/incubator-celeborn/pull/1314?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-d29ya2VyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvY2VsZWJvcm4vc2VydmljZS9kZXBsb3kvd29ya2VyL0NvbnRyb2xsZXIuc2NhbGE=) | `0.00% <0.00%> (ø)` | |
   | [...service/deploy/worker/storage/StorageManager.scala](https://codecov.io/gh/apache/incubator-celeborn/pull/1314?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-d29ya2VyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvY2VsZWJvcm4vc2VydmljZS9kZXBsb3kvd29ya2VyL3N0b3JhZ2UvU3RvcmFnZU1hbmFnZXIuc2NhbGE=) | `20.25% <0.00%> (+20.25%)` | :arrow_up: |
   | [...cala/org/apache/celeborn/common/CelebornConf.scala](https://codecov.io/gh/apache/incubator-celeborn/pull/1314?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvY2VsZWJvcm4vY29tbW9uL0NlbGVib3JuQ29uZi5zY2FsYQ==) | `81.29% <83.34%> (+0.13%)` | :arrow_up: |
   | ... and [22 more](https://codecov.io/gh/apache/incubator-celeborn/pull/1314?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   :mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-celeborn] waitinfuture commented on a diff in pull request #1314: [CELEBORN-356] [FLINK] Support release single partition resource

Posted by "waitinfuture (via GitHub)" <gi...@apache.org>.
waitinfuture commented on code in PR #1314:
URL: https://github.com/apache/incubator-celeborn/pull/1314#discussion_r1146212184


##########
client/src/test/scala/org/apache/celeborn/client/WithShuffleClientSuite.scala:
##########
@@ -80,6 +80,22 @@ trait WithShuffleClientSuite extends CelebornFunSuite {
     count =
       partitionLocationInfos.map(r => r.getMasterPartitions().size()).sum
     Assert.assertEquals(count, numMappers + 1)
+
+    celebornConf.set(CelebornConf.BATCH_HANDLE_RELEASE_PARTITION_ENABLED.key, "false")

Review Comment:
   We can also add a case to set batch true.



##########
worker/src/main/scala/org/apache/celeborn/service/deploy/worker/storage/StorageManager.scala:
##########
@@ -416,6 +416,40 @@ final private[worker] class StorageManager(conf: CelebornConf, workerSource: Abs
     }.toSeq.sortBy(_._2).reverse.take(conf.metricsAppTopDiskUsageCount * 2).toMap.asJava
   }
 
+  def cleanFile(shuffleKey: String, fileName: String): Unit = {
+    val fileInfo = getFileInfo(shuffleKey, fileName)
+    if (fileInfo != null) {
+      cleanFileInternal(shuffleKey, fileInfo)
+    }
+  }
+
+  def cleanFileInternal(shuffleKey: String, fileInfo: FileInfo): Boolean = {
+    var isHdfsExpired = false
+    if (fileInfo.isHdfs) {
+      isHdfsExpired = true
+      val hdfsFileWriter = hdfsWriters.get(fileInfo.getFilePath)
+      if (hdfsFileWriter != null) {
+        hdfsFileWriter.destroy(new IOException(
+          s"Destroy FileWriter ${hdfsFileWriter} caused by shuffle ${shuffleKey} expired."))
+        hdfsWriters.remove(fileInfo.getFilePath)

Review Comment:
   Will it happen that hdfsFileWriter is null but the directory is not empty? cc @zhongqiangczq  Why is old code written that way?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-celeborn] zhongqiangczq commented on a diff in pull request #1314: [CELEBORN-356] [FLINK] Support release single partition resource

Posted by "zhongqiangczq (via GitHub)" <gi...@apache.org>.
zhongqiangczq commented on code in PR #1314:
URL: https://github.com/apache/incubator-celeborn/pull/1314#discussion_r1147077142


##########
worker/src/main/scala/org/apache/celeborn/service/deploy/worker/storage/StorageManager.scala:
##########
@@ -424,30 +458,10 @@ final private[worker] class StorageManager(conf: CelebornConf, workerSource: Abs
         var isHdfsExpired = false
         if (removedFileInfos != null) {
           removedFileInfos.asScala.foreach {
-            case (_, fileInfo) => {
-              if (fileInfo.isHdfs) {
+            case (_, fileInfo) =>
+              if (cleanFileInternal(shuffleKey, fileInfo)) {
                 isHdfsExpired = true

Review Comment:
   " if (cleanFileInternal(shuffleKey, fileInfo)) {
                   isHdfsExpired = true" => isHdfsExpired = cleanFileInternal(shuffleKey, fileInfo)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org