You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@celeborn.apache.org by "wilsonjie (via GitHub)" <gi...@apache.org> on 2023/10/17 08:52:26 UTC

[PR] [CELEBORN-1046] Add an expiration time configuration for app director… [incubator-celeborn]

wilsonjie opened a new pull request, #1998:
URL: https://github.com/apache/incubator-celeborn/pull/1998

   ### What changes were proposed in this pull request?
   Add a configuration "celeborn.worker.storage.expireDirs.timeout" with a default value of 6h in rsswork. This configuration is used to set the expiration time for app local directories.
   
   
   https://issues.apache.org/jira/browse/CELEBORN-1046
   ### Why are the changes needed?
   When Celeborn periodically deletes the directories of an app, it determines whether the app needs to be deleted based on the shuffleKeySet in memory. However, this method may not accurately indicate the completion of the app and could potentially lead to the unintentional deletion of shuffle data.
   
   ### Does this PR introduce _any_ user-facing change?
   
   
   
   ### How was this patch tested?
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CELEBORN-1046] Add an expiration time configuration for app directory to clean up [incubator-celeborn]

Posted by "SteNicholas (via GitHub)" <gi...@apache.org>.
SteNicholas commented on code in PR #1998:
URL: https://github.com/apache/incubator-celeborn/pull/1998#discussion_r1361824195


##########
common/src/main/scala/org/apache/celeborn/common/CelebornConf.scala:
##########
@@ -2083,6 +2084,14 @@ object CelebornConf extends Logging {
       .intConf
       .createWithDefault(16)
 
+  val WORKER_STORAGE_EXPIRE_DIR_TIMEOUT: ConfigEntry[Long] =
+    buildConf("celeborn.worker.storage.expireDirs.timeout")
+      .categories("worker")
+      .version("0.3.1")
+      .doc(s"The timeout for a expire dirs to be deleted on disk.")
+      .timeConf(TimeUnit.MILLISECONDS)
+      .createWithDefaultString("6h")

Review Comment:
   @wilsonjie, yes.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CELEBORN-1046] Add an expiration time configuration for app directory to clean up [incubator-celeborn]

Posted by "wilsonjie (via GitHub)" <gi...@apache.org>.
wilsonjie commented on code in PR #1998:
URL: https://github.com/apache/incubator-celeborn/pull/1998#discussion_r1361789938


##########
common/src/main/scala/org/apache/celeborn/common/CelebornConf.scala:
##########
@@ -2083,6 +2084,14 @@ object CelebornConf extends Logging {
       .intConf
       .createWithDefault(16)
 
+  val WORKER_STORAGE_EXPIRE_DIR_TIMEOUT: ConfigEntry[Long] =
+    buildConf("celeborn.worker.storage.expireDirs.timeout")
+      .categories("worker")
+      .version("0.3.1")
+      .doc(s"The timeout for a expire dirs to be deleted on disk.")
+      .timeConf(TimeUnit.MILLISECONDS)
+      .createWithDefaultString("6h")

Review Comment:
   When the execution time of most tasks exceeds the expiration time, the possibility of accidental deletion increases. In our production environment, we set it to 12 hours.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CELEBORN-1046] Add an expiration time configuration for app directory to clean up [incubator-celeborn]

Posted by "SteNicholas (via GitHub)" <gi...@apache.org>.
SteNicholas commented on code in PR #1998:
URL: https://github.com/apache/incubator-celeborn/pull/1998#discussion_r1361773283


##########
common/src/main/scala/org/apache/celeborn/common/CelebornConf.scala:
##########
@@ -2083,6 +2084,14 @@ object CelebornConf extends Logging {
       .intConf
       .createWithDefault(16)
 
+  val WORKER_STORAGE_EXPIRE_DIR_TIMEOUT: ConfigEntry[Long] =
+    buildConf("celeborn.worker.storage.expireDirs.timeout")
+      .categories("worker")
+      .version("0.3.1")
+      .doc(s"The timeout for a expire dirs to be deleted on disk.")
+      .timeConf(TimeUnit.MILLISECONDS)
+      .createWithDefaultString("6h")

Review Comment:
   Why is the default value of `celeborn.worker.storage.expireDirs.timeout` 6h?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CELEBORN-1046] Add an expiration time configuration for app directory to clean up [incubator-celeborn]

Posted by "SteNicholas (via GitHub)" <gi...@apache.org>.
SteNicholas commented on code in PR #1998:
URL: https://github.com/apache/incubator-celeborn/pull/1998#discussion_r1361786369


##########
docs/configuration/worker.md:
##########
@@ -18,7 +18,7 @@ license: |
 
 <!--begin-include-->
 | Key | Default | Description | Since |
-| --- | ------- | ----------- | ----- |
+| --- | ------- | ----------- |-------|

Review Comment:
   Is this change needed?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CELEBORN-1046] Add an expiration time configuration for app director… [incubator-celeborn]

Posted by "SteNicholas (via GitHub)" <gi...@apache.org>.
SteNicholas commented on code in PR #1998:
URL: https://github.com/apache/incubator-celeborn/pull/1998#discussion_r1361763885


##########
docs/configuration/worker.md:
##########
@@ -103,6 +103,7 @@ license: |
 | celeborn.worker.storage.dirs | &lt;undefined&gt; | Directory list to store shuffle data. It's recommended to configure one directory on each disk. Storage size limit can be set for each directory. For the sake of performance, there should be no more than 2 flush threads on the same disk partition if you are using HDD, and should be 8 or more flush threads on the same disk partition if you are using SSD. For example: `dir1[:capacity=][:disktype=][:flushthread=],dir2[:capacity=][:disktype=][:flushthread=]` | 0.2.0 | 
 | celeborn.worker.storage.disk.reserve.size | 5G | Celeborn worker reserved space for each disk. | 0.3.0 | 
 | celeborn.worker.storage.workingDir | celeborn-worker/shuffle_data | Worker's working dir path name. | 0.3.0 | 
+| celeborn.worker.storage.expireDirs.timeout | 12h | The timeout for a expire dirs to be deleted on disk. | 0.3.1 |

Review Comment:
   ```suggestion
   | celeborn.worker.storage.expireDirs.timeout | 6h | The timeout for a expire dirs to be deleted on disk. | 0.3.1 |
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CELEBORN-1046] Add an expiration time configuration for app directory to clean up [incubator-celeborn]

Posted by "wilsonjie (via GitHub)" <gi...@apache.org>.
wilsonjie commented on code in PR #1998:
URL: https://github.com/apache/incubator-celeborn/pull/1998#discussion_r1361796502


##########
docs/configuration/worker.md:
##########
@@ -18,7 +18,7 @@ license: |
 
 <!--begin-include-->
 | Key | Default | Description | Since |
-| --- | ------- | ----------- | ----- |
+| --- | ------- | ----------- |-------|

Review Comment:
   fixed
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CELEBORN-1046] Add an expiration time configuration for app directory to clean up [incubator-celeborn]

Posted by "wilsonjie (via GitHub)" <gi...@apache.org>.
wilsonjie commented on code in PR #1998:
URL: https://github.com/apache/incubator-celeborn/pull/1998#discussion_r1361782115


##########
common/src/main/scala/org/apache/celeborn/common/CelebornConf.scala:
##########
@@ -2083,6 +2084,14 @@ object CelebornConf extends Logging {
       .intConf
       .createWithDefault(16)
 
+  val WORKER_STORAGE_EXPIRE_DIR_TIMEOUT: ConfigEntry[Long] =
+    buildConf("celeborn.worker.storage.expireDirs.timeout")
+      .categories("worker")
+      .version("0.3.1")

Review Comment:
   fixed



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CELEBORN-1046] Add an expiration time configuration for app directory to clean up [incubator-celeborn]

Posted by "waitinfuture (via GitHub)" <gi...@apache.org>.
waitinfuture commented on PR #1998:
URL: https://github.com/apache/incubator-celeborn/pull/1998#issuecomment-1766215523

   Merged to main/0.3


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CELEBORN-1046] Add an expiration time configuration for app directory to clean up [incubator-celeborn]

Posted by "SteNicholas (via GitHub)" <gi...@apache.org>.
SteNicholas commented on code in PR #1998:
URL: https://github.com/apache/incubator-celeborn/pull/1998#discussion_r1361798047


##########
common/src/main/scala/org/apache/celeborn/common/CelebornConf.scala:
##########
@@ -2083,6 +2084,14 @@ object CelebornConf extends Logging {
       .intConf
       .createWithDefault(16)
 
+  val WORKER_STORAGE_EXPIRE_DIR_TIMEOUT: ConfigEntry[Long] =
+    buildConf("celeborn.worker.storage.expireDirs.timeout")
+      .categories("worker")
+      .version("0.3.1")

Review Comment:
   ```suggestion
         .version("0.3.2")
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CELEBORN-1046] Add an expiration time configuration for app directory to clean up [incubator-celeborn]

Posted by "waitinfuture (via GitHub)" <gi...@apache.org>.
waitinfuture commented on code in PR #1998:
URL: https://github.com/apache/incubator-celeborn/pull/1998#discussion_r1361773806


##########
common/src/main/scala/org/apache/celeborn/common/CelebornConf.scala:
##########
@@ -2083,6 +2084,14 @@ object CelebornConf extends Logging {
       .intConf
       .createWithDefault(16)
 
+  val WORKER_STORAGE_EXPIRE_DIR_TIMEOUT: ConfigEntry[Long] =
+    buildConf("celeborn.worker.storage.expireDirs.timeout")
+      .categories("worker")
+      .version("0.3.1")

Review Comment:
   0.3.2



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CELEBORN-1046] Add an expiration time configuration for app directory to clean up [incubator-celeborn]

Posted by "wilsonjie (via GitHub)" <gi...@apache.org>.
wilsonjie commented on code in PR #1998:
URL: https://github.com/apache/incubator-celeborn/pull/1998#discussion_r1361821706


##########
common/src/main/scala/org/apache/celeborn/common/CelebornConf.scala:
##########
@@ -2083,6 +2084,14 @@ object CelebornConf extends Logging {
       .intConf
       .createWithDefault(16)
 
+  val WORKER_STORAGE_EXPIRE_DIR_TIMEOUT: ConfigEntry[Long] =
+    buildConf("celeborn.worker.storage.expireDirs.timeout")
+      .categories("worker")
+      .version("0.3.1")
+      .doc(s"The timeout for a expire dirs to be deleted on disk.")
+      .timeConf(TimeUnit.MILLISECONDS)
+      .createWithDefaultString("6h")

Review Comment:
   Can it be consistent with the expiration time of hdfs dir ?   celeborn.master.hdfs.expireDirs.timeout=1h
   
   Because there is no default value, it is very easy to reproduce the problem.
   
   
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CELEBORN-1046] Add an expiration time configuration for app directory to clean up [incubator-celeborn]

Posted by "SteNicholas (via GitHub)" <gi...@apache.org>.
SteNicholas commented on code in PR #1998:
URL: https://github.com/apache/incubator-celeborn/pull/1998#discussion_r1361793651


##########
common/src/main/scala/org/apache/celeborn/common/CelebornConf.scala:
##########
@@ -2083,6 +2084,14 @@ object CelebornConf extends Logging {
       .intConf
       .createWithDefault(16)
 
+  val WORKER_STORAGE_EXPIRE_DIR_TIMEOUT: ConfigEntry[Long] =
+    buildConf("celeborn.worker.storage.expireDirs.timeout")
+      .categories("worker")
+      .version("0.3.1")
+      .doc(s"The timeout for a expire dirs to be deleted on disk.")
+      .timeConf(TimeUnit.MILLISECONDS)
+      .createWithDefaultString("6h")

Review Comment:
   @wilsonjie, could this config have no default value? Because this value of this config has no recommended value and has different value in various environment.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CELEBORN-1046] Add an expiration time configuration for app directory to clean up [incubator-celeborn]

Posted by "SteNicholas (via GitHub)" <gi...@apache.org>.
SteNicholas commented on code in PR #1998:
URL: https://github.com/apache/incubator-celeborn/pull/1998#discussion_r1361793651


##########
common/src/main/scala/org/apache/celeborn/common/CelebornConf.scala:
##########
@@ -2083,6 +2084,14 @@ object CelebornConf extends Logging {
       .intConf
       .createWithDefault(16)
 
+  val WORKER_STORAGE_EXPIRE_DIR_TIMEOUT: ConfigEntry[Long] =
+    buildConf("celeborn.worker.storage.expireDirs.timeout")
+      .categories("worker")
+      .version("0.3.1")
+      .doc(s"The timeout for a expire dirs to be deleted on disk.")
+      .timeConf(TimeUnit.MILLISECONDS)
+      .createWithDefaultString("6h")

Review Comment:
   @wilsonjie, could this config have no default value?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CELEBORN-1046] Add an expiration time configuration for app directory to clean up [incubator-celeborn]

Posted by "wilsonjie (via GitHub)" <gi...@apache.org>.
wilsonjie commented on code in PR #1998:
URL: https://github.com/apache/incubator-celeborn/pull/1998#discussion_r1361821706


##########
common/src/main/scala/org/apache/celeborn/common/CelebornConf.scala:
##########
@@ -2083,6 +2084,14 @@ object CelebornConf extends Logging {
       .intConf
       .createWithDefault(16)
 
+  val WORKER_STORAGE_EXPIRE_DIR_TIMEOUT: ConfigEntry[Long] =
+    buildConf("celeborn.worker.storage.expireDirs.timeout")
+      .categories("worker")
+      .version("0.3.1")
+      .doc(s"The timeout for a expire dirs to be deleted on disk.")
+      .timeConf(TimeUnit.MILLISECONDS)
+      .createWithDefaultString("6h")

Review Comment:
   Can it be consistent with the expiration time of hdfs dir ?   celeborn.master.hdfs.expireDirs.timeout=1h
   
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CELEBORN-1046] Add an expiration time configuration for app directory to clean up [incubator-celeborn]

Posted by "waitinfuture (via GitHub)" <gi...@apache.org>.
waitinfuture closed pull request #1998: [CELEBORN-1046] Add an expiration time configuration for app directory to clean up
URL: https://github.com/apache/incubator-celeborn/pull/1998


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org