You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/03/03 21:59:06 UTC
[GitHub] [spark] hiboyang commented on a change in pull request #31715: [SPARK-34601][SHUFFLE] Add spark.shuffle.markFileLostOnExecutorLost to not delete shuffle file on executor lost event
hiboyang commented on a change in pull request #31715:
URL: https://github.com/apache/spark/pull/31715#discussion_r586824068
##########
File path: core/src/main/scala/org/apache/spark/internal/config/package.scala
##########
@@ -2117,4 +2117,12 @@ package object config {
// batch of block will be loaded in memory with memory mapping, which has higher overhead
// with small MB sized chunk of data.
.createWithDefaultString("3m")
+
+ private[spark] val MARK_FILE_LOST_ON_EXECUTOR_LOST =
+ ConfigBuilder("spark.shuffle.markFileLostOnExecutorLost")
+ .doc("Mark shuffle file lost when executor is lost. People could set this to false when " +
+ "using remote shuffle services because the shuffle file is not stored on the executor.")
Review comment:
This is good suggestion, will do the rewording, thanks!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org