[GitHub] [spark] dongjoon-hyun commented on a diff in pull request #37203: [SPARK-39755][CORE] Randomization in Spark local directory for K8 resource managers

[GitBox <gi...@apache.org>]

dongjoon-hyun commented on code in PR #37203:
URL: https://github.com/apache/spark/pull/37203#discussion_r943873697


##########
core/src/main/scala/org/apache/spark/util/Utils.scala:
##########
@@ -919,8 +925,13 @@ private[spark] object Utils extends Logging {
       // created the directories already, and that they are secured so that only the
       // user has access to them.
       randomizeInPlace(getYarnLocalDirs(conf).split(","))
-    } else if (conf.getenv("SPARK_EXECUTOR_DIRS") != null) {
-      conf.getenv("SPARK_EXECUTOR_DIRS").split(File.pathSeparator)
+    } else if (isRunningInK8sContainer(conf)) {
+      // Randomizing the shuffle location in case of K8s so that all disk get fair changes to
+      // get selected.
+      randomizeInPlace(conf.getenv("SPARK_LOCAL_DIRS").split(","))

Review Comment:
   > QQ: I see both `SPARK_LOCAL_DIRS` and `SPARK_EXECUTOR_DIRS` in use within k8s resource manager - do we not want to do this for `SPARK_EXECUTOR_DIRS` @dongjoon-hyun ?
   
   We want it, @mridulm . That was my first comment here.
   - https://github.com/apache/spark/pull/37203#pullrequestreview-1045944162



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org