[GitHub] [spark] HyukjinKwon commented on a diff in pull request #37203: [SPARK-39755][CORE] Randomization in Spark local directory for other resource managers

[GitBox <gi...@apache.org>]

HyukjinKwon commented on code in PR #37203:
URL: https://github.com/apache/spark/pull/37203#discussion_r928339985


##########
core/src/main/scala/org/apache/spark/util/Utils.scala:
##########
@@ -920,9 +920,9 @@ private[spark] object Utils extends Logging {
       // user has access to them.
       randomizeInPlace(getYarnLocalDirs(conf).split(","))
     } else if (conf.getenv("SPARK_EXECUTOR_DIRS") != null) {
-      conf.getenv("SPARK_EXECUTOR_DIRS").split(File.pathSeparator)
+      randomizeInPlace(conf.getenv("SPARK_EXECUTOR_DIRS").split(File.pathSeparator))

Review Comment:
   Hm, quick question though. `getConfiguredLocalDirs` gets invoked multiple times, and won't it return a different directory every time when it's called?
   
   This `spark.local.dir` can be used to pass `spark.files` around, e.g., `SparkFiles.get`. Wonder if that would be affected.
   
   If this is a K8S specific issue, I think it's best to localize the fix into K8S.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org