[jira] [Assigned] (SPARK-58857) LocalDirsFeatureStep discards the result of Utils.randomize, so configured local dirs are never shuffled

["Yang Jie (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-58857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Yang Jie reassigned SPARK-58857:
--------------------------------

    Assignee: Yang Jie

> LocalDirsFeatureStep discards the result of Utils.randomize, so configured local dirs are never shuffled
> --------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-58857
>                 URL: https://issues.apache.org/jira/browse/SPARK-58857
>             Project: Spark
>          Issue Type: Bug
>          Components: Kubernetes
>    Affects Versions: 5.0.0
>            Reporter: Yang Jie
>            Assignee: Yang Jie
>            Priority: Major
>              Labels: pull-request-available
>
> `LocalDirsFeatureStep.configurePod` calls `Utils.randomize` on the local dirs resolved from configuration but discards the result, so that branch never randomizes.
> {code}
> val resolvedLocalDirs = Option(conf.sparkConf.getenv("SPARK_LOCAL_DIRS"))
>   .orElse(conf.getOption("spark.local.dir"))
>   .getOrElse(defaultLocalDir)
>   .split(",")
> randomize(resolvedLocalDirs)                      // result dropped
> localDirs = resolvedLocalDirs.toImmutableArraySeq  // original order
> {code}
> `Utils.randomize[T](seq: IterableOnce[T]): Seq[T]` is `randomizeInPlace(seq.iterator.toArray).toImmutableArraySeq` - it shuffles a fresh copy and returns it, so the caller observes no change to the argument. The in-place variant is `Utils.randomizeInPlace`. The sibling call fifteen lines earlier, for the pod-template branch, binds the result correctly:
> {code}
> var localDirs = randomize(pod.container.getVolumeMounts.asScala
>   .filter(_.getName.startsWith("spark-local-dir-"))
>   .map(_.getMountPath))
> {code}
> Both calls arrived in the same commit, 9f9af2a7bf6 (SPARK-39755, "Improve LocalDirsFeatureStep to randomize local directories"). An earlier revision of that PR used `randomizeInPlace`; review suggested `Utils.randomize` instead, and the next revision adopted it at both sites, binding the result at one and dropping it at the other. During that review a committer twice noted the test did not fail as expected, which is the symptom of this bug, but the discrepancy was attributed to force-pushes and the assertions were loosened to accept either order.
> So `SPARK_LOCAL_DIRS`, the emptyDir volume list, and the `spark-local-dir-N` to path pairing are emitted in configured order, identically for every driver and executor pod in an application.
> Scope of the observable effect, so this is not oversold: no measurable I/O or capacity skew follows. `DiskBlockManager.getFile` picks a directory by `nonNegativeHash(filename) % localDirs.length`, which spreads files near-uniformly within each executor regardless of the order it receives, and in this branch every path is backed by an emptyDir volume this step creates, so all of them share one node filesystem (or RAM under `spark.kubernetes.local.dirs.tmpfs`). What is real is that the randomization SPARK-39755 added, and its JIRA and release note describe, does not happen on this branch, and the suite's `// SPARK-39755 : Changes the method to test randomization` comment describes behavior that never occurs.
> Fix: bind the result, matching the sibling call. Deleting the call as dead code is a defensible alternative if the randomization is not considered worth having.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org