[jira] [Updated] (SPARK-58693) Do not abort K8s shuffle data recovery when a directory listing returns null

["ASF GitHub Bot (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-58693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

ASF GitHub Bot updated SPARK-58693:
-----------------------------------
    Labels: pull-request-available  (was: )

> Do not abort K8s shuffle data recovery when a directory listing returns null
> ----------------------------------------------------------------------------
>
>                 Key: SPARK-58693
>                 URL: https://issues.apache.org/jira/browse/SPARK-58693
>             Project: Spark
>          Issue Type: Bug
>          Components: Kubernetes
>    Affects Versions: 5.0.0
>            Reporter: Yang Jie
>            Priority: Major
>              Labels: pull-request-available
>
> `KubernetesLocalDiskShuffleExecutorComponents.recoverDiskStore` collects candidate files with a strict `Array.map`/`flatMap` chain that never handles `File.listFiles` returning null. A single bad entry throws, `initializeExecutor` swallows the exception via `Utils.tryLogNonFatalError`, and the executor starts with nothing recovered.
> Two reachable cases:
> 1. A configured local directory with fewer than three path components. The scan walks two levels up with `new File(new File(new File(s).getParent).getParent)`; for `/data` the second `getParent` returns null and the `File` constructor throws. `/data` is the mount path used by the Local Storage example in `running-on-kubernetes.md`, two blocks above the recovery instructions. Since `map` is strict, one such entry also discards every correctly nested directory configured alongside it. An empty string behaves the same way, and the existing `filter(_ != null)` does not stop it because `String.split(",")` yields `""` rather than null.
> 2. A directory in the walk whose `listFiles()` returns null: removed while the walk is in progress, or present but not readable by the executor uid.
> The only null check in the method, `if (files != null)` applied to the result of `Array.flatMap`, can never fire, so the walk reads as null-safe when it is not.
> In both cases the executor logs one NPE stack trace that does not mention shuffle recovery, and the job recomputes every map output on the reused PVC. `SPARK-40459` already established that a single file should not block the rest of the recovery; the collection phase is still all-or-nothing.
> Fix: route every listing through a helper that logs and skips when `listFiles` returns null, tolerate a shallow local directory by skipping it with a warning, and drop the dead guard. This mirrors what `SPARK-57530` did for `SparkFileUtils.recursiveList`.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org