You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "LuciferYang (via GitHub)" <gi...@apache.org> on 2026/06/30 11:14:20 UTC
[PR] [SPARK-57695][CORE] Make TestUtils.recursiveList null-safe by reusing Utils.recursiveList [spark]
LuciferYang opened a new pull request, #56901:
URL: https://github.com/apache/spark/pull/56901
### What changes were proposed in this pull request?
`TestUtils.recursiveList` duplicated the recursive directory walk already provided by `Utils.recursiveList` (i.e. `SparkFileUtils.recursiveList`). This PR removes the duplicated copy and delegates to `Utils.recursiveList`:
```scala
// before
def recursiveList(f: File): Array[File] = {
require(f.isDirectory)
val current = f.listFiles
current ++ current.filter(_.isDirectory).flatMap(recursiveList)
}
// after
def recursiveList(f: File): Array[File] = Utils.recursiveList(f)
```
### Why are the changes needed?
The duplicated implementation carried the same two issues that were just fixed in `SparkFileUtils.recursiveList` under [SPARK-57530](https://issues.apache.org/jira/browse/SPARK-57530):
1. It called `File.listFiles` without a null check, so an IO error (or the directory being removed mid-walk) would throw an NPE.
2. The `current ++ ... .flatMap(...)` form had no linear-time guarantee.
By delegating to `Utils.recursiveList`, `TestUtils.recursiveList` automatically picks up the null-safety (a directory that cannot be listed is skipped with a warning instead of throwing) and the O(n) traversal from SPARK-57530, and the duplicated logic is removed.
This is a follow-up to SPARK-57530, which is already merged to master.
### Does this PR introduce _any_ user-facing change?
No. `TestUtils` is a test-only `private[spark]` helper; this is an internal refactor with no behavior change for any successful directory walk.
### How was this patch tested?
Existing tests that use `TestUtils.recursiveList` continue to exercise it. The behavioral contract for a readable directory tree is unchanged (same set of files returned); the only difference is that an unreadable directory is now skipped with a warning rather than throwing, matching `Utils.recursiveList`.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57695][CORE] Make TestUtils.recursiveList null-safe by reusing Utils.recursiveList [spark]
Posted by "MaxGekk (via GitHub)" <gi...@apache.org>.
MaxGekk commented on PR #56901:
URL: https://github.com/apache/spark/pull/56901#issuecomment-4848525046
The failed CI is not related to the PR, I believe.
+1, LGTM. Merging to master/4.x.
Thank you, @LuciferYang.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57695][CORE] Make TestUtils.recursiveList null-safe by reusing Utils.recursiveList [spark]
Posted by "MaxGekk (via GitHub)" <gi...@apache.org>.
MaxGekk closed pull request #56901: [SPARK-57695][CORE] Make TestUtils.recursiveList null-safe by reusing Utils.recursiveList
URL: https://github.com/apache/spark/pull/56901
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org