You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by do...@apache.org on 2024/03/01 03:08:49 UTC
(spark) branch branch-3.4 updated: [SPARK-47236][CORE] Fix `deleteRecursivelyUsingJavaIO` to skip non-existing file input
This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch branch-3.4
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.4 by this push:
new 58a4a49389a5 [SPARK-47236][CORE] Fix `deleteRecursivelyUsingJavaIO` to skip non-existing file input
58a4a49389a5 is described below
commit 58a4a49389a5f9979f7dabc5320116a212eb4bdb
Author: Dongjoon Hyun <dh...@apple.com>
AuthorDate: Thu Feb 29 19:08:15 2024 -0800
[SPARK-47236][CORE] Fix `deleteRecursivelyUsingJavaIO` to skip non-existing file input
### What changes were proposed in this pull request?
This PR aims to fix `deleteRecursivelyUsingJavaIO` to skip non-existing file input.
### Why are the changes needed?
`deleteRecursivelyUsingJavaIO` is a fallback of `deleteRecursivelyUsingUnixNative`.
We should have identical capability. Currently, it fails.
```
[info] java.nio.file.NoSuchFileException: /Users/dongjoon/APACHE/spark-merge/target/tmp/spark-e264d853-42c0-44a2-9a30-22049522b04f
[info] at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
[info] at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)
[info] at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
[info] at java.base/sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55)
[info] at java.base/sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:148)
[info] at java.base/java.nio.file.Files.readAttributes(Files.java:1851)
[info] at org.apache.spark.network.util.JavaUtils.deleteRecursivelyUsingJavaIO(JavaUtils.java:126)
```
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
This is difficult to test this `private static` Java method. I tested this with #45344 .
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #45346 from dongjoon-hyun/SPARK-47236.
Authored-by: Dongjoon Hyun <dh...@apple.com>
Signed-off-by: Dongjoon Hyun <dh...@apple.com>
(cherry picked from commit 1cd7bab5c5c2bd8d595b131c88e6576486dbf123)
Signed-off-by: Dongjoon Hyun <dh...@apple.com>
---
.../src/main/java/org/apache/spark/network/util/JavaUtils.java | 1 +
1 file changed, 1 insertion(+)
diff --git a/common/network-common/src/main/java/org/apache/spark/network/util/JavaUtils.java b/common/network-common/src/main/java/org/apache/spark/network/util/JavaUtils.java
index 7e410e9eab22..59744ec5748a 100644
--- a/common/network-common/src/main/java/org/apache/spark/network/util/JavaUtils.java
+++ b/common/network-common/src/main/java/org/apache/spark/network/util/JavaUtils.java
@@ -124,6 +124,7 @@ public class JavaUtils {
private static void deleteRecursivelyUsingJavaIO(
File file,
FilenameFilter filter) throws IOException {
+ if (!file.exists()) return;
BasicFileAttributes fileAttributes =
Files.readAttributes(file.toPath(), BasicFileAttributes.class);
if (fileAttributes.isDirectory() && !isSymlink(file)) {
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org