[spark] branch branch-3.3 updated: [SPARK-38652][K8S] `uploadFileUri` should preserve file scheme

[do...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

dongjoon pushed a commit to branch branch-3.3
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-3.3 by this push:
     new d503629  [SPARK-38652][K8S] `uploadFileUri` should preserve file scheme
d503629 is described below

commit d503629d444f72a536d85ad1b7208f583ad30974
Author: Dongjoon Hyun <do...@apache.org>
AuthorDate: Wed Mar 30 08:26:41 2022 -0700

    [SPARK-38652][K8S] `uploadFileUri` should preserve file scheme
    
    ### What changes were proposed in this pull request?
    
    This PR replaces `new Path(fileUri.getPath)` with `new Path(fileUri)`.
    By using `Path` class constructor with URI parameter, we can preserve file scheme.
    
    ### Why are the changes needed?
    
    If we use, `Path` class constructor with `String` parameter, it loses file scheme information.
    Although the original code works so far, it fails at Apache Hadoop 3.3.2 and breaks dependency upload feature which is covered by K8s Minikube integration tests.
    
    ```scala
    test("uploadFileUri") {
       val fileUri = org.apache.spark.util.Utils.resolveURI("/tmp/1.txt")
       assert(new Path(fileUri).toString == "file:/private/tmp/1.txt")
       assert(new Path(fileUri.getPath).toString == "/private/tmp/1.txt")
    }
    ```
    
    ### Does this PR introduce _any_ user-facing change?
    
    No, this will prevent a regression at Apache Spark 3.3.0 instead.
    
    ### How was this patch tested?
    
    Pass the CIs.
    
    In addition, this PR and #36009 will recover K8s IT `DepsTestsSuite`.
    
    Closes #36010 from dongjoon-hyun/SPARK-38652.
    
    Authored-by: Dongjoon Hyun <do...@apache.org>
    Signed-off-by: Dongjoon Hyun <do...@apache.org>
    (cherry picked from commit cab8aa1c4fe66c4cb1b69112094a203a04758f76)
    Signed-off-by: Dongjoon Hyun <do...@apache.org>
---
 .../src/main/scala/org/apache/spark/deploy/k8s/KubernetesUtils.scala    | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/KubernetesUtils.scala b/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/KubernetesUtils.scala
index a05d07a..b9cf111 100644
--- a/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/KubernetesUtils.scala
+++ b/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/KubernetesUtils.scala
@@ -320,7 +320,7 @@ object KubernetesUtils extends Logging {
             fs.mkdirs(new Path(s"${uploadPath}/${randomDirName}"))
             val targetUri = s"${uploadPath}/${randomDirName}/${fileUri.getPath.split("/").last}"
             log.info(s"Uploading file: ${fileUri.getPath} to dest: $targetUri...")
-            uploadFileToHadoopCompatibleFS(new Path(fileUri.getPath), new Path(targetUri), fs)
+            uploadFileToHadoopCompatibleFS(new Path(fileUri), new Path(targetUri), fs)
             targetUri
           } catch {
             case e: Exception =>

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org