[jira] [Created] (SPARK-58969) [K8S] Local dependency uploads lose URI fragment aliases

["Kishan Ghetia (Jira)" <ji...@apache.org>]

Kishan Ghetia created SPARK-58969:
-------------------------------------

             Summary: [K8S] Local dependency uploads lose URI fragment aliases
                 Key: SPARK-58969
                 URL: https://issues.apache.org/jira/browse/SPARK-58969
             Project: Spark
          Issue Type: Bug
          Components: Bug
    Affects Versions: 4.1.2
            Reporter: Kishan Ghetia


Spark's Kubernetes submission path drops URI fragments from locally uploaded dependencies.

Spark accepts dependency URIs in the form `source#alias`. The fragment defines
the application-visible name. For example:

--files /tmp/query-random.sql#query.sql

The physical object should be uploaded as `query-random.sql`, while the resulting `spark.files` entry should retain `#query.sql`. This allows the driver to open `./query.sql` and executors to resolve `SparkFiles.get("query.sql")`.

On Kubernetes, `BasicDriverFeatureStep` passes local FILES, JARS, and
SUBMIT_PYTHON_FILES through `uploadAndTransformFileUris`, but does not restore
their fragments afterward. Archive fragments are handled separately and do not
have this problem.

This differs from YARN, where the distributed-cache implementation retains the
URI fragment and uses it as the localized link name.

The issue can be reproduced by running this test - [https://github.com/malware27/spark/commit/0fce035c761a17555b16b16d22141ac3a683e81a] 

 

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org