[jira] [Assigned] (SPARK-55071) Make spark.addArtifact work with Windows paths

["Herman van Hövell (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-55071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Herman van Hövell reassigned SPARK-55071:
-----------------------------------------

    Assignee: Alex Khakhlyuk

> Make spark.addArtifact work with Windows paths
> ----------------------------------------------
>
>                 Key: SPARK-55071
>                 URL: https://issues.apache.org/jira/browse/SPARK-55071
>             Project: Spark
>          Issue Type: Bug
>          Components: Connect, PySpark
>    Affects Versions: 4.1.1
>            Reporter: Alex Khakhlyuk
>            Assignee: Alex Khakhlyuk
>            Priority: Major
>              Labels: pull-request-available
>
> Currently, spark.addArtifact in pyspark connect does not support absolute Windows paths.
> E.g. this code
>  
> {code:java}
> spark.addArtifact("C:\\path\\to\\file.py", pyfile=True){code}
>  
> will result in the following error
>  
> {code:java}
> PySparkRuntimeError: [UNSUPPORTED_OPERATION] c scheme is not supported. {code}
>  
> This error is caused by urlparse function in [artifact.py.|https://github.com/apache/spark/blob/ac13473fff64919e8e7756e3a42ce3a68627dd73/python/pyspark/sql/connect/client/artifact.py#L188] It incorrectly interprets local Windows path, e.g. `C:...`, as a URI with 'C' scheme and throws an error because this URI scheme is not known and not supported.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org