[jira] [Commented] (SPARK-35124) Local mode fails to start cluster due to configuration value escape issue

["Malthe Borch (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-35124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17325228#comment-17325228 ] 

Malthe Borch commented on SPARK-35124:
--------------------------------------

The issue seems to stem from the way Python's "Popen" command handles arguments on Windows – basically it doesn't have a clue that "&" has a special meaning on Windows (and a number of other characters – in particular {{&}}, {{|}}, {{(}}, {{)}}, {{<}}, {{>}}, {{^}}).

> Local mode fails to start cluster due to configuration value escape issue
> -------------------------------------------------------------------------
>
>                 Key: SPARK-35124
>                 URL: https://issues.apache.org/jira/browse/SPARK-35124
>             Project: Spark
>          Issue Type: Bug
>          Components: Java API
>    Affects Versions: 3.1.1
>            Reporter: Malthe Borch
>            Priority: Major
>
> At least on Windows and perhaps also other systems, running Spark in local mode fails to start a cluster when a configuration key contains a value with "&" (ampersand) in it.
> This happens during the "getOrCreate()" call from a Spark session builder.
> The reason seems to be incorrect or insufficient escaping since on Windows, the attempt to start a Spark process actually ends up running multiple commands for each occurrence of the "&" character.
> On Windows specifically, the correct way to escape "&" would be "^&", but I have not been able yet to figure out exactly where the process is started and how the configuration is passed.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org