[jira] [Created] (SPARK-17256) spark-submit.cmd cannot work if path has space and cut off double-quoted arguments

["Quanmao Liu (JIRA)" <ji...@apache.org>]

Quanmao Liu created SPARK-17256:
-----------------------------------

             Summary: spark-submit.cmd cannot work if path has space and cut off double-quoted arguments
                 Key: SPARK-17256
                 URL: https://issues.apache.org/jira/browse/SPARK-17256
             Project: Spark
          Issue Type: Bug
          Components: Deploy
    Affects Versions: 2.0.0
         Environment: Windows
            Reporter: Quanmao Liu


The key problem is  { cmd /V /E /C "~%dp0spark-xxx.cmd" } cannot accept arguments has double quotes, even if itself of "spark-xxx.cmd".

Double quoted command in spark/bin/spark-submit.cmd (and spark-class.cmd etc) as following:
 cmd /V /E /C "%~dp0spark-submit2.cmd" %*
Will cut off the argument that double quoted, and not effect if full path of spark-submit.cmd has space.

Example case :
spark-submit.cmd --jars xxxx  "jdbc:mysql://localhost:3306/lzdb?user=guest&password=abc123" my_table

(1) Cannot just start spark-submit.cmd if it's full path has space, for example  "D:\opengit\spark\bin - Copy\spark-submit.cmd" :

D:\opengit\spark\bin - Copy>
spark-submit.cmd --jars just-to-start "jdbc:mysql://localhost:3306/lzdb?user=guest&password=abc123" my_table
---- will get erorr : 
'D:\opengit\spark\bin' is not recognized as an internal or external command,
operable program or batch file.
'password' is not recognized as an internal or external command,
operable program or batch file.

(2) spark-submit.cmd can be started but will cut off arguments if path's no space . for example of above but no space:
D:\opengit\spark\bin>
spark-submit.cmd --jars just-to-start "jdbc:mysql://localhost:3306/lzdb?user=guest&password=abc123" my_table

----- will get error :
'password' is not recognized as an internal or external command,
operable program or batch file.

(1) A solution for this is just removing the "cmd /V /E /C" in the spark/bin/xxx.cmd files , but this will has possible "pollution" to the environment variables in the same cmd window (and later derived cmd window like started in it by "start cmd").
But this will solve the space in path problem.

(2) A mitigation is checking before adding double quotes, and added if really need. This just works well if path no space.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org