[jira] [Created] (SPARK-2221) Spark fails on windows YARN (HortonWorks 2.4)

["Vlad Rabenok (JIRA)" <ji...@apache.org>]

Vlad Rabenok created SPARK-2221:
-----------------------------------

             Summary: Spark fails on windows YARN (HortonWorks 2.4)
                 Key: SPARK-2221
                 URL: https://issues.apache.org/jira/browse/SPARK-2221
             Project: Spark
          Issue Type: Bug
          Components: YARN
    Affects Versions: 1.0.0
         Environment: Windows 2008 R2
HortonWorks Hadoop cluster 2.4
Spark 1.0
            Reporter: Vlad Rabenok


Windows based Yarn cluster failed to execute submitted Spark job.
Can be reproduced by submitting JavaSparkPi  on  Yarn  cluster.

C:\hdp\spark-1.0.0-bin-hadoop2\bin>spark-submit --class org.apache.spark.examples.JavaSparkPi ./../lib/spark-examples-1.0.0-hadoop2.2.0.jar --master yarn-cluster --deploy-mode cluster

The origin of the problems  is  org.apache.spark.deploy.yarn.ExecutorRunnableUtil.scala. JavaOpts  parameters that contain environment variable need to be quoted.
- '%' should be escaped  like  "kill %%p" . Still doesn't have any sense on Windows. Ideally should be passed through "spark.executor.extraJavaOptions".
- javaOpts += "-Djava.io.tmpdir=\"" + new Path(Environment.PWD.$(), YarnConfiguration.DEFAULT_CONTAINER_TEMP_DIR) + "\""




--
This message was sent by Atlassian JIRA
(v6.2#6252)