You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Sean Owen (JIRA)" <ji...@apache.org> on 2015/03/14 09:08:38 UTC
[jira] [Resolved] (SPARK-2221) Spark fails on windows YARN
(HortonWorks 2.4)
[ https://issues.apache.org/jira/browse/SPARK-2221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sean Owen resolved SPARK-2221.
------------------------------
Resolution: Duplicate
Same issue -- quoting and Windows shell. Resolving in favor of the other since it has more discussion
> Spark fails on windows YARN (HortonWorks 2.4)
> ---------------------------------------------
>
> Key: SPARK-2221
> URL: https://issues.apache.org/jira/browse/SPARK-2221
> Project: Spark
> Issue Type: Bug
> Components: YARN
> Affects Versions: 1.0.0
> Environment: Windows 2008 R2
> HortonWorks Hadoop cluster 2.4
> Spark 1.0
> Reporter: Vlad Rabenok
>
> Windows based Yarn cluster failed to execute submitted Spark job.
> Can be reproduced by submitting JavaSparkPi on Yarn cluster.
> C:\hdp\spark-1.0.0-bin-hadoop2\bin>spark-submit --class org.apache.spark.examples.JavaSparkPi ./../lib/spark-examples-1.0.0-hadoop2.2.0.jar --master yarn-cluster --deploy-mode cluster
> The origin of the problems is org.apache.spark.deploy.yarn.ExecutorRunnableUtil.scala. JavaOpts parameters that contain environment variable need to be quoted.
> - '%' should be escaped like "kill %%p" . Still doesn't have any sense on Windows. Ideally should be passed through "spark.executor.extraJavaOptions".
> - javaOpts += "-Djava.io.tmpdir=\"" + new Path(Environment.PWD.$(), YarnConfiguration.DEFAULT_CONTAINER_TEMP_DIR) + "\""
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org