[jira] [Assigned] (SPARK-18444) SparkR running in yarn-cluster mode should not download Spark package

["Apache Spark (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-18444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Apache Spark reassigned SPARK-18444:
------------------------------------

    Assignee: Apache Spark

> SparkR running in yarn-cluster mode should not download Spark package
> ---------------------------------------------------------------------
>
>                 Key: SPARK-18444
>                 URL: https://issues.apache.org/jira/browse/SPARK-18444
>             Project: Spark
>          Issue Type: Bug
>          Components: SparkR
>            Reporter: Yanbo Liang
>            Assignee: Apache Spark
>
> When running SparkR job in yarn-cluster mode, it will download Spark package from apache website which is not necessary.
> {code}
> ./bin/spark-submit --master yarn-cluster ./examples/src/main/r/dataframe.R
> {code}
> The following is output:
> {code}
> Attaching package: ‘SparkR’
> The following objects are masked from ‘package:stats’:
>     cov, filter, lag, na.omit, predict, sd, var, window
> The following objects are masked from ‘package:base’:
>     as.data.frame, colnames, colnames<-, drop, endsWith, intersect,
>     rank, rbind, sample, startsWith, subset, summary, transform, union
> Spark not found in SPARK_HOME:
> Spark not found in the cache directory. Installation will start.
> MirrorUrl not provided.
> Looking for preferred site from apache website...
> ......
> {code} 
> There's no {{SPARK_HOME}} in yarn-cluster mode since the R process is in a remote host of the yarn cluster rather than in the client host. The JVM comes up first and the R process then connects to it. So in such cases we should never have to download Spark as Spark is already running.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org