[jira] [Commented] (SPARK-6797) Add support for YARN cluster mode

["Sun Rui (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-6797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14568863#comment-14568863 ] 

Sun Rui commented on SPARK-6797:
--------------------------------

@shivaram, I can run SparkR in the YARN cluster mode (need to set SPARK_HOME for the AM). So could you explain in more detail what the problem is. For now, SparkR package has to be installed into the R environment on each worker node. Do you mean that we ship SparkR package source to AM and worker nodes and compile/install it for each application? or Do you mean  that we support something like spark-submit --rfiles (similar to --pyfiles) on which the primary R script depends?

> Add support for YARN cluster mode
> ---------------------------------
>
>                 Key: SPARK-6797
>                 URL: https://issues.apache.org/jira/browse/SPARK-6797
>             Project: Spark
>          Issue Type: Improvement
>          Components: SparkR
>            Reporter: Shivaram Venkataraman
>            Assignee: Sun Rui
>            Priority: Critical
>
> SparkR currently does not work in YARN cluster mode as the R package is not shipped along with the assembly jar to the YARN AM. We could try to use the support for archives in YARN to send out the R package as a zip file.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org