You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Thomas Graves (JIRA)" <ji...@apache.org> on 2016/07/07 15:11:11 UTC
[jira] [Comment Edited] (SPARK-16265) Add option to SparkSubmit to
ship driver JRE to YARN
[ https://issues.apache.org/jira/browse/SPARK-16265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15366251#comment-15366251 ]
Thomas Graves edited comment on SPARK-16265 at 7/7/16 3:10 PM:
---------------------------------------------------------------
Have you tried shipping your jre as gzip file in the distributed cache and pointing to it with: spark.yarn.appMasterEnv.JAVA_HOME=<path-to-java> and spark.executorEnv.JAVA_HOME=<path-to-java>?
Not positive that works but you should try it first.
I'd rather not get into the business of installing software with Spark. It then starts to extend to things like I want to install python version x, I want to install R version Y, etc. If there is a generic way we can provide support for this I'm fine with it. For instance we already can ship most things in distributed cache and generically set any env variable, that makes it possible to configure many things.
Also as stated above generally in the yarn environment java would be one of those things installed on the clusters. yes we try to keep things isolated and you should ship as much stuff as possible with your application but at this point we also rely on the hadoop libraries and other things that you could have incompatibilities with. obviously docker or something like that could give you an entire environment.
was (Author: tgraves):
Have you tried shipping your jre as gzip file in the distributed cache and pointing to it with: spark.yarn.appMasterEnv.JAVA_HOME=<path-to-java> and spark.executorEnv.JAVA_HOME=<path-to-java>?
Not positive that works but you should try it first.
I'd rather not get into the business of installing software with Spark. It then starts to extend to things like I want to install python version x, I want to install R version Y, etc.
Also as stated above generally in the yarn environment java would be one of those things installed on the clusters. yes we try to keep things isolated and you should ship as much stuff as possible with your application but at this point we also rely on the hadoop libraries and other things that you could have incompatibilities with. obviously docker or something like that could give you an entire environment.
> Add option to SparkSubmit to ship driver JRE to YARN
> ----------------------------------------------------
>
> Key: SPARK-16265
> URL: https://issues.apache.org/jira/browse/SPARK-16265
> Project: Spark
> Issue Type: Improvement
> Affects Versions: 1.6.2
> Reporter: Andrew Duffy
>
> Add an option to {{SparkSubmit}} to allow the driver to package up it's version of the JRE to be shipped to a YARN cluster. This allows deploying Spark applications to a YARN cluster in which its required Java version need not match one of the versions already installed on the YARN cluster, useful in situations in which the Spark Application developer does not have administrative access over the YARN cluster (ex. school or corporate environment) but still wants to use certain language features in their code.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org