[jira] [Commented] (SPARK-1350) YARN ContainerLaunchContext should use cluster's JAVA_HOME

["Thomas Graves (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-1350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13955209#comment-13955209 ] 

Thomas Graves commented on SPARK-1350:
--------------------------------------

I assume you are referring to the one in ClientBase?

It actually does not use the same java as specified by JAVA_HOME on the client.  If JAVA_HOME is set on the client then it just adds $JAVA_HOME/bin to the java command to launch the AM. Yarn can be configured (and is the default config) to put JAVA_HOME into the container launch script, so the launch command will use the JAVA_HOME specified by yarn.   I agree its a bit hacky and could be improved though.  On MapReduce it just always uses "$JAVA_HOME/bin/java"

The main idea is that I want $JAVA_HOME to be included in the launch command so that its not relying on the default install of java on the system.  I want it to use the JAVA_HOME specified by yarn and I also want to allow it to be overridden so that a user could choose to run either 32 bit or 64 bit jdk.  

export SPARK_YARN_USER_ENV="JAVA_HOME=/myyarinstall/jdk64/current"

I was told this goes against how spark is in general setup where it relies on whatever is installed on the system so this was the compromise.



> YARN ContainerLaunchContext should use cluster's JAVA_HOME
> ----------------------------------------------------------
>
>                 Key: SPARK-1350
>                 URL: https://issues.apache.org/jira/browse/SPARK-1350
>             Project: Spark
>          Issue Type: Bug
>          Components: YARN
>    Affects Versions: 0.9.0
>            Reporter: Sandy Ryza
>            Assignee: Sandy Ryza
>             Fix For: 1.0.0
>
>
> {code}
>     var javaCommand = "java"
>     val javaHome = System.getenv("JAVA_HOME")
>     if ((javaHome != null && !javaHome.isEmpty()) || env.isDefinedAt("JAVA_HOME")) {
>       javaCommand = Environment.JAVA_HOME.$() + "/bin/java"
>     }
> {code}
> Currently, if JAVA_HOME is specified on the client, it will be used instead of the value given on the cluster.  This makes it so that Java must be installed in the same place on the client as on the cluster.
> This is a possibly incompatible change that we should get in before 1.0.



--
This message was sent by Atlassian JIRA
(v6.2#6252)