You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Ryan Williams <ry...@gmail.com> on 2016/08/18 17:49:35 UTC
Setting YARN executors' JAVA_HOME
I need to tell YARN a JAVA_HOME to use when spawning containers (to run a
Java 8 app on Java 7 YARN).
The only way I've found that works is
setting SPARK_YARN_USER_ENV="JAVA_HOME=/path/to/java8".
The code
<https://github.com/apache/spark/blob/b72bb62d421840f82d663c6b8e3922bd14383fbb/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala#L762>
implies that this is deprecated and users should use "the config", but I
can't figure out what config is being referenced.
Passing "--conf spark.yarn.appMasterEnv.JAVA_HOME=/path/to/java8" seems to
set it for the AM but not for executors.
Likewise, spark.executor.extraLibraryPath and spark.driver.extraLibraryPath
don't appear to set JAVA_HOME (and maybe aren't even supposed to?).
The 1.0.1 docs
<https://spark.apache.org/docs/1.0.1/running-on-yarn.html#environment-variables>
are the last ones to reference the SPARK_YARN_USER_ENV var, afaict.
What's the preferred way of passing YARN a custom JAVA_HOME that will be
applied to executors' containers?
Thanks!
Re: Setting YARN executors' JAVA_HOME
Posted by Ryan Williams <ry...@gmail.com>.
Ah, I guess I missed that by only looking in the YARN config docs, but this
is a more general parameter and not documented there. Thanks!
On Thu, Aug 18, 2016 at 2:51 PM dhruve ashar <dh...@gmail.com> wrote:
> Hi Ryan,
>
> You can get more info on this here: Spark documentation
> <http://spark.apache.org/docs/latest/configuration.html>.
>
> The page addresses what you need. You can look for spark.executorEnv.[EnvironmentVariableName]
> and set your java home as
> spark.executorEnv.JAVA_HOME=....
>
> Regards,
> Dhruve
>
> On Thu, Aug 18, 2016 at 12:49 PM, Ryan Williams <
> ryan.blake.williams@gmail.com> wrote:
>
>> I need to tell YARN a JAVA_HOME to use when spawning containers (to run a
>> Java 8 app on Java 7 YARN).
>>
>> The only way I've found that works is
>> setting SPARK_YARN_USER_ENV="JAVA_HOME=/path/to/java8".
>>
>> The code
>> <https://github.com/apache/spark/blob/b72bb62d421840f82d663c6b8e3922bd14383fbb/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala#L762>
>> implies that this is deprecated and users should use "the config", but I
>> can't figure out what config is being referenced.
>>
>> Passing "--conf spark.yarn.appMasterEnv.JAVA_HOME=/path/to/java8" seems
>> to set it for the AM but not for executors.
>>
>> Likewise, spark.executor.extraLibraryPath and
>> spark.driver.extraLibraryPath don't appear to set JAVA_HOME (and maybe
>> aren't even supposed to?).
>>
>> The 1.0.1 docs
>> <https://spark.apache.org/docs/1.0.1/running-on-yarn.html#environment-variables>
>> are the last ones to reference the SPARK_YARN_USER_ENV var, afaict.
>>
>> What's the preferred way of passing YARN a custom JAVA_HOME that will be
>> applied to executors' containers?
>>
>> Thanks!
>>
>
>
>
> --
> -Dhruve Ashar
>
>
Re: Setting YARN executors' JAVA_HOME
Posted by dhruve ashar <dh...@gmail.com>.
Hi Ryan,
You can get more info on this here: Spark documentation
<http://spark.apache.org/docs/latest/configuration.html>.
The page addresses what you need. You can look for
spark.executorEnv.[EnvironmentVariableName]
and set your java home as
spark.executorEnv.JAVA_HOME=....
Regards,
Dhruve
On Thu, Aug 18, 2016 at 12:49 PM, Ryan Williams <
ryan.blake.williams@gmail.com> wrote:
> I need to tell YARN a JAVA_HOME to use when spawning containers (to run a
> Java 8 app on Java 7 YARN).
>
> The only way I've found that works is setting SPARK_YARN_USER_ENV="
> JAVA_HOME=/path/to/java8".
>
> The code
> <https://github.com/apache/spark/blob/b72bb62d421840f82d663c6b8e3922bd14383fbb/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala#L762>
> implies that this is deprecated and users should use "the config", but I
> can't figure out what config is being referenced.
>
> Passing "--conf spark.yarn.appMasterEnv.JAVA_HOME=/path/to/java8" seems
> to set it for the AM but not for executors.
>
> Likewise, spark.executor.extraLibraryPath and
> spark.driver.extraLibraryPath don't appear to set JAVA_HOME (and maybe
> aren't even supposed to?).
>
> The 1.0.1 docs
> <https://spark.apache.org/docs/1.0.1/running-on-yarn.html#environment-variables>
> are the last ones to reference the SPARK_YARN_USER_ENV var, afaict.
>
> What's the preferred way of passing YARN a custom JAVA_HOME that will be
> applied to executors' containers?
>
> Thanks!
>
--
-Dhruve Ashar