You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Jan Algermissen <al...@icloud.com> on 2015/08/23 12:56:59 UTC
How to set environment of worker applications
Hi,
I am starting a spark streaming job in standalone mode with spark-submit.
Is there a way to make the UNIX environment variables with which spark-submit is started available to the processes started on the worker nodes?
Jan
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: How to set environment of worker applications
Posted by Jan Algermissen <al...@icloud.com>.
Finally, I found the solution:
on the spark context you can set spark.executorEnv.[EnvironmentVariableName] and these will be available in the environment of the executors
This is in fact documented, but somehow I missed it.
https://spark.apache.org/docs/latest/configuration.html#runtime-environment
Thanks for all the help though.
Jan
On 25 Aug 2015, at 08:57, Hemant Bhanawat <he...@gmail.com> wrote:
> Ok, I went in the direction of system vars since beginning probably because the question was to pass variables to a particular job.
>
> Anyway, the decision to use either system vars or environment vars would solely depend on whether you want to make them available to all the spark processes on a node or to a particular job.
>
> Are there any other reasons why one would prefer one over the other?
>
>
> On Mon, Aug 24, 2015 at 8:48 PM, Raghavendra Pandey <ra...@gmail.com> wrote:
> System properties and environment variables are two different things.. One can use spark.executor.extraJavaOptions to pass system properties and spark-env.sh to pass environment variables.
>
> -raghav
>
> On Mon, Aug 24, 2015 at 1:00 PM, Hemant Bhanawat <he...@gmail.com> wrote:
> That's surprising. Passing the environment variables using spark.executor.extraJavaOptions=-Dmyenvvar=xxx to the executor and then fetching them using System.getProperty("myenvvar") has worked for me.
>
> What is the error that you guys got?
>
> On Mon, Aug 24, 2015 at 12:10 AM, Sathish Kumaran Vairavelu <vs...@gmail.com> wrote:
> spark-env.sh works for me in Spark 1.4 but not spark.executor.extraJavaOptions.
>
> On Sun, Aug 23, 2015 at 11:27 AM Raghavendra Pandey <ra...@gmail.com> wrote:
> I think the only way to pass on environment variables to worker node is to write it in spark-env.sh file on each worker node.
>
> On Sun, Aug 23, 2015 at 8:16 PM, Hemant Bhanawat <he...@gmail.com> wrote:
> Check for spark.driver.extraJavaOptions and spark.executor.extraJavaOptions in the following article. I think you can use -D to pass system vars:
>
> spark.apache.org/docs/latest/configuration.html#runtime-environment
>
> Hi,
>
> I am starting a spark streaming job in standalone mode with spark-submit.
>
> Is there a way to make the UNIX environment variables with which spark-submit is started available to the processes started on the worker nodes?
>
> Jan
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>
>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: How to set environment of worker applications
Posted by Hemant Bhanawat <he...@gmail.com>.
Ok, I went in the direction of system vars since beginning probably because
the question was to pass variables to a particular job.
Anyway, the decision to use either system vars or environment vars would
solely depend on whether you want to make them available to all the spark
processes on a node or to a particular job.
Are there any other reasons why one would prefer one over the other?
On Mon, Aug 24, 2015 at 8:48 PM, Raghavendra Pandey <
raghavendra.pandey@gmail.com> wrote:
> System properties and environment variables are two different things.. One
> can use spark.executor.extraJavaOptions to pass system properties and
> spark-env.sh to pass environment variables.
>
> -raghav
>
> On Mon, Aug 24, 2015 at 1:00 PM, Hemant Bhanawat <he...@gmail.com>
> wrote:
>
>> That's surprising. Passing the environment variables using
>> spark.executor.extraJavaOptions=-Dmyenvvar=xxx to the executor and then
>> fetching them using System.getProperty("myenvvar") has worked for me.
>>
>> What is the error that you guys got?
>>
>> On Mon, Aug 24, 2015 at 12:10 AM, Sathish Kumaran Vairavelu <
>> vsathishkumaran@gmail.com> wrote:
>>
>>> spark-env.sh works for me in Spark 1.4 but not
>>> spark.executor.extraJavaOptions.
>>>
>>> On Sun, Aug 23, 2015 at 11:27 AM Raghavendra Pandey <
>>> raghavendra.pandey@gmail.com> wrote:
>>>
>>>> I think the only way to pass on environment variables to worker node is
>>>> to write it in spark-env.sh file on each worker node.
>>>>
>>>> On Sun, Aug 23, 2015 at 8:16 PM, Hemant Bhanawat <he...@gmail.com>
>>>> wrote:
>>>>
>>>>> Check for spark.driver.extraJavaOptions and
>>>>> spark.executor.extraJavaOptions in the following article. I think you can
>>>>> use -D to pass system vars:
>>>>>
>>>>> spark.apache.org/docs/latest/configuration.html#runtime-environment
>>>>> Hi,
>>>>>
>>>>> I am starting a spark streaming job in standalone mode with
>>>>> spark-submit.
>>>>>
>>>>> Is there a way to make the UNIX environment variables with which
>>>>> spark-submit is started available to the processes started on the worker
>>>>> nodes?
>>>>>
>>>>> Jan
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>>>>> For additional commands, e-mail: user-help@spark.apache.org
>>>>>
>>>>>
>>>>
>>
>
Re: How to set environment of worker applications
Posted by Raghavendra Pandey <ra...@gmail.com>.
System properties and environment variables are two different things.. One
can use spark.executor.extraJavaOptions to pass system properties and
spark-env.sh to pass environment variables.
-raghav
On Mon, Aug 24, 2015 at 1:00 PM, Hemant Bhanawat <he...@gmail.com>
wrote:
> That's surprising. Passing the environment variables using
> spark.executor.extraJavaOptions=-Dmyenvvar=xxx to the executor and then
> fetching them using System.getProperty("myenvvar") has worked for me.
>
> What is the error that you guys got?
>
> On Mon, Aug 24, 2015 at 12:10 AM, Sathish Kumaran Vairavelu <
> vsathishkumaran@gmail.com> wrote:
>
>> spark-env.sh works for me in Spark 1.4 but not
>> spark.executor.extraJavaOptions.
>>
>> On Sun, Aug 23, 2015 at 11:27 AM Raghavendra Pandey <
>> raghavendra.pandey@gmail.com> wrote:
>>
>>> I think the only way to pass on environment variables to worker node is
>>> to write it in spark-env.sh file on each worker node.
>>>
>>> On Sun, Aug 23, 2015 at 8:16 PM, Hemant Bhanawat <he...@gmail.com>
>>> wrote:
>>>
>>>> Check for spark.driver.extraJavaOptions and
>>>> spark.executor.extraJavaOptions in the following article. I think you can
>>>> use -D to pass system vars:
>>>>
>>>> spark.apache.org/docs/latest/configuration.html#runtime-environment
>>>> Hi,
>>>>
>>>> I am starting a spark streaming job in standalone mode with
>>>> spark-submit.
>>>>
>>>> Is there a way to make the UNIX environment variables with which
>>>> spark-submit is started available to the processes started on the worker
>>>> nodes?
>>>>
>>>> Jan
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>>>> For additional commands, e-mail: user-help@spark.apache.org
>>>>
>>>>
>>>
>
Re: How to set environment of worker applications
Posted by Hemant Bhanawat <he...@gmail.com>.
That's surprising. Passing the environment variables using
spark.executor.extraJavaOptions=-Dmyenvvar=xxx to the executor and then
fetching them using System.getProperty("myenvvar") has worked for me.
What is the error that you guys got?
On Mon, Aug 24, 2015 at 12:10 AM, Sathish Kumaran Vairavelu <
vsathishkumaran@gmail.com> wrote:
> spark-env.sh works for me in Spark 1.4 but not
> spark.executor.extraJavaOptions.
>
> On Sun, Aug 23, 2015 at 11:27 AM Raghavendra Pandey <
> raghavendra.pandey@gmail.com> wrote:
>
>> I think the only way to pass on environment variables to worker node is
>> to write it in spark-env.sh file on each worker node.
>>
>> On Sun, Aug 23, 2015 at 8:16 PM, Hemant Bhanawat <he...@gmail.com>
>> wrote:
>>
>>> Check for spark.driver.extraJavaOptions and
>>> spark.executor.extraJavaOptions in the following article. I think you can
>>> use -D to pass system vars:
>>>
>>> spark.apache.org/docs/latest/configuration.html#runtime-environment
>>> Hi,
>>>
>>> I am starting a spark streaming job in standalone mode with spark-submit.
>>>
>>> Is there a way to make the UNIX environment variables with which
>>> spark-submit is started available to the processes started on the worker
>>> nodes?
>>>
>>> Jan
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>>> For additional commands, e-mail: user-help@spark.apache.org
>>>
>>>
>>
Re: How to set environment of worker applications
Posted by Sathish Kumaran Vairavelu <vs...@gmail.com>.
spark-env.sh works for me in Spark 1.4 but not
spark.executor.extraJavaOptions.
On Sun, Aug 23, 2015 at 11:27 AM Raghavendra Pandey <
raghavendra.pandey@gmail.com> wrote:
> I think the only way to pass on environment variables to worker node is to
> write it in spark-env.sh file on each worker node.
>
> On Sun, Aug 23, 2015 at 8:16 PM, Hemant Bhanawat <he...@gmail.com>
> wrote:
>
>> Check for spark.driver.extraJavaOptions and
>> spark.executor.extraJavaOptions in the following article. I think you can
>> use -D to pass system vars:
>>
>> spark.apache.org/docs/latest/configuration.html#runtime-environment
>> Hi,
>>
>> I am starting a spark streaming job in standalone mode with spark-submit.
>>
>> Is there a way to make the UNIX environment variables with which
>> spark-submit is started available to the processes started on the worker
>> nodes?
>>
>> Jan
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>> For additional commands, e-mail: user-help@spark.apache.org
>>
>>
>
Re: How to set environment of worker applications
Posted by Raghavendra Pandey <ra...@gmail.com>.
I think the only way to pass on environment variables to worker node is to
write it in spark-env.sh file on each worker node.
On Sun, Aug 23, 2015 at 8:16 PM, Hemant Bhanawat <he...@gmail.com>
wrote:
> Check for spark.driver.extraJavaOptions and
> spark.executor.extraJavaOptions in the following article. I think you can
> use -D to pass system vars:
>
> spark.apache.org/docs/latest/configuration.html#runtime-environment
> Hi,
>
> I am starting a spark streaming job in standalone mode with spark-submit.
>
> Is there a way to make the UNIX environment variables with which
> spark-submit is started available to the processes started on the worker
> nodes?
>
> Jan
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>
Re: How to set environment of worker applications
Posted by Hemant Bhanawat <he...@gmail.com>.
Check for spark.driver.extraJavaOptions and spark.executor.extraJavaOptions
in the following article. I think you can use -D to pass system vars:
spark.apache.org/docs/latest/configuration.html#runtime-environment
Hi,
I am starting a spark streaming job in standalone mode with spark-submit.
Is there a way to make the UNIX environment variables with which
spark-submit is started available to the processes started on the worker
nodes?
Jan
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org