You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Chen Song <ch...@gmail.com> on 2014/07/17 23:05:26 UTC
how to pass extra Java opts to workers for spark streaming jobs
I am using spark 0.9.0 and I am able to submit job to YARN,
https://spark.apache.org/docs/0.9.0/running-on-yarn.html.
I am trying to turn on gc logging on executors but could not find a way to
set extra Java opts for workers.
I tried to set spark.executor.extraJavaOptions but that did not work.
Any idea on how I should do this?
--
Chen Song
Re: how to pass extra Java opts to workers for spark streaming jobs
Posted by Chen Song <ch...@gmail.com>.
Thanks Andrew, I tried and it works.
On Fri, Jul 18, 2014 at 12:53 AM, Andrew Or <an...@databricks.com> wrote:
> You will need to include that in the SPARK_JAVA_OPTS environment variable,
> so add the following line to spark-env.sh:
>
> export SPARK_JAVA_OPTS=" -XX:+UseConcMarkSweepGC"
>
> This should propagate to the executors. (Though you should double check,
> since 0.9 is a little old and I could be forgetting something) If you wish
> to add spark options in addition to this, simply append them to the
> environment variable:
>
> export SPARK_JAVA_OPTS=" -XX:+UseConcMarkSweepGC -Dspark.config.one=value
> -Dspark.config.two=value"
>
> (Please note that this is only for Spark 0.9. The part where we set Spark
> options within SPARK_JAVA_OPTS is deprecated as of 1.0)
>
>
> 2014-07-17 21:08 GMT-07:00 Chen Song <ch...@gmail.com>:
>
> Thanks Andrew.
>>
>> Say that I want to turn on CMS gc for each worker.
>>
>> All I need to do is add the following line to conf/spark-env.sh on node
>> where I submit the application.
>>
>> -XX:+UseConcMarkSweepGC
>>
>> Is that correct?
>>
>> Will this option be populated to each worker in yarn?
>>
>>
>>
>> On Thu, Jul 17, 2014 at 9:26 PM, Andrew Or <an...@databricks.com> wrote:
>>
>>> Hi Chen,
>>>
>>> spark.executor.extraJavaOptions is introduced in Spark 1.0, not in Spark
>>> 0.9. You need to
>>>
>>> export SPARK_JAVA_OPTS=" -Dspark.config1=value1 -Dspark.config2=value2"
>>>
>>> in conf/spark-env.sh.
>>>
>>> Let me know if that works.
>>> Andrew
>>>
>>>
>>> 2014-07-17 18:15 GMT-07:00 Tathagata Das <ta...@gmail.com>:
>>>
>>> Can you check in the environment tab of Spark web ui to see whether this
>>>> configuration parameter is in effect?
>>>>
>>>> TD
>>>>
>>>>
>>>> On Thu, Jul 17, 2014 at 2:05 PM, Chen Song <ch...@gmail.com>
>>>> wrote:
>>>>
>>>>> I am using spark 0.9.0 and I am able to submit job to YARN,
>>>>> https://spark.apache.org/docs/0.9.0/running-on-yarn.html.
>>>>>
>>>>> I am trying to turn on gc logging on executors but could not find a
>>>>> way to set extra Java opts for workers.
>>>>>
>>>>> I tried to set spark.executor.extraJavaOptions but that did not work.
>>>>>
>>>>> Any idea on how I should do this?
>>>>>
>>>>> --
>>>>> Chen Song
>>>>>
>>>>>
>>>>
>>>
>>
>>
>> --
>> Chen Song
>>
>>
>
--
Chen Song
Re: how to pass extra Java opts to workers for spark streaming jobs
Posted by Andrew Or <an...@databricks.com>.
You will need to include that in the SPARK_JAVA_OPTS environment variable,
so add the following line to spark-env.sh:
export SPARK_JAVA_OPTS=" -XX:+UseConcMarkSweepGC"
This should propagate to the executors. (Though you should double check,
since 0.9 is a little old and I could be forgetting something) If you wish
to add spark options in addition to this, simply append them to the
environment variable:
export SPARK_JAVA_OPTS=" -XX:+UseConcMarkSweepGC -Dspark.config.one=value
-Dspark.config.two=value"
(Please note that this is only for Spark 0.9. The part where we set Spark
options within SPARK_JAVA_OPTS is deprecated as of 1.0)
2014-07-17 21:08 GMT-07:00 Chen Song <ch...@gmail.com>:
> Thanks Andrew.
>
> Say that I want to turn on CMS gc for each worker.
>
> All I need to do is add the following line to conf/spark-env.sh on node
> where I submit the application.
>
> -XX:+UseConcMarkSweepGC
>
> Is that correct?
>
> Will this option be populated to each worker in yarn?
>
>
>
> On Thu, Jul 17, 2014 at 9:26 PM, Andrew Or <an...@databricks.com> wrote:
>
>> Hi Chen,
>>
>> spark.executor.extraJavaOptions is introduced in Spark 1.0, not in Spark
>> 0.9. You need to
>>
>> export SPARK_JAVA_OPTS=" -Dspark.config1=value1 -Dspark.config2=value2"
>>
>> in conf/spark-env.sh.
>>
>> Let me know if that works.
>> Andrew
>>
>>
>> 2014-07-17 18:15 GMT-07:00 Tathagata Das <ta...@gmail.com>:
>>
>> Can you check in the environment tab of Spark web ui to see whether this
>>> configuration parameter is in effect?
>>>
>>> TD
>>>
>>>
>>> On Thu, Jul 17, 2014 at 2:05 PM, Chen Song <ch...@gmail.com>
>>> wrote:
>>>
>>>> I am using spark 0.9.0 and I am able to submit job to YARN,
>>>> https://spark.apache.org/docs/0.9.0/running-on-yarn.html.
>>>>
>>>> I am trying to turn on gc logging on executors but could not find a way
>>>> to set extra Java opts for workers.
>>>>
>>>> I tried to set spark.executor.extraJavaOptions but that did not work.
>>>>
>>>> Any idea on how I should do this?
>>>>
>>>> --
>>>> Chen Song
>>>>
>>>>
>>>
>>
>
>
> --
> Chen Song
>
>
Re: how to pass extra Java opts to workers for spark streaming jobs
Posted by Chen Song <ch...@gmail.com>.
Thanks Andrew.
Say that I want to turn on CMS gc for each worker.
All I need to do is add the following line to conf/spark-env.sh on node
where I submit the application.
-XX:+UseConcMarkSweepGC
Is that correct?
Will this option be populated to each worker in yarn?
On Thu, Jul 17, 2014 at 9:26 PM, Andrew Or <an...@databricks.com> wrote:
> Hi Chen,
>
> spark.executor.extraJavaOptions is introduced in Spark 1.0, not in Spark
> 0.9. You need to
>
> export SPARK_JAVA_OPTS=" -Dspark.config1=value1 -Dspark.config2=value2"
>
> in conf/spark-env.sh.
>
> Let me know if that works.
> Andrew
>
>
> 2014-07-17 18:15 GMT-07:00 Tathagata Das <ta...@gmail.com>:
>
> Can you check in the environment tab of Spark web ui to see whether this
>> configuration parameter is in effect?
>>
>> TD
>>
>>
>> On Thu, Jul 17, 2014 at 2:05 PM, Chen Song <ch...@gmail.com>
>> wrote:
>>
>>> I am using spark 0.9.0 and I am able to submit job to YARN,
>>> https://spark.apache.org/docs/0.9.0/running-on-yarn.html.
>>>
>>> I am trying to turn on gc logging on executors but could not find a way
>>> to set extra Java opts for workers.
>>>
>>> I tried to set spark.executor.extraJavaOptions but that did not work.
>>>
>>> Any idea on how I should do this?
>>>
>>> --
>>> Chen Song
>>>
>>>
>>
>
--
Chen Song
Re: how to pass extra Java opts to workers for spark streaming jobs
Posted by Andrew Or <an...@databricks.com>.
Hi Chen,
spark.executor.extraJavaOptions is introduced in Spark 1.0, not in Spark
0.9. You need to
export SPARK_JAVA_OPTS=" -Dspark.config1=value1 -Dspark.config2=value2"
in conf/spark-env.sh.
Let me know if that works.
Andrew
2014-07-17 18:15 GMT-07:00 Tathagata Das <ta...@gmail.com>:
> Can you check in the environment tab of Spark web ui to see whether this
> configuration parameter is in effect?
>
> TD
>
>
> On Thu, Jul 17, 2014 at 2:05 PM, Chen Song <ch...@gmail.com> wrote:
>
>> I am using spark 0.9.0 and I am able to submit job to YARN,
>> https://spark.apache.org/docs/0.9.0/running-on-yarn.html.
>>
>> I am trying to turn on gc logging on executors but could not find a way
>> to set extra Java opts for workers.
>>
>> I tried to set spark.executor.extraJavaOptions but that did not work.
>>
>> Any idea on how I should do this?
>>
>> --
>> Chen Song
>>
>>
>
Re: how to pass extra Java opts to workers for spark streaming jobs
Posted by Tathagata Das <ta...@gmail.com>.
Can you check in the environment tab of Spark web ui to see whether this
configuration parameter is in effect?
TD
On Thu, Jul 17, 2014 at 2:05 PM, Chen Song <ch...@gmail.com> wrote:
> I am using spark 0.9.0 and I am able to submit job to YARN,
> https://spark.apache.org/docs/0.9.0/running-on-yarn.html.
>
> I am trying to turn on gc logging on executors but could not find a way to
> set extra Java opts for workers.
>
> I tried to set spark.executor.extraJavaOptions but that did not work.
>
> Any idea on how I should do this?
>
> --
> Chen Song
>
>