You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Peng Cheng <pc...@uow.edu.au> on 2015/06/12 20:17:15 UTC
[Spark 1.4.0]How to set driver's system property using spark-submit
options?
In Spark <1.3.x, the system property of the driver can be set by --conf
option, shared between setting spark properties and system properties.
In Spark 1.4.0 this feature is removed, the driver instead log the following
warning:
Warning: Ignoring non-spark config property: xxx.xxx=vvvvv
How do set driver's system property in 1.4.0? Is there a reason it is
removed without a deprecation warning?
Thanks a lot for your advices.
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-1-4-0-How-to-set-driver-s-system-property-using-spark-submit-options-tp23298.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: [Spark 1.4.0]How to set driver's system property using
spark-submit options?
Posted by Peng Cheng <rh...@gmail.com>.
Thanks all for your information. Andrew, I dig out one of your old post
which is relevant:
http://apache-spark-user-list.1001560.n3.nabble.com/little-confused-about-SPARK-JAVA-OPTS-alternatives-td5798.html
But didn't mention how to supply the properties that don't start with spark.
On 12 June 2015 at 19:39, Ted Yu <yu...@gmail.com> wrote:
> This is the SPARK JIRA which introduced the warning:
>
> [SPARK-7037] [CORE] Inconsistent behavior for non-spark config properties
> in spark-shell and spark-submit
>
> On Fri, Jun 12, 2015 at 4:34 PM, Peng Cheng <rh...@gmail.com> wrote:
>
>> Hi Andrew,
>>
>> Thanks a lot! Indeed, it doesn't start with spark, the following
>> properties are read by implementation of the driver rather than spark conf:
>>
>> --conf spooky.root=s3n://spooky- \
>> --conf spooky.checkpoint=s3://spooky-checkpoint \
>>
>> This used to work from Spark 1.0.0 to 1.3.1. Do you know the new way to
>> set the same properties?
>>
>> Yours Peng
>>
>> On 12 June 2015 at 14:20, Andrew Or <an...@databricks.com> wrote:
>>
>>> Hi Peng,
>>>
>>> Setting properties through --conf should still work in Spark 1.4. From
>>> the warning it looks like the config you are trying to set does not start
>>> with the prefix "spark.". What is the config that you are trying to set?
>>>
>>> -Andrew
>>>
>>> 2015-06-12 11:17 GMT-07:00 Peng Cheng <pc...@uow.edu.au>:
>>>
>>>> In Spark <1.3.x, the system property of the driver can be set by --conf
>>>> option, shared between setting spark properties and system properties.
>>>>
>>>> In Spark 1.4.0 this feature is removed, the driver instead log the
>>>> following
>>>> warning:
>>>>
>>>> Warning: Ignoring non-spark config property: xxx.xxx=vvvvv
>>>>
>>>> How do set driver's system property in 1.4.0? Is there a reason it is
>>>> removed without a deprecation warning?
>>>>
>>>> Thanks a lot for your advices.
>>>>
>>>>
>>>>
>>>> --
>>>> View this message in context:
>>>> http://apache-spark-user-list.1001560.n3.nabble.com/Spark-1-4-0-How-to-set-driver-s-system-property-using-spark-submit-options-tp23298.html
>>>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>>>> For additional commands, e-mail: user-help@spark.apache.org
>>>>
>>>>
>>>
>>
>
Re: [Spark 1.4.0]How to set driver's system property using
spark-submit options?
Posted by Ted Yu <yu...@gmail.com>.
This is the SPARK JIRA which introduced the warning:
[SPARK-7037] [CORE] Inconsistent behavior for non-spark config properties
in spark-shell and spark-submit
On Fri, Jun 12, 2015 at 4:34 PM, Peng Cheng <rh...@gmail.com> wrote:
> Hi Andrew,
>
> Thanks a lot! Indeed, it doesn't start with spark, the following
> properties are read by implementation of the driver rather than spark conf:
>
> --conf spooky.root=s3n://spooky- \
> --conf spooky.checkpoint=s3://spooky-checkpoint \
>
> This used to work from Spark 1.0.0 to 1.3.1. Do you know the new way to
> set the same properties?
>
> Yours Peng
>
> On 12 June 2015 at 14:20, Andrew Or <an...@databricks.com> wrote:
>
>> Hi Peng,
>>
>> Setting properties through --conf should still work in Spark 1.4. From
>> the warning it looks like the config you are trying to set does not start
>> with the prefix "spark.". What is the config that you are trying to set?
>>
>> -Andrew
>>
>> 2015-06-12 11:17 GMT-07:00 Peng Cheng <pc...@uow.edu.au>:
>>
>>> In Spark <1.3.x, the system property of the driver can be set by --conf
>>> option, shared between setting spark properties and system properties.
>>>
>>> In Spark 1.4.0 this feature is removed, the driver instead log the
>>> following
>>> warning:
>>>
>>> Warning: Ignoring non-spark config property: xxx.xxx=vvvvv
>>>
>>> How do set driver's system property in 1.4.0? Is there a reason it is
>>> removed without a deprecation warning?
>>>
>>> Thanks a lot for your advices.
>>>
>>>
>>>
>>> --
>>> View this message in context:
>>> http://apache-spark-user-list.1001560.n3.nabble.com/Spark-1-4-0-How-to-set-driver-s-system-property-using-spark-submit-options-tp23298.html
>>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>>> For additional commands, e-mail: user-help@spark.apache.org
>>>
>>>
>>
>
Re: [Spark 1.4.0]How to set driver's system property using
spark-submit options?
Posted by Peng Cheng <rh...@gmail.com>.
Hi Andrew,
Thanks a lot! Indeed, it doesn't start with spark, the following properties
are read by implementation of the driver rather than spark conf:
--conf spooky.root=s3n://spooky- \
--conf spooky.checkpoint=s3://spooky-checkpoint \
This used to work from Spark 1.0.0 to 1.3.1. Do you know the new way to set
the same properties?
Yours Peng
On 12 June 2015 at 14:20, Andrew Or <an...@databricks.com> wrote:
> Hi Peng,
>
> Setting properties through --conf should still work in Spark 1.4. From the
> warning it looks like the config you are trying to set does not start with
> the prefix "spark.". What is the config that you are trying to set?
>
> -Andrew
>
> 2015-06-12 11:17 GMT-07:00 Peng Cheng <pc...@uow.edu.au>:
>
>> In Spark <1.3.x, the system property of the driver can be set by --conf
>> option, shared between setting spark properties and system properties.
>>
>> In Spark 1.4.0 this feature is removed, the driver instead log the
>> following
>> warning:
>>
>> Warning: Ignoring non-spark config property: xxx.xxx=vvvvv
>>
>> How do set driver's system property in 1.4.0? Is there a reason it is
>> removed without a deprecation warning?
>>
>> Thanks a lot for your advices.
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-spark-user-list.1001560.n3.nabble.com/Spark-1-4-0-How-to-set-driver-s-system-property-using-spark-submit-options-tp23298.html
>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>> For additional commands, e-mail: user-help@spark.apache.org
>>
>>
>
Re: [Spark 1.4.0]How to set driver's system property using
spark-submit options?
Posted by Andrew Or <an...@databricks.com>.
Hi Peng,
Setting properties through --conf should still work in Spark 1.4. From the
warning it looks like the config you are trying to set does not start with
the prefix "spark.". What is the config that you are trying to set?
-Andrew
2015-06-12 11:17 GMT-07:00 Peng Cheng <pc...@uow.edu.au>:
> In Spark <1.3.x, the system property of the driver can be set by --conf
> option, shared between setting spark properties and system properties.
>
> In Spark 1.4.0 this feature is removed, the driver instead log the
> following
> warning:
>
> Warning: Ignoring non-spark config property: xxx.xxx=vvvvv
>
> How do set driver's system property in 1.4.0? Is there a reason it is
> removed without a deprecation warning?
>
> Thanks a lot for your advices.
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Spark-1-4-0-How-to-set-driver-s-system-property-using-spark-submit-options-tp23298.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>