You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Renu Yadav <yr...@gmail.com> on 2021/05/07 10:33:50 UTC
Re: Updating spark-env.sh per application
Hi Team,
Is it possible to override the variable of spark-env.sh on application
level ?
Thanks & Regards,
Renu Yadav
On Fri, May 7, 2021 at 12:16 PM Renu Yadav <yr...@gmail.com> wrote:
> Hi Team,
>
> Is it possible to override the variable of spark-env.sh on application
> level ?
>
> Thanks & Regards,
> Renu Yadav
>
>
Re: Updating spark-env.sh per application
Posted by Mich Talebzadeh <mi...@gmail.com>.
Hi Renu,
SPARK_DIST_CLASSPATH is specific to hadoop API and it is not part of
conf/spark-env.sh, although you can add to this file as per doc here
<https://spark.apache.org/docs/latest/hadoop-provided.html>
However, two things will throw light into this
1. Version of Spark you are using?
2. What is the drive to modify this classpath within the app itself, can
you add the jar file to spark-submit with classpath etc? It is not common
to modify the env variables after spark job kicks in
HTH
view my Linkedin profile
<https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.
On Sun, 9 May 2021 at 04:32, Renu Yadav <yr...@gmail.com> wrote:
> Hi Mich,
>
> In spark-env.sh , SPARK_DIST_CLASSPATH is set . I want to override this
> variable during runtime as wanted to exclude one lib class from it.
>
>
>
> On Fri, 7 May, 2021, 6:51 pm Mich Talebzadeh, <mi...@gmail.com>
> wrote:
>
>> Hi,
>>
>> Environment variables Re read in when spark-submit kicks off. What
>> exactly you need to refresh at the application level?
>>
>> HTH
>>
>> On Fri, 7 May 2021 at 11:34, Renu Yadav <yr...@gmail.com> wrote:
>>
>>> Hi Team,
>>>
>>> Is it possible to override the variable of spark-env.sh on application
>>> level ?
>>>
>>> Thanks & Regards,
>>> Renu Yadav
>>>
>>>
>>> On Fri, May 7, 2021 at 12:16 PM Renu Yadav <yr...@gmail.com> wrote:
>>>
>>>> Hi Team,
>>>>
>>>> Is it possible to override the variable of spark-env.sh on application
>>>> level ?
>>>>
>>>> Thanks & Regards,
>>>> Renu Yadav
>>>>
>>>> --
>>
>>
>>
>> view my Linkedin profile
>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>
>>
>>
>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>> any loss, damage or destruction of data or any other property which may
>> arise from relying on this email's technical content is explicitly
>> disclaimed. The author will in no case be liable for any monetary damages
>> arising from such loss, damage or destruction.
>>
>>
>>
>
Re: Updating spark-env.sh per application
Posted by Jacek Laskowski <ja...@japila.pl>.
Hi,
The easiest (but perhaps not necessarily the most flexible) is simply to
use two different versions of spark-submit script with the env var set to
two different values. Have you tried it yet?
Pozdrawiam,
Jacek Laskowski
----
https://about.me/JacekLaskowski
"The Internals Of" Online Books <https://books.japila.pl/>
Follow me on https://twitter.com/jaceklaskowski
<https://twitter.com/jaceklaskowski>
On Sun, May 9, 2021 at 5:32 AM Renu Yadav <yr...@gmail.com> wrote:
> Hi Mich,
>
> In spark-env.sh , SPARK_DIST_CLASSPATH is set . I want to override this
> variable during runtime as wanted to exclude one lib class from it.
>
>
>
> On Fri, 7 May, 2021, 6:51 pm Mich Talebzadeh, <mi...@gmail.com>
> wrote:
>
>> Hi,
>>
>> Environment variables Re read in when spark-submit kicks off. What
>> exactly you need to refresh at the application level?
>>
>> HTH
>>
>> On Fri, 7 May 2021 at 11:34, Renu Yadav <yr...@gmail.com> wrote:
>>
>>> Hi Team,
>>>
>>> Is it possible to override the variable of spark-env.sh on application
>>> level ?
>>>
>>> Thanks & Regards,
>>> Renu Yadav
>>>
>>>
>>> On Fri, May 7, 2021 at 12:16 PM Renu Yadav <yr...@gmail.com> wrote:
>>>
>>>> Hi Team,
>>>>
>>>> Is it possible to override the variable of spark-env.sh on application
>>>> level ?
>>>>
>>>> Thanks & Regards,
>>>> Renu Yadav
>>>>
>>>> --
>>
>>
>>
>> view my Linkedin profile
>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>
>>
>>
>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>> any loss, damage or destruction of data or any other property which may
>> arise from relying on this email's technical content is explicitly
>> disclaimed. The author will in no case be liable for any monetary damages
>> arising from such loss, damage or destruction.
>>
>>
>>
>
Re: Updating spark-env.sh per application
Posted by Renu Yadav <yr...@gmail.com>.
Hi Mich,
In spark-env.sh , SPARK_DIST_CLASSPATH is set . I want to override this
variable during runtime as wanted to exclude one lib class from it.
On Fri, 7 May, 2021, 6:51 pm Mich Talebzadeh, <mi...@gmail.com>
wrote:
> Hi,
>
> Environment variables Re read in when spark-submit kicks off. What exactly
> you need to refresh at the application level?
>
> HTH
>
> On Fri, 7 May 2021 at 11:34, Renu Yadav <yr...@gmail.com> wrote:
>
>> Hi Team,
>>
>> Is it possible to override the variable of spark-env.sh on application
>> level ?
>>
>> Thanks & Regards,
>> Renu Yadav
>>
>>
>> On Fri, May 7, 2021 at 12:16 PM Renu Yadav <yr...@gmail.com> wrote:
>>
>>> Hi Team,
>>>
>>> Is it possible to override the variable of spark-env.sh on application
>>> level ?
>>>
>>> Thanks & Regards,
>>> Renu Yadav
>>>
>>> --
>
>
>
> view my Linkedin profile
> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>
>
>
> *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed.
> The author will in no case be liable for any monetary damages arising from
> such loss, damage or destruction.
>
>
>
Re: Updating spark-env.sh per application
Posted by Mich Talebzadeh <mi...@gmail.com>.
Hi,
Environment variables Re read in when spark-submit kicks off. What exactly
you need to refresh at the application level?
HTH
On Fri, 7 May 2021 at 11:34, Renu Yadav <yr...@gmail.com> wrote:
> Hi Team,
>
> Is it possible to override the variable of spark-env.sh on application
> level ?
>
> Thanks & Regards,
> Renu Yadav
>
>
> On Fri, May 7, 2021 at 12:16 PM Renu Yadav <yr...@gmail.com> wrote:
>
>> Hi Team,
>>
>> Is it possible to override the variable of spark-env.sh on application
>> level ?
>>
>> Thanks & Regards,
>> Renu Yadav
>>
>> --
view my Linkedin profile
<https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.