You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by Gauthier Feuillen <ga...@dataroots.io> on 2018/03/26 20:08:14 UTC

Local dirs

Hi

I am trying to change the spark.local.dir property. I am running spark on yarn and have already tried the following properties:

export LOCAL_DIRS=<my-tmp>

spark.yarn.appMasterEnv.LOCAL_DIRS=<my-tmp>
spark.yarn.appMasterEnv.SPARK_LOCAL_DIRS=<my-tmp>
spark.yarn.nodemanager.local-dirs=/<my-tmp>
spark.local.dir=<my-tmp>

But still it does not change the executors local directory. Is there a way to change this value without changing yarn-site.xml ? 

Thanks in advance !
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org

Re: Local dirs

Posted by Gauthier Feuillen <ga...@dataroots.io>.

Thanks

> On 26 Mar 2018, at 22:09, Marcelo Vanzin <va...@cloudera.com> wrote:
> 
> On Mon, Mar 26, 2018 at 1:08 PM, Gauthier Feuillen
> <ga...@dataroots.io> wrote:
>> Is there a way to change this value without changing yarn-site.xml ?
> 
> No. Local dirs are defined by the NodeManager, and Spark cannot override them.
> 
> -- 
> Marcelo


---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org

Re: Local dirs

Posted by Marcelo Vanzin <va...@cloudera.com>.

On Mon, Mar 26, 2018 at 1:08 PM, Gauthier Feuillen
<ga...@dataroots.io> wrote:
> Is there a way to change this value without changing yarn-site.xml ?

No. Local dirs are defined by the NodeManager, and Spark cannot override them.

-- 
Marcelo

---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org