spark.local.dir configuration

[Jung <jb...@naver.com>]

Hi all,
In the standalone mode, spark.local.dir is ignored after Spark Worker launched.
This is a scenario assuming that we have 1 master node and 1 worker node.
1. $SPARK_HOME/sbin/start-all.sh to launch Spark Master&Worker
2. Modify worker node configuration($SPARK_HOME/conf/spark-defaults.conf) spark.local.dir to another directory( /tmp_new).
3. run spark-shell from master node.

Executor in this scenario creates scratch directory like "spark-bb0876f2-7fa9-4f15-b790-24252183a4f1" under /tmp not /tmp_new.
Because worker set immutable SparkConf instance at the first time it launched and refer to this variable when create new executor which wants to change its scratch dir.
Can I change application's spark.local.dir without restarting spark workers?

Thanks,
Jung

Re: spark.local.dir configuration

[Takeshi Yamamuro <li...@gmail.com>]

Hi,

No, there is no way to change local dir paths after Worker initialized.
That is, dir paths are cached when a first executor is launched, then
following executors reference the paths.
Details can be found in codes below;
https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/deploy/worker/Worker.scala#L449
https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/util/Utils.scala#L679


On Wed, Feb 24, 2016 at 5:13 PM, Jung <jb...@naver.com> wrote:

> Hi all,
> In the standalone mode, spark.local.dir is ignored after Spark Worker
> launched.
> This is a scenario assuming that we have 1 master node and 1 worker node.
> 1. $SPARK_HOME/sbin/start-all.sh to launch Spark Master&Worker
> 2. Modify worker node configuration($SPARK_HOME/conf/spark-defaults.conf)
> spark.local.dir to another directory( /tmp_new).
> 3. run spark-shell from master node.
>
> Executor in this scenario creates scratch directory like
> "spark-bb0876f2-7fa9-4f15-b790-24252183a4f1" under /tmp not /tmp_new.
> Because worker set immutable SparkConf instance at the first time it
> launched and refer to this variable when create new executor which wants to
> change its scratch dir.
> Can I change application's spark.local.dir without restarting spark
> workers?
>
> Thanks,
> Jung




-- 
---
Takeshi Yamamuro