You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by Ryan Compton <co...@gmail.com> on 2013/08/23 01:24:30 UTC

Can't set log directory?

I have an nfs directory "/data" that all my nodes can read/write to. I
installed Spark in there:

bash-3.2$ echo $SPARK_HOME_DATA
/data/test/code/spark-0.7.3

Now, I don't want to communicate with '/data' all the time, since that
will stress my network. Aside from logs, I don't think it will be a
problem. I've set my spark-env.conf to look like:

#!/usr/bin/env bash

SCALA_HOME=/data/test/code/scala-2.9.3
SPARK_MASTER_IP=10.10.1.19
SPARK_WORKER_INSTANCES=1
SPARK_WORKER_DIR=/tmp/spark-worker-dir

I want to log to $SPARK_WORKER_DIR. But when I start the cluster logs
are going to '/data':

10.10.1.18: starting spark.deploy.worker.Worker, logging to
/data/test/code/spark-0.7.3/bin/../logs/spark-rfcompton-spark.deploy.worker.Worker-1-node18.out

What's wrong?