You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by Baoqiang Cao <bq...@gmail.com> on 2014/08/09 16:21:04 UTC

set SPARK_LOCAL_DIRS issue

Hi

I’m trying to using a specific dir for spark working directory since I have limited space at /tmp. I tried: 
1)
export SPARK_LOCAL_DIRS=“/mnt/data/tmp”
or 2)
SPARK_LOCAL_DIRS=“/mnt/data/tmp” in spark-env.sh

But neither worked, since the output of spark still saying 

ERROR DiskBlockObjectWriter: Uncaught exception while reverting partial writes to file /tmp/spark-local-20140809134509-0502/34/shuffle_0_436_1
java.io.FileNotFoundException: /tmp/spark-local-20140809134509-0502/34/shuffle_0_436_1 (No space left on device)

anybody help with correctly setting up the “tmp” directory?

Best,
Baoqiang Cao
Blog: http://baoqiang.org
Email: bqcaomail@gmail.com

Re: set SPARK_LOCAL_DIRS issue

Posted by Andrew Ash <an...@andrewash.com>.

// assuming Spark 1.0

Hi Baoqiang,

In my experience for the standalone cluster you need to set
SPARK_WORKER_DIR not SPARK_LOCAL_DIRS to control where shuffle files are
written.  I think this is a documentation issue that could be improved, as
http://spark.apache.org/docs/latest/spark-standalone.html suggests using
SPARK_LOCAL_DIRS for scratch, and I'm not sure that it actually does
anything.

Did you see anything in /mnt/data/tmp when you used SPARK_LOCAL_DIRS?

Cheers!
Andrew

On Sat, Aug 9, 2014 at 7:21 AM, Baoqiang Cao <bq...@gmail.com> wrote:

> Hi
>
> I’m trying to using a specific dir for spark working directory since I
> have limited space at /tmp. I tried:
> 1)
> export SPARK_LOCAL_DIRS=“/mnt/data/tmp”
> or 2)
> SPARK_LOCAL_DIRS=“/mnt/data/tmp” in spark-env.sh
>
> But neither worked, since the output of spark still saying
>
> ERROR DiskBlockObjectWriter: Uncaught exception while reverting partial
> writes to file /tmp/spark-local-20140809134509-0502/34/shuffle_0_436_1
> java.io.FileNotFoundException:
> /tmp/spark-local-20140809134509-0502/34/shuffle_0_436_1 (No space left on
> device)
>
> anybody help with correctly setting up the “tmp” directory?
>
> Best,
> Baoqiang Cao
> Blog: http://baoqiang.org
> Email: bqcaomail@gmail.com
>
>
>
>
>