[jira] [Created] (SPARK-5838) Changing SPARK_LOCAL_DIRS option in spark-env.sh does not take effect without daemon restart

["Theodore Vasiloudis (JIRA)" <ji...@apache.org>]

Theodore Vasiloudis created SPARK-5838:
------------------------------------------

             Summary: Changing SPARK_LOCAL_DIRS option in spark-env.sh does not take effect without daemon restart
                 Key: SPARK-5838
                 URL: https://issues.apache.org/jira/browse/SPARK-5838
             Project: Spark
          Issue Type: Bug
          Components: Deploy, EC2, Spark Submit
    Affects Versions: 1.1.1
            Reporter: Theodore Vasiloudis
            Priority: Minor


This issue has already been mentioned in the mailing list here: http://apache-spark-user-list.1001560.n3.nabble.com/set-spark-local-dir-on-driver-program-doesn-t-take-effect-td11040.html

The problem usually has to do with Spark creating too many files during shuffles, filling up the small amount of disk space that most EC2 instances have for root on /mnt2.

The workaround is to set SPARK_LOCAL_DIRS to a larger volume (e.g. to the /mnt/spark volume only, removing /mnt2).

However for these changes to take effect, the daemons need to be restarted with sbin/stop-all -> sbin/start-all. 
Even more troubling is the fact that the Web UI-> Environment reports that the spark.local.dir is set to the new path, but Spark still spills to /mnt2 as well.

To my knowledge this is not mentioned anywhere in the documentation or any other mailing list reply except for the one I linked.

I guess possible solutions are to either ensure the change does take effect so that reality agrees with what the Web UI is reporting, or include a section on the documentation of EC2 for this kind of problem.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org