[jira] [Resolved] (SPARK-2974) Utils.getLocalDir() may return non-existent spark.local.dir directory

["Patrick Wendell (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-2974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Patrick Wendell resolved SPARK-2974.
------------------------------------

       Resolution: Fixed
    Fix Version/s: 1.1.0

Issue resolved by pull request 2002
[https://github.com/apache/spark/pull/2002]

> Utils.getLocalDir() may return non-existent spark.local.dir directory
> ---------------------------------------------------------------------
>
>                 Key: SPARK-2974
>                 URL: https://issues.apache.org/jira/browse/SPARK-2974
>             Project: Spark
>          Issue Type: Bug
>          Components: Spark Core
>    Affects Versions: 1.1.0
>            Reporter: Josh Rosen
>            Assignee: Josh Rosen
>            Priority: Blocker
>             Fix For: 1.1.0
>
>
> The patch for [SPARK-2324] modified Spark to ignore a certain number of invalid local directories.  Unfortunately, the {{Utils.getLocalDir()}} method returns the _first_ local directory from {{spark.local.dir}}, which might not exist.  This can lead to confusing FileNotFound errors when executors attempt to fetch files. 
> (I commented on this at https://github.com/apache/spark/pull/1274#issuecomment-51537965, but I'm opening a JIRA so we don't forget to fix it).



--
This message was sent by Atlassian JIRA
(v6.2#6252)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org