You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Josh Rosen (JIRA)" <ji...@apache.org> on 2015/03/17 17:34:40 UTC

[jira] [Resolved] (SPARK-6313) Fetch File Lock file creation doesnt work when Spark working dir is on a NFS mount

     [ https://issues.apache.org/jira/browse/SPARK-6313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Josh Rosen resolved SPARK-6313.
-------------------------------
       Resolution: Fixed
    Fix Version/s: 1.2.2
                   1.3.1
                   1.4.0

Issue resolved by pull request 5036
[https://github.com/apache/spark/pull/5036]

> Fetch File Lock file creation doesnt work when Spark working dir is on a NFS mount
> ----------------------------------------------------------------------------------
>
>                 Key: SPARK-6313
>                 URL: https://issues.apache.org/jira/browse/SPARK-6313
>             Project: Spark
>          Issue Type: Bug
>          Components: Spark Core
>    Affects Versions: 1.2.0, 1.3.0, 1.2.1
>            Reporter: Nathan McCarthy
>            Priority: Critical
>             Fix For: 1.4.0, 1.3.1, 1.2.2
>
>
> When running in cluster mode and mounting the spark work dir on a NFS volume (or some volume which doesn't support file locking), the fetchFile (used for downloading JARs etc on the executors) method in Spark Utils class will fail. This file locking was introduced as an improvement with SPARK-2713. 
> See https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/util/Utils.scala#L415 
> Introduced in 1.2 in commit; https://github.com/apache/spark/commit/7aacb7bfad4ec73fd8f18555c72ef696 
> As this locking is for optimisation for fetching files, could we take a different approach here to create a temp/advisory lock file? 
> Typically you would just mount local disks (in say ext4 format) and provide this as a comma separated list however we are trying to run Spark on MapR. With MapR we can do a loop back mount to a volume on the local node and take advantage of MapRs disk pools. This also means we dont need specific mounts for Spark and improves the generic nature of the cluster. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org