You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Kay Ousterhout (JIRA)" <ji...@apache.org> on 2015/07/24 22:42:04 UTC

[jira] [Created] (SPARK-9326) Spark never closes the lock file used to prevent concurrent downloads

Kay Ousterhout created SPARK-9326:
-------------------------------------

             Summary: Spark never closes the lock file used to prevent concurrent downloads
                 Key: SPARK-9326
                 URL: https://issues.apache.org/jira/browse/SPARK-9326
             Project: Spark
          Issue Type: Bug
    Affects Versions: 1.4.0
            Reporter: Kay Ousterhout
            Assignee: Kay Ousterhout
            Priority: Minor


A lock file is used to ensure multiple executors running on the same machine don't download the same file concurrently. Spark never closes these lock files (we release the lock, but releasing the lock does not close the  underlying file). In theory, if an executor fetched a large number of files, this could eventually lead to an issue with too many open files.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org