You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Josh Rosen (JIRA)" <ji...@apache.org> on 2015/07/26 22:37:04 UTC
[jira] [Resolved] (SPARK-9326) Spark never closes the lock file
used to prevent concurrent downloads
[ https://issues.apache.org/jira/browse/SPARK-9326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Josh Rosen resolved SPARK-9326.
-------------------------------
Resolution: Fixed
Fix Version/s: 1.5.0
1.4.2
> Spark never closes the lock file used to prevent concurrent downloads
> ---------------------------------------------------------------------
>
> Key: SPARK-9326
> URL: https://issues.apache.org/jira/browse/SPARK-9326
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 1.4.0
> Reporter: Kay Ousterhout
> Assignee: Kay Ousterhout
> Priority: Minor
> Fix For: 1.4.2, 1.5.0
>
>
> A lock file is used to ensure multiple executors running on the same machine don't download the same file concurrently. Spark never closes these lock files (we release the lock, but releasing the lock does not close the underlying file). In theory, if an executor fetched a large number of files, this could eventually lead to an issue with too many open files.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org