You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Andrew Or (JIRA)" <ji...@apache.org> on 2015/06/30 01:33:04 UTC
[jira] [Updated] (SPARK-8716) Remove executor shared cache feature
[ https://issues.apache.org/jira/browse/SPARK-8716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrew Or updated SPARK-8716:
-----------------------------
Priority: Major (was: Minor)
> Remove executor shared cache feature
> ------------------------------------
>
> Key: SPARK-8716
> URL: https://issues.apache.org/jira/browse/SPARK-8716
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 1.2.0
> Reporter: Andrew Or
> Assignee: Josh Rosen
>
> More specifically, this is the feature that is currently flagged by `spark.files.useFetchCache`. There are several reasons why we should remove it.
> (1) It doesn't even work. Recently, each executor gets its own unique temp directory for security reasons.
> (2) There is no way to fix it. The constraints in (1) are fundamentally opposed to sharing resources across executors.
> (3) It is very complex. The method Utils.fetchFile would be greatly simplified without this feature that is not even used.
> (4) There are no tests for it and it is difficult to test.
> Note that we can't just revert the respective patches because they were merged a long time ago.
> Related issues: SPARK-8130, SPARK-6313, SPARK-2713
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org