[jira] [Created] (SPARK-8716) Remove executor shared cache feature

["Andrew Or (JIRA)" <ji...@apache.org>]

Andrew Or created SPARK-8716:
--------------------------------

             Summary: Remove executor shared cache feature
                 Key: SPARK-8716
                 URL: https://issues.apache.org/jira/browse/SPARK-8716
             Project: Spark
          Issue Type: Improvement
          Components: Spark Core
    Affects Versions: 1.2.0
            Reporter: Andrew Or
            Assignee: Josh Rosen
            Priority: Minor


More specifically, this is the feature that is currently flagged by `spark.files.useFetchCache`. There are several reasons why we should remove it.

(1) It doesn't even work. Recently, each executor gets its own unique temp directory for security reasons.

(2) There is no way to fix it. The constraints in (1) are fundamentally opposed to sharing resources across executors.

(3) It is very complex. The method Utils.fetchFile would be greatly simplified without this feature that is not even used.

(4) There are no tests for it and it is difficult to test.

Note that we can't just revert the respective patches because they were merged a long time ago.

Related issues: SPARK-8130, SPARK-6313, SPARK-2713



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org