[jira] [Commented] (SPARK-8716) Remove executor shared cache feature

["Marcelo Vanzin (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-8716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14606627#comment-14606627 ] 

Marcelo Vanzin commented on SPARK-8716:
---------------------------------------

bq. (1) It doesn't even work. Recently, each executor gets its own unique temp directory for security reasons.

I thought directories were per app, not per executor? (At least that was my goal when I changed that code.)

Not arguing against getting rid of the feature, just that this particular justification doesn't sound right.

> Remove executor shared cache feature
> ------------------------------------
>
>                 Key: SPARK-8716
>                 URL: https://issues.apache.org/jira/browse/SPARK-8716
>             Project: Spark
>          Issue Type: Improvement
>          Components: Spark Core
>    Affects Versions: 1.2.0
>            Reporter: Andrew Or
>            Assignee: Josh Rosen
>            Priority: Minor
>
> More specifically, this is the feature that is currently flagged by `spark.files.useFetchCache`. There are several reasons why we should remove it.
> (1) It doesn't even work. Recently, each executor gets its own unique temp directory for security reasons.
> (2) There is no way to fix it. The constraints in (1) are fundamentally opposed to sharing resources across executors.
> (3) It is very complex. The method Utils.fetchFile would be greatly simplified without this feature that is not even used.
> (4) There are no tests for it and it is difficult to test.
> Note that we can't just revert the respective patches because they were merged a long time ago.
> Related issues: SPARK-8130, SPARK-6313, SPARK-2713



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org