[jira] [Comment Edited] (MESOS-336) Mesos slave should cache executors

["Bernd Mathiske (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/MESOS-336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13968838#comment-13968838 ] 

Bernd Mathiske edited comment on MESOS-336 at 4/14/14 9:05 PM:
---------------------------------------------------------------

Symlinks only work well under the assumption that the fetched content is not mutable or at least mutations are idempotent (e.g. when involving a build process). So, if we are to solve the general case, we must not use symlinks. I'll copy and extract repeatedly in a first implementation. This also takes care of the multi-user problem, with separate chmods for each copy. And the main issue, repeated downloading, remains solved.

However, after that I can add a framework-selectable switch in the URI protobuf that causes symlinks to avoid copying. I suppose it is OK then if hidden files are not covered by symlinks, since the fallback of copying can still be used for this presumably rare case.



was (Author: bernd-mesos):
Sym links only work well under the assumption that the fetched content is not mutable or at least mutations are idempotent (e.g. when involving a build process). So, if we are to solve the general case, we must not use sum links. I'll copy and extract repeatedly in a first implementation. This also takes care of the multi-user problem, with separate chmods for each copy. And the main issue, repeated downloading, remains solved.

However, after that I can add a framework-selectable switch in the URI protobuf that causes symlinks to avoid copying. I suppose it is OK then if hidden files are not covered by symlinks, since the fallback of copying can still be used for this presumably rare case.


> Mesos slave should cache executors
> ----------------------------------
>
>                 Key: MESOS-336
>                 URL: https://issues.apache.org/jira/browse/MESOS-336
>             Project: Mesos
>          Issue Type: Improvement
>          Components: slave
>            Reporter: brian wickman
>            Assignee: Bernd Mathiske
>              Labels: newbie
>
> The slave should be smarter about how it handles pulling down executors.  In our environment, executors rarely change but the slave will always pull it down from regardless HDFS.  This puts undue stress on our HDFS clusters, and is not resilient to reduced HDFS availability.



--
This message was sent by Atlassian JIRA
(v6.2#6252)