How do you use the fetcher cache?

[Benjamin Bannier <be...@mesosphere.io>]

Hi,

In mesos-0.23.0 we added support for caching fetched artifacts (as
described by `CommandInfo::URI`).

Here if caching was enabled for an URI re-downloading of known artifacts
could be avoided if the artifacts were still inside a slave-internal
LRU-style artifact cache.

Currently the caching layer does not support updating of cached entries,
but there exist a number of proposals how this functionality could be
added. We would be interested in your feedback on

* how you currently use artifact caching, e.g., do you use caching for
  all schemes, or only enable it selectively (for which schemes?), and
* your approaches/hacks to handle artifact updates.

There currently is a patch in review to refetch cached entries if a
remote URI's size or modification time changed since the last fetch, see
https://issues.apache.org/jira/browse/MESOS-3785. Since this approach
handles at least some use cases without requiring any user involvement,
we would be interested in knowing if such an automated approach would be
beneficial in general, or if it would still require workarounds for your
use case.

Alternatively, more explicit approaches could be taken to give users
more control. It is not clear to us at the moment how that could be
implemented for arbitrary remotes (the fetcher is currently also being
refactored into an interface to enable support for more protocols).


Thanks & cheers,

Benjamin