Re: Review Request 21316: Preview of MESOS-336

[Bernd Mathiske <be...@mesosphere.io>]

> On May 29, 2014, 3:05 p.m., Tom Arnfeld wrote:
> > include/mesos/mesos.proto, line 198
> > <https://reviews.apache.org/r/21316/diff/6/?file=594885#file594885line198>
> >
> >     What's the reason you're only caching on a per-framework basis? Surely if the URI is identical the content would be... If I have multiple Hadoop or Spark clients running but using the same executor, it's a shame they wouldn't share a cache entry.
> 
> Bernd Mathiske wrote:
>     Security concerns. I don't know to what extent what users trust the code behind a URI to remain the same. I can change it to per slave if there is consensus. Or we could have a flag that decides between the two options.

After some thought, I have come to the conclusion that I should extend the enum type that indicates how to cache NOW, not later. Here is what I have in mind in mesos.proto:

enum FetcherCache {
  CACHE_NONE = 0;
  CACHE_EXTERNAL = 1;
  CACHE_PER_SLAVE = 2;
  CACHE_PER_FRAMEWORK = 3;
}

I'll implement this before the next patch update.


- Bernd


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/21316/#review44301
-----------------------------------------------------------


On May 31, 2014, 8:04 a.m., Bernd Mathiske wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/21316/
> -----------------------------------------------------------
> 
> (Updated May 31, 2014, 8:04 a.m.)
> 
> 
> Review request for mesos and Benjamin Hindman.
> 
> 
> Bugs: MESOS-336
>     https://issues.apache.org/jira/browse/MESOS-336
> 
> 
> Repository: mesos-git
> 
> 
> Description
> -------
> 
> The first cut at fetcher caching. See MESOS-336 JIRA for explanation for this approach: keep the cache info in the MesosContainerizerProcess in the save, leverage actor single-threadedness to deal with concurrency issues without head ache. 
> 
> Features so far:
> - If URI flag "fetched_externally" (default: false) is set,  the fetcher does what it did in Mesos 0.18 and before.
> - If URI flag "cached" (default: false) is not set, the fetcher also fetches every time as in Mesos 0.18 and before.
> - If URI flag "cached" is set, the UIR is only fetched once and all subsequent fetch attempts copy from the cache file.
> - URIs are cached separately per framework (ID).
> - Recovery is implemented by simply wiping the entire cache.
> - GC for cache files. Global flag sets lifetime after last use. Default is 1 hour.
> 
> Potential future features:
> - symlinks instead of copying
> - extraction directly from URI, without cache file
> - combine that with symlinks
> - Refreshing, explicit cache invalidation
> - ...
> 
> 
> Diffs
> -----
> 
>   include/mesos/mesos.proto ce780ca 
>   src/Makefile.am ae576c5 
>   src/launcher/fetcher.cpp c4425eb 
>   src/local/local.cpp 5d26aff 
>   src/slave/constants.hpp ace4590 
>   src/slave/constants.cpp 51f65bb 
>   src/slave/containerizer/mesos_containerizer.hpp 1f5908a 
>   src/slave/containerizer/mesos_containerizer.cpp d01d443 
>   src/slave/containerizer/mesos_fetcher.hpp PRE-CREATION 
>   src/slave/containerizer/mesos_fetcher.cpp PRE-CREATION 
>   src/slave/flags.hpp 15e5b64 
>   src/slave/slave.hpp 769bd00 
>   src/slave/slave.cpp a4b9570 
>   src/tests/containerizer_tests.cpp 2f4888d 
>   src/tests/fetcher_tests.cpp PRE-CREATION 
>   src/tests/slave_tests.cpp 85ca5c4 
> 
> Diff: https://reviews.apache.org/r/21316/diff/
> 
> 
> Testing
> -------
> 
> Tests have been written, have been run successfully, and are included in the patch. 
> 
> 
> Thanks,
> 
> Bernd Mathiske
> 
>