Fetching and Caching Binaries from HDFS

[Zameer Manji <zm...@gmail.com>]

Hey,

I noticed in MESOS-336 that there was some discussion on how to cache the
Mesos
executor so it does not need to be repeatedly fetched from HDFS. This
parallels
a problem faced by users of Aurora which is how to fetch binaries needed for
tasks. Twitter mitigated this problem by caching fetched binaries from HDFS
on
the slave file system and having the first process of each task fetch
binaries
from the cache if possible. If it is not possible to fetch it from the
cache,
the process places the binary in the cache for subsequent task starts on the
same slave.

The code that does this and a brief explanation on how it works can be
found in
this gist: https://gist.github.com/zmanji/f41df77510ef9d00265a. I hope it
serves
as a good example on how this problem can be mitigated.

-- 
Zameer Manji