containers, aurora and mesos

[Jay Buffington <me...@jaybuff.com>]

Hi,

I need to provide isolation between my mesos tasks running via Aurora
because they each expect a unique view of mounts.  I've settled on a simple
approach of just using the unshare syscall to create a new mount namespace,
calling mount to create the mounts the app expects, then exec'ing the
application.

AFAICT, I have three options of where to create the mount namespace:

   1) in the mesos-slave process
   2) in the aurora executor using a custom sandbox provider
   3) in an inline shell script in Process() in a .aurora file

For #1: my reading of the MESOS-816 jira is that the hooks I need to do
something like this are a work in progress, and once that's completed,
Aurora will need changes to expose this feature.  Has any discussion taken
place on what that might look like?

#2 is something I discussed with Brian Wickman a couple of weeks ago
(assuming I understood him correctly :).  It looks like DirectorySandbox is
currently hardcoded in to the AuroraExecutor.  I suspect it would be a
simple change to dynamically import modules that implemented the
SandboxInterface based on a config option.  It's not clear to me where the
user could pass data into the TaskConfig to specify the sandbox provider or
where to stick options to configure the provider.  I recent commits related
to AURORA-158 to add Metadata to TaskConfig.  Perhaps this would be a good
use of metadata?  Or should I add another field specifically for this?

#3 is what I'm doing now, but it requires me to give CAP_SYS_ADMIN (i.e.
sudo) to the user running the process.  I was hoping to get around that
constraint by using user namespaces, but my kernel isn't new enough.  Since
mesos-slave and AuroraExecutor both run as root the first two options don't
have this limitation.

Any feedback is appreciated!

Thanks,
Jay

Re: containers, aurora and mesos

[Bill Farner <wf...@apache.org>]

AFAICT your assessment is accurate.  We haven't ventured to solve isolation
in aurora, instead delegating those responsibilities to mesos.  This means
our current answer is (1).  (2) would, as you suspect, require some
additional work to enable.  Wickman is probably the best person to continue
this discussion with, so you might consider opening a ticket to open the
discussion about how we can enable this.

-=Bill


On Fri, Apr 11, 2014 at 8:32 PM, Jay Buffington <me...@jaybuff.com> wrote:

> Hi,
>
> I need to provide isolation between my mesos tasks running via Aurora
> because they each expect a unique view of mounts.  I've settled on a simple
> approach of just using the unshare syscall to create a new mount namespace,
> calling mount to create the mounts the app expects, then exec'ing the
> application.
>
> AFAICT, I have three options of where to create the mount namespace:
>
>    1) in the mesos-slave process
>    2) in the aurora executor using a custom sandbox provider
>    3) in an inline shell script in Process() in a .aurora file
>
> For #1: my reading of the MESOS-816 jira is that the hooks I need to do
> something like this are a work in progress, and once that's completed,
> Aurora will need changes to expose this feature.  Has any discussion taken
> place on what that might look like?
>
> #2 is something I discussed with Brian Wickman a couple of weeks ago
> (assuming I understood him correctly :).  It looks like DirectorySandbox is
> currently hardcoded in to the AuroraExecutor.  I suspect it would be a
> simple change to dynamically import modules that implemented the
> SandboxInterface based on a config option.  It's not clear to me where the
> user could pass data into the TaskConfig to specify the sandbox provider or
> where to stick options to configure the provider.  I recent commits related
> to AURORA-158 to add Metadata to TaskConfig.  Perhaps this would be a good
> use of metadata?  Or should I add another field specifically for this?
>
> #3 is what I'm doing now, but it requires me to give CAP_SYS_ADMIN (i.e.
> sudo) to the user running the process.  I was hoping to get around that
> constraint by using user namespaces, but my kernel isn't new enough.  Since
> mesos-slave and AuroraExecutor both run as root the first two options don't
> have this limitation.
>
> Any feedback is appreciated!
>
> Thanks,
> Jay
>