[jira] [Updated] (MESOS-816) Allow delegation to shell scripts for isolation

["Niklas Quarfot Nielsen (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/MESOS-816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Niklas Quarfot Nielsen updated MESOS-816:
-----------------------------------------

    Description: 
Being able to delegate isolation to shell scripts could make it easier to leverage the machinery provided by the LXC tools, LibVirt, VirtualBox, Docker and similar containerization systems.

Why go through command line tools for isolation? We have seen many requests for isolation, covering a wide variety of scenarios:

- Setups requiring multiple versions of the same language (Ruby 1.8, Ruby 1.9).
- Setups requiring installation and configuration of RPM-packaged applications.
- Build-and-test setups, where sharing the environment of the host would impact reproducibility.
- Integration of 3rd party, service-oriented applications.
- Launching applications with Docker.
- Launching multiple instances of a Mesos framework that, like Hadoop, has significant system setup and dependencies.

To cover these and other use cases, it seems reasonable to allow Mesos to delegate to external programs for isolation:

- It makes it easier to experiment with new containerization tools.
- It allows for site administrators to customize containerization, or even implement new containerization mechanisms, without impacting their ability to keep pace with Mesos development.
- Many external programs exist for containerization -- Docker, LXC tools, LibVirt -- which handle a great deal of the book-keeping around finding and efficiently cloning disk images and setting up the guest system (its hostname, TTYs, /dev/*, /proc).

The scenarios listed above can be understood in terms of three use cases:

- The containerized system service scenario, wherein an application, installed with RPM or a similar tool, is started and managed by the init system within a container. Percona MySQL is an example of such an application.
- The containerized application scenario, wherein an application is installed or unpacked and then configured and launched in a single command. For example, running a custom Rails app with bundle install && bundle exec rails.
- The containerized framework/executor scenario, wherein the application is Spark, Hadoop or another Mesos framework/executor pair.

One way to achieve this could be to introduce an External Isolator, which works in parallel with the existing process/posix and cgroups isolators. The responsibility of this isolator would be to act as a thin layer to external isolators. Calls for task launching, stopping or any other resource change would be serialized and passed to the external isolators by the Mesos External Isolator. 

We think an approach like this adds a lot of flexibility while still keeping a good clean architecture and avoids using executors for isolation.
However, we are currently exploring how to solve this problem so feel free to opt in with ideas, comments and suggestions.


  was:
Delegating to shell scripts for isolation would make it easily to leverage the machinery provided by the LXC tools, LibVirt, VirtualBox and similar systems.

Why go through command line tools for isolation? Such tools manage a great deal of the book-keeping around finding and efficiently cloning disk images and setting up the guest system (its hostname, TTYs, /dev/*, /proc) in a way that is broadly useful.

A shell isolator would be called by the slave, with arguments allowing it to ultimately start the executor under isolation.

One possible design is as follows:

* The slave calls the isolator with two arguments: a path to a Mesos boot strap tool (a reimagining of mesos-launcher?) and a URL indicating how to connect to the slave.

    mesos-libvirt /usr/sbin/mesos-bootstrap localhost:5051

* The isolator sets up a container and runs mesos-bootstrap therein, copying it (and attendant libraries?) as needed.

* The mesos-bootstrap tool connects to the slave and retrieves the ExecutorInfo structure. It uses this information to launch the executor.

* The executor connects to the slave and receives TaskInfos.

Some questions arise with this design:

* How do we ensure libraries needed by mesos-bootstrap are present?

* What can we do to make it easier for the user to include libraries with the executor? Should directories to copy or union mount in to the container be part of the argument specification for the isolator?


> Allow delegation to shell scripts for isolation
> -----------------------------------------------
>
>                 Key: MESOS-816
>                 URL: https://issues.apache.org/jira/browse/MESOS-816
>             Project: Mesos
>          Issue Type: Improvement
>          Components: isolation, slave
>            Reporter: Jason Dusek
>            Priority: Minor
>         Attachments: mesos-shell-isolator.jpg
>
>   Original Estimate: 72h
>  Remaining Estimate: 72h
>
> Being able to delegate isolation to shell scripts could make it easier to leverage the machinery provided by the LXC tools, LibVirt, VirtualBox, Docker and similar containerization systems.
> Why go through command line tools for isolation? We have seen many requests for isolation, covering a wide variety of scenarios:
> - Setups requiring multiple versions of the same language (Ruby 1.8, Ruby 1.9).
> - Setups requiring installation and configuration of RPM-packaged applications.
> - Build-and-test setups, where sharing the environment of the host would impact reproducibility.
> - Integration of 3rd party, service-oriented applications.
> - Launching applications with Docker.
> - Launching multiple instances of a Mesos framework that, like Hadoop, has significant system setup and dependencies.
> To cover these and other use cases, it seems reasonable to allow Mesos to delegate to external programs for isolation:
> - It makes it easier to experiment with new containerization tools.
> - It allows for site administrators to customize containerization, or even implement new containerization mechanisms, without impacting their ability to keep pace with Mesos development.
> - Many external programs exist for containerization -- Docker, LXC tools, LibVirt -- which handle a great deal of the book-keeping around finding and efficiently cloning disk images and setting up the guest system (its hostname, TTYs, /dev/*, /proc).
> The scenarios listed above can be understood in terms of three use cases:
> - The containerized system service scenario, wherein an application, installed with RPM or a similar tool, is started and managed by the init system within a container. Percona MySQL is an example of such an application.
> - The containerized application scenario, wherein an application is installed or unpacked and then configured and launched in a single command. For example, running a custom Rails app with bundle install && bundle exec rails.
> - The containerized framework/executor scenario, wherein the application is Spark, Hadoop or another Mesos framework/executor pair.
> One way to achieve this could be to introduce an External Isolator, which works in parallel with the existing process/posix and cgroups isolators. The responsibility of this isolator would be to act as a thin layer to external isolators. Calls for task launching, stopping or any other resource change would be serialized and passed to the external isolators by the Mesos External Isolator. 
> We think an approach like this adds a lot of flexibility while still keeping a good clean architecture and avoids using executors for isolation.
> However, we are currently exploring how to solve this problem so feel free to opt in with ideas, comments and suggestions.



--
This message was sent by Atlassian JIRA
(v6.1#6144)