Docker Containerization

[Tom Arnfeld <to...@duedil.com>]

Hey everyone,

First off; this is a long email... so brace yourself. I appreciate your
time and patience.

I wanted to open up discussion around this, since I've spoken to several
people in isolation and feel it would be great to come to some kind of
resolution. My aim here is to try and improve the new Docker Containerizer
and help to support new use cases that are currently not catered for.

I get the impression Docker + Mesos use cases fall into two groups. There's
the group of users that wish to say "I want to run a docker container on S
slave with R resources", which in almost every case I have come across is
for long running services. The other group is more along the lines of, "I
want to user docker instead of cgroup isolation directly".

There are currently two mechanisms to achieve these things, the External
Containerizer (0.19.0) and the Docker Containerizer (0.20.0), both of which
allow you to run docker containers in some way or another.

- The external containerizer implementation inside mesos knows nothing of
Docker and any Docker specific responsibilities are handed out to another
subprocess (Deimos or alike).
- The docker containerizer is a native implementation of Docker built in to
mesos, and requires zero additional dependencies or configuration (except
for Docker itself).

Each containerizer allows a framework to send additional details with a
`TaskInfo` protobuf to configure the underlying Docker container that's
going to be launched.

- The external containerizer allows the user to specify an image to use
(`ContainerInfo.image`) if they choose
- The docker containerizer requires the user to also specify an image to
use (`ContainerInfo.DockerInfo.image`)

Now here's the difference...

- The external containerizer allows the user to specify a list of "args"
which currently equate to CLI arguments to the `docker run` invocation.
- The external containerizer will be responsible for managing isolation for
all executors.
- The external containerizer is still given the opportunity to act even if
no image or arguments for the container are provided.

- The docker containerizer has a set of explicit, structured options that
define the container, e.g `ContainerInfo.volumes` and `Volume` protobufs
(though there is a limited set of supported options, these are growing over
time)
- The docker containerizer will *only* isolate tasks that include a
`ContainerInfo.type == ContainerInfo.Type.DOCKER` and pass on anything else

@tnachen/@benh (and others involved) have taken the approach to abstract
the container options to a subset of structured values, to separate
frameworks from the docker command line tool and the arguments it accepts.
Maybe in the future Mesos may use the Docker Remote API instead of a
subprocess to `docker run`, and this allows such a change to be
transparent. This is really great imo.

Back to the use cases...

> "I want to run a docker container on S slave with R resources"

This is covered really well with the docker containerizer. A framework can
choose which docker image is used, specify options in an abstract manner,
and Mesos is going to handle the rest. The user must specify an image to
run, but that's OK because the user knows they want to use a specific
image, and that they want to use Docker.

> "I want to user docker instead of cgroup isolation directly"

This use case isn't supported at all by the docker containerizer. From a
system administrators perspective, Docker can be a great tool to completely
isolate processes from one another and more importantly from the host. It
allows different dependencies to be installed and used for different
processes, and means host machines (mesos slaves) can be very pure. If the
person deploying Mesos is separated enough from those using it, the
administrator might not want to enforce that every user using any framework
*must* supply `ContainerInfo` details, especially if they're launching
tasks that require very common things (e.g python27). These common tools
may or may not be installed on the host.

My motivation here is to reduce the amount users or framework developers
need to understand about how the Mesos cluster is put together (and the
specifics of what is going on behind the scenes) and let them focus on
trying out frameworks or building their own.

Essentially what this means is, the ability to launch Mesos tasks and
executors inside docker containers transparently to the user. We've taken
this approach with the External Containerizer and our docker containerizer
implementation, and has proved to work very well for just "getting started"
on our cluster. The system administrators can sleep peacefully knowing
although anyone can run pretty much what they like on the cluster, slaves
aren't going to slowflake or become damaged. The
`--default_container_image` command line option supported by the
`mesos-slave` process was very useful here, but I can see how that explicit
approach can cause problems once you introduce the concept of multiple
containerizers.

-----------------------------------------------------------

Given this background, I have a few questions.

- Can the docker containerizer support more friendly defaults? If I only
want my mesos cluster to containerizer things with Docker, but don't wish
to require every user specify an image for their tasks.

- Since all of this Docker work has made it's way into the mesos core, a
lot of implicit decisions have been made about what options to support, how
to expose them to users, and how the workflow looks for frameworks and
users. I think this is pretty limiting, and given Mesos is designed to be
the fundamental building blocks for your datacenter, building for specific
workflows concerns me. Is finding solutions to all of these workflows
really something the mesos-core team should focus on?

- I think the external containerizer is a value asset to Mesos even without
Docker. There have been a few questions on the mailing list about using
other types of isolation (on windows for example). Could more work be done
to unity configuration for each built-in containerizer? I feel like lots of
different command line options might be a little confusing over time.

I'm very interested to hear what others think, and please do correct me if
I'm wrong in anything I've said. Looking forward to the discussion.

Tom.

Re: Docker Containerization

[Tom Arnfeld <to...@duedil.com>]

Hey,

Thank you both for taking the time to respond. I wanted to be very
expressive and give context to the points I was making, hopefully it wasn't
too ranty. Ben, I've made some inline comments below.

I've had some chats with Ian Downes about doing something
> like this for the MesosContainerizer too (but using a tarball/zip or path
> that gets read-only bind-mounted into the container instead of a Docker
> image). If we don't already have a JIRA issue for this specific feature, we
> should add one so folks can follow along.


This is very interesting, and can definitely see how that would be useful.

We'd really like to
> consolidate our own internal containerizers/isolation code with
> libcontainer. That consolidation will likely be a long process, but it's
> well worth it in the long run IMHO.


I completely agree, though probably quite painful to change I can imagine
Mesos making use of the same constructs as docker under the hood, and as
you mentioned, completely sidestep the need to interact with the docker
daemon in some cases.

As you mention below,
> this is a huge asset to Mesos and something I'd personally like to make
> sure continues to work well, as would Till Toenshoff I'm sure. If you have
> other suggestions for how to more generically support workflows we'd love
> to hear them!


Personally, from the discussions we've had here around what we plan to
implement in the future and what we've been working on already the external
containerizer API does indeed fit very well with zero adjustments, plus
leveraging the ability to swap in our own logic where needed.

Can you elaborate on what you mean when you say "unity configuration"? If
> you're referring to command line flags, I'd love to hear your suggestions.
> If you're referring to broader support for sharing components between
> containerizers, I agree wholeheartedly. In fact, while the
> containerizer/isolator part of the code base has seen quite a bit of
> volatility in the last year I think it's due in for even more going
> forward. Ultimately I'd like to see us reusing our existing isolators with
> something like the Docker containerizer (and other containerizers) so that
> folks can benefit from the isolation work that's being done irregardless of
> how the container gets initially created!


My mistake, that was meant to read "unify configuration". I think the
containerizers could go a long way to sharing more concepts, and as you
suggest, interact with each other irregardless of how the container is
originally created (e.g container usage stats and limits, perhaps).

I also feel the Docker Containerizer and the EC have diverged somewhat,
especially around the ContainerInfo protobufs. I feel the EC could benefit
from the more declarative options (i.e Volume's), though I do think it
is necessary to support an arbitrary set of configuration options
(key=value) that each specific containerizer could make use of, even if
just until the option made its way top level. It would also be powerful to
mix containerizers purely based on the ContainerInfo.TYPE field,
automatically. If each containerizer were able to accept or reject
responsibility based on a TaskInfo (kind of like how it works now, but more
explicit), you could create more heterogeneous slaves. I am aware Mesos
does some of these things, but I think there's some space to refine.

We're very much looking forward to subsequent iterations of this area of
Mesos, there's lots going on! :-)

Tom.


On 5 September 2014 19:49, Timothy Chen <tn...@gmail.com> wrote:

> Hi Tom,
>
> As Ben mentioned it's definitely doable to introduce a default image
> for docker. I was hesitant to use the existing default image flag in
> the earlier point of the docker development as we didn't have
> DockerInfo and was just reusing the ContainerInfo from the
> CommandInfo, which makes it hard to distinguish between different
> containerizers what it was meant for.
>
> Now with the new API it's definitely worth revisiting this and make
> sure we have a good solution that is maintainable and also works well
> with mulitple containerizers.
>
> I just created this, let's discuss more on the JIRA ticket:
> https://issues.apache.org/jira/browse/MESOS-1768
>
> Tim
>
> On Fri, Sep 5, 2014 at 9:50 AM, Benjamin Hindman <be...@eecs.berkeley.edu>
> wrote:
> > I appreciate the thoughtful email Tom! There's a bunch there so I've just
> > made some inline comments addressing your final questions. ;-)
> >
> > - Can the docker containerizer support more friendly defaults? If I only
> >> want my mesos cluster to containerizer things with Docker, but don't
> wish
> >> to require every user specify an image for their tasks.
> >>
> >
> > Requiring a user to specify an image for their task/executor was indeed
> one
> > of the biggest simplifying design decision we made for this first
> iteration
> > of Docker in Mesos. There was some discussion on JIRA about this too.
> That
> > all being said, I don't see any major blockers to introducing the idea
> of a
> > "default image" for a task/executor that doesn't specify any details in
> > ContainerInfo. I've had some chats with Ian Downes about doing something
> > like this for the MesosContainerizer too (but using a tarball/zip or path
> > that gets read-only bind-mounted into the container instead of a Docker
> > image). If we don't already have a JIRA issue for this specific feature,
> we
> > should add one so folks can follow along.
> >
> > - Since all of this Docker work has made it's way into the mesos core, a
> >> lot of implicit decisions have been made about what options to support,
> how
> >> to expose them to users, and how the workflow looks for frameworks and
> >> users. I think this is pretty limiting, and given Mesos is designed to
> be
> >> the fundamental building blocks for your datacenter, building for
> specific
> >> workflows concerns me. Is finding solutions to all of these workflows
> >> really something the mesos-core team should focus on?
> >>
> >
> > The explicit limiting was done because we wanted to abstract ourselves
> away
> > from the Docker CLI (as you mentioned earlier) and this is just the first
> > iteration. Tim Chen is already working on the second iteration which
> > includes things like bridge networking. Even so, we're really trying to
> > abstract ourselves as much as possible since even Docker itself is being
> > decomposed into libcontainer (and maybe more?) and we'd really like to
> > consolidate our own internal containerizers/isolation code with
> > libcontainer. That consolidation will likely be a long process, but it's
> > well worth it in the long run IMHO.
> >
> > Regarding workflows, I still think the external containerizer is the best
> > bet to capture an organization specific workflow. As you mention below,
> > this is a huge asset to Mesos and something I'd personally like to make
> > sure continues to work well, as would Till Toenshoff I'm sure. If you
> have
> > other suggestions for how to more generically support workflows we'd love
> > to hear them!
> >
> > - I think the external containerizer is a value asset to Mesos even
> without
> >> Docker. There have been a few questions on the mailing list about using
> >> other types of isolation (on windows for example). Could more work be
> done
> >> to unity configuration for each built-in containerizer? I feel like
> lots of
> >> different command line options might be a little confusing over time.
> >>
> >
> > Can you elaborate on what you mean when you say "unity configuration"? If
> > you're referring to command line flags, I'd love to hear your
> suggestions.
> > If you're referring to broader support for sharing components between
> > containerizers, I agree wholeheartedly. In fact, while the
> > containerizer/isolator part of the code base has seen quite a bit of
> > volatility in the last year I think it's due in for even more going
> > forward. Ultimately I'd like to see us reusing our existing isolators
> with
> > something like the Docker containerizer (and other containerizers) so
> that
> > folks can benefit from the isolation work that's being done irregardless
> of
> > how the container gets initially created! And as I mentioned above, it
> > might make sense to directly consume libcontainer here too, possibly even
> > contributing some of our isolators to it.
> >
> > Ben.
>

Re: Docker Containerization

[Timothy Chen <tn...@gmail.com>]

Hi Tom,

As Ben mentioned it's definitely doable to introduce a default image
for docker. I was hesitant to use the existing default image flag in
the earlier point of the docker development as we didn't have
DockerInfo and was just reusing the ContainerInfo from the
CommandInfo, which makes it hard to distinguish between different
containerizers what it was meant for.

Now with the new API it's definitely worth revisiting this and make
sure we have a good solution that is maintainable and also works well
with mulitple containerizers.

I just created this, let's discuss more on the JIRA ticket:
https://issues.apache.org/jira/browse/MESOS-1768

Tim

On Fri, Sep 5, 2014 at 9:50 AM, Benjamin Hindman <be...@eecs.berkeley.edu> wrote:
> I appreciate the thoughtful email Tom! There's a bunch there so I've just
> made some inline comments addressing your final questions. ;-)
>
> - Can the docker containerizer support more friendly defaults? If I only
>> want my mesos cluster to containerizer things with Docker, but don't wish
>> to require every user specify an image for their tasks.
>>
>
> Requiring a user to specify an image for their task/executor was indeed one
> of the biggest simplifying design decision we made for this first iteration
> of Docker in Mesos. There was some discussion on JIRA about this too. That
> all being said, I don't see any major blockers to introducing the idea of a
> "default image" for a task/executor that doesn't specify any details in
> ContainerInfo. I've had some chats with Ian Downes about doing something
> like this for the MesosContainerizer too (but using a tarball/zip or path
> that gets read-only bind-mounted into the container instead of a Docker
> image). If we don't already have a JIRA issue for this specific feature, we
> should add one so folks can follow along.
>
> - Since all of this Docker work has made it's way into the mesos core, a
>> lot of implicit decisions have been made about what options to support, how
>> to expose them to users, and how the workflow looks for frameworks and
>> users. I think this is pretty limiting, and given Mesos is designed to be
>> the fundamental building blocks for your datacenter, building for specific
>> workflows concerns me. Is finding solutions to all of these workflows
>> really something the mesos-core team should focus on?
>>
>
> The explicit limiting was done because we wanted to abstract ourselves away
> from the Docker CLI (as you mentioned earlier) and this is just the first
> iteration. Tim Chen is already working on the second iteration which
> includes things like bridge networking. Even so, we're really trying to
> abstract ourselves as much as possible since even Docker itself is being
> decomposed into libcontainer (and maybe more?) and we'd really like to
> consolidate our own internal containerizers/isolation code with
> libcontainer. That consolidation will likely be a long process, but it's
> well worth it in the long run IMHO.
>
> Regarding workflows, I still think the external containerizer is the best
> bet to capture an organization specific workflow. As you mention below,
> this is a huge asset to Mesos and something I'd personally like to make
> sure continues to work well, as would Till Toenshoff I'm sure. If you have
> other suggestions for how to more generically support workflows we'd love
> to hear them!
>
> - I think the external containerizer is a value asset to Mesos even without
>> Docker. There have been a few questions on the mailing list about using
>> other types of isolation (on windows for example). Could more work be done
>> to unity configuration for each built-in containerizer? I feel like lots of
>> different command line options might be a little confusing over time.
>>
>
> Can you elaborate on what you mean when you say "unity configuration"? If
> you're referring to command line flags, I'd love to hear your suggestions.
> If you're referring to broader support for sharing components between
> containerizers, I agree wholeheartedly. In fact, while the
> containerizer/isolator part of the code base has seen quite a bit of
> volatility in the last year I think it's due in for even more going
> forward. Ultimately I'd like to see us reusing our existing isolators with
> something like the Docker containerizer (and other containerizers) so that
> folks can benefit from the isolation work that's being done irregardless of
> how the container gets initially created! And as I mentioned above, it
> might make sense to directly consume libcontainer here too, possibly even
> contributing some of our isolators to it.
>
> Ben.

Re: Docker Containerization

[Benjamin Hindman <be...@eecs.berkeley.edu>]

I appreciate the thoughtful email Tom! There's a bunch there so I've just
made some inline comments addressing your final questions. ;-)

- Can the docker containerizer support more friendly defaults? If I only
> want my mesos cluster to containerizer things with Docker, but don't wish
> to require every user specify an image for their tasks.
>

Requiring a user to specify an image for their task/executor was indeed one
of the biggest simplifying design decision we made for this first iteration
of Docker in Mesos. There was some discussion on JIRA about this too. That
all being said, I don't see any major blockers to introducing the idea of a
"default image" for a task/executor that doesn't specify any details in
ContainerInfo. I've had some chats with Ian Downes about doing something
like this for the MesosContainerizer too (but using a tarball/zip or path
that gets read-only bind-mounted into the container instead of a Docker
image). If we don't already have a JIRA issue for this specific feature, we
should add one so folks can follow along.

- Since all of this Docker work has made it's way into the mesos core, a
> lot of implicit decisions have been made about what options to support, how
> to expose them to users, and how the workflow looks for frameworks and
> users. I think this is pretty limiting, and given Mesos is designed to be
> the fundamental building blocks for your datacenter, building for specific
> workflows concerns me. Is finding solutions to all of these workflows
> really something the mesos-core team should focus on?
>

The explicit limiting was done because we wanted to abstract ourselves away
from the Docker CLI (as you mentioned earlier) and this is just the first
iteration. Tim Chen is already working on the second iteration which
includes things like bridge networking. Even so, we're really trying to
abstract ourselves as much as possible since even Docker itself is being
decomposed into libcontainer (and maybe more?) and we'd really like to
consolidate our own internal containerizers/isolation code with
libcontainer. That consolidation will likely be a long process, but it's
well worth it in the long run IMHO.

Regarding workflows, I still think the external containerizer is the best
bet to capture an organization specific workflow. As you mention below,
this is a huge asset to Mesos and something I'd personally like to make
sure continues to work well, as would Till Toenshoff I'm sure. If you have
other suggestions for how to more generically support workflows we'd love
to hear them!

- I think the external containerizer is a value asset to Mesos even without
> Docker. There have been a few questions on the mailing list about using
> other types of isolation (on windows for example). Could more work be done
> to unity configuration for each built-in containerizer? I feel like lots of
> different command line options might be a little confusing over time.
>

Can you elaborate on what you mean when you say "unity configuration"? If
you're referring to command line flags, I'd love to hear your suggestions.
If you're referring to broader support for sharing components between
containerizers, I agree wholeheartedly. In fact, while the
containerizer/isolator part of the code base has seen quite a bit of
volatility in the last year I think it's due in for even more going
forward. Ultimately I'd like to see us reusing our existing isolators with
something like the Docker containerizer (and other containerizers) so that
folks can benefit from the isolation work that's being done irregardless of
how the container gets initially created! And as I mentioned above, it
might make sense to directly consume libcontainer here too, possibly even
contributing some of our isolators to it.

Ben.