You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mesos.apache.org by Tobias Knaup <to...@knaup.me> on 2014/06/21 00:40:32 UTC
Docker support in Mesos core
Hi all,
We've got a lot of feedback from folks who use Mesos to run Dockers at
scale via Deimos, and the main wish was to make Docker a first class
citizen in Mesos, instead of a plugin that needs to be installed
separately. Mesosphere wants to contribute this and I already chatted with
Ben H about what an implementation could look like.
I'd love for folks on here that are working with Docker to chime in!
I created a JIRA here: https://issues.apache.org/jira/browse/MESOS-1524
Cheers,
Tobi
Re: Docker support in Mesos core
Posted by Tom Arnfeld <to...@duedil.com>.
Hey Everyone,
Excited to see discussions of this. Something I started playing around with just as the external containerizer was coming to life!
Diptanu – A few responses to your notes...
> a. Mesos understanding docker metrics which should be straightforward because docker writes all its metrics in the following fashin for cpu, blkio, memory etc - /sys/fs/cgroup/cpu/docker/<conntainerid>
Are these paths not operating system dependent? I'm not too familiar with cross-platform cgroups, so I am no doubt wrong here. These cgroup metrics are also the ones Mesos currently uses (both for usage statistics and for memory/cpu limits) so they can be pulled out much the same.
> b. Easier way to map tasks to docker containers, probably the external containerizer takes care of it to a large extent. It would be helpful if there was a blog about it's API and internals by the core committers explaining the design. Even a simple example in the mesos codebase using the external containerizer would help.
That's an interesting one, a blog post would be awesome. The containerizers right now currently use the "ContainerId" string provided to them from Mesos (I believe this is just the TaskID but i'm not certain of that). This helps ensure consistency of how containerizers are implemented, and makes them much simpler.
> c. stdout and stderr of docker containers in the Mesos task stdout and stderr logs. Does the external containerizer already takes care of it? I had to write a service which runs on every slave for exposing the container logs to an user.
The external containerizer itself doesn't help you with this. The logs from the containerizer calls are dumped into the sandbox, however it's up to the containerizer (e.g Deimos) to redirect the logs from the container it launches. Deimos does take care of this already (as seen here https://github.com/mesosphere/deimos/blob/master/deimos/containerizer/docker.py#L132).
> e. Translate all task constraints to docker run flags. This is probably the easiest and I know it's super easy to implement with the external containerizer.
The current Docker containerizer implementations both do this, they support CPU, Memory and Ports. Docker currently doesn't support changing these limits on a running container, so you have to go behind docker and write to the cgroup limits yourself. There's also no way to change the port mappings of a container that I know of.
Hope that answers some of your questions!
Tom.
On 21 Jun 2014, at 00:20, Diptanu Choudhury <di...@gmail.com> wrote:
> Great timing for this thread!
>
> I have been working on this for the past few months and here is what I am doing and would be nice if docker was supported straight way in Mesos. So here goes the features that I would personally love to see in Mesos Core from the perspective of an user which I had to implement on my own -
>
> a. Mesos understanding docker metrics which should be straightforward because docker writes all its metrics in the following fashin for cpu, blkio, memory etc - /sys/fs/cgroup/cpu/docker/<conntainerid>
> I am sending all these metrics right now as a framework message back to my framework/scheduler but it would be cool if Mesos took care of them.
>
> b. Easier way to map tasks to docker containers, probably the external containerizer takes care of it to a large extent. It would be helpful if there was a blog about it's API and internals by the core committers explaining the design. Even a simple example in the mesos codebase using the external containerizer would help.
>
> c. stdout and stderr of docker containers in the Mesos task stdout and stderr logs. Does the external containerizer already takes care of it? I had to write a service which runs on every slave for exposing the container logs to an user.
>
> d. Mesos GC of tasks taking care of cleaning up docker containers which have terminated. Right now the way I implemented this is that the service which exposes the logs of a container also listens to docker events and when a container exits, it knows that this has to be cleaned up and so removes it after a fixed amount of time[configurable through a Rest API/config file]
>
> e. Translate all task constraints to docker run flags. This is probably the easiest and I know it's super easy to implement with the external containerizer.
>
>
> On Fri, Jun 20, 2014 at 3:40 PM, Tobias Knaup <to...@knaup.me> wrote:
> Hi all,
>
> We've got a lot of feedback from folks who use Mesos to run Dockers at scale via Deimos, and the main wish was to make Docker a first class citizen in Mesos, instead of a plugin that needs to be installed separately. Mesosphere wants to contribute this and I already chatted with Ben H about what an implementation could look like.
>
> I'd love for folks on here that are working with Docker to chime in!
> I created a JIRA here: https://issues.apache.org/jira/browse/MESOS-1524
>
> Cheers,
>
> Tobi
>
>
>
> --
> Thanks,
> Diptanu Choudhury
> Web - www.linkedin.com/in/diptanu
> Twitter - @diptanu
Re: Docker support in Mesos core
Posted by Tom Arnfeld <to...@duedil.com>.
Hey Everyone,
Excited to see discussions of this. Something I started playing around with just as the external containerizer was coming to life!
Diptanu – A few responses to your notes...
> a. Mesos understanding docker metrics which should be straightforward because docker writes all its metrics in the following fashin for cpu, blkio, memory etc - /sys/fs/cgroup/cpu/docker/<conntainerid>
Are these paths not operating system dependent? I'm not too familiar with cross-platform cgroups, so I am no doubt wrong here. These cgroup metrics are also the ones Mesos currently uses (both for usage statistics and for memory/cpu limits) so they can be pulled out much the same.
> b. Easier way to map tasks to docker containers, probably the external containerizer takes care of it to a large extent. It would be helpful if there was a blog about it's API and internals by the core committers explaining the design. Even a simple example in the mesos codebase using the external containerizer would help.
That's an interesting one, a blog post would be awesome. The containerizers right now currently use the "ContainerId" string provided to them from Mesos (I believe this is just the TaskID but i'm not certain of that). This helps ensure consistency of how containerizers are implemented, and makes them much simpler.
> c. stdout and stderr of docker containers in the Mesos task stdout and stderr logs. Does the external containerizer already takes care of it? I had to write a service which runs on every slave for exposing the container logs to an user.
The external containerizer itself doesn't help you with this. The logs from the containerizer calls are dumped into the sandbox, however it's up to the containerizer (e.g Deimos) to redirect the logs from the container it launches. Deimos does take care of this already (as seen here https://github.com/mesosphere/deimos/blob/master/deimos/containerizer/docker.py#L132).
> e. Translate all task constraints to docker run flags. This is probably the easiest and I know it's super easy to implement with the external containerizer.
The current Docker containerizer implementations both do this, they support CPU, Memory and Ports. Docker currently doesn't support changing these limits on a running container, so you have to go behind docker and write to the cgroup limits yourself. There's also no way to change the port mappings of a container that I know of.
Hope that answers some of your questions!
Tom.
On 21 Jun 2014, at 00:20, Diptanu Choudhury <di...@gmail.com> wrote:
> Great timing for this thread!
>
> I have been working on this for the past few months and here is what I am doing and would be nice if docker was supported straight way in Mesos. So here goes the features that I would personally love to see in Mesos Core from the perspective of an user which I had to implement on my own -
>
> a. Mesos understanding docker metrics which should be straightforward because docker writes all its metrics in the following fashin for cpu, blkio, memory etc - /sys/fs/cgroup/cpu/docker/<conntainerid>
> I am sending all these metrics right now as a framework message back to my framework/scheduler but it would be cool if Mesos took care of them.
>
> b. Easier way to map tasks to docker containers, probably the external containerizer takes care of it to a large extent. It would be helpful if there was a blog about it's API and internals by the core committers explaining the design. Even a simple example in the mesos codebase using the external containerizer would help.
>
> c. stdout and stderr of docker containers in the Mesos task stdout and stderr logs. Does the external containerizer already takes care of it? I had to write a service which runs on every slave for exposing the container logs to an user.
>
> d. Mesos GC of tasks taking care of cleaning up docker containers which have terminated. Right now the way I implemented this is that the service which exposes the logs of a container also listens to docker events and when a container exits, it knows that this has to be cleaned up and so removes it after a fixed amount of time[configurable through a Rest API/config file]
>
> e. Translate all task constraints to docker run flags. This is probably the easiest and I know it's super easy to implement with the external containerizer.
>
>
> On Fri, Jun 20, 2014 at 3:40 PM, Tobias Knaup <to...@knaup.me> wrote:
> Hi all,
>
> We've got a lot of feedback from folks who use Mesos to run Dockers at scale via Deimos, and the main wish was to make Docker a first class citizen in Mesos, instead of a plugin that needs to be installed separately. Mesosphere wants to contribute this and I already chatted with Ben H about what an implementation could look like.
>
> I'd love for folks on here that are working with Docker to chime in!
> I created a JIRA here: https://issues.apache.org/jira/browse/MESOS-1524
>
> Cheers,
>
> Tobi
>
>
>
> --
> Thanks,
> Diptanu Choudhury
> Web - www.linkedin.com/in/diptanu
> Twitter - @diptanu
Re: Docker support in Mesos core
Posted by Tobias Knaup <to...@knaup.me>.
Hey Diptanu, these are all great suggestions. Do you mind creating JIRAs
for them?
Thanks!
On Fri, Jun 20, 2014 at 4:20 PM, Diptanu Choudhury <di...@gmail.com>
wrote:
> Great timing for this thread!
>
> I have been working on this for the past few months and here is what I am
> doing and would be nice if docker was supported straight way in Mesos. So
> here goes the features that I would personally love to see in Mesos Core
> from the perspective of an user which I had to implement on my own -
>
> a. Mesos understanding docker metrics which should be straightforward
> because docker writes all its metrics in the following fashin for cpu,
> blkio, memory etc - /sys/fs/cgroup/cpu/docker/<conntainerid>
> I am sending all these metrics right now as a framework message back to my
> framework/scheduler but it would be cool if Mesos took care of them.
>
> b. Easier way to map tasks to docker containers, probably the external
> containerizer takes care of it to a large extent. It would be helpful if
> there was a blog about it's API and internals by the core committers
> explaining the design. Even a simple example in the mesos codebase using
> the external containerizer would help.
>
> c. stdout and stderr of docker containers in the Mesos task stdout and
> stderr logs. Does the external containerizer already takes care of it? I
> had to write a service which runs on every slave for exposing the container
> logs to an user.
>
> d. Mesos GC of tasks taking care of cleaning up docker containers which
> have terminated. Right now the way I implemented this is that the service
> which exposes the logs of a container also listens to docker events and
> when a container exits, it knows that this has to be cleaned up and so
> removes it after a fixed amount of time[configurable through a Rest
> API/config file]
>
> e. Translate all task constraints to docker run flags. This is probably
> the easiest and I know it's super easy to implement with the external
> containerizer.
>
>
> On Fri, Jun 20, 2014 at 3:40 PM, Tobias Knaup <to...@knaup.me> wrote:
>
>> Hi all,
>>
>> We've got a lot of feedback from folks who use Mesos to run Dockers at
>> scale via Deimos, and the main wish was to make Docker a first class
>> citizen in Mesos, instead of a plugin that needs to be installed
>> separately. Mesosphere wants to contribute this and I already chatted with
>> Ben H about what an implementation could look like.
>>
>> I'd love for folks on here that are working with Docker to chime in!
>> I created a JIRA here: https://issues.apache.org/jira/browse/MESOS-1524
>>
>> Cheers,
>>
>> Tobi
>>
>
>
>
> --
> Thanks,
> Diptanu Choudhury
> Web - www.linkedin.com/in/diptanu
> Twitter - @diptanu <http://twitter.com/diptanu>
>
Re: Docker support in Mesos core
Posted by Tobias Knaup <to...@knaup.me>.
Hey Diptanu, these are all great suggestions. Do you mind creating JIRAs
for them?
Thanks!
On Fri, Jun 20, 2014 at 4:20 PM, Diptanu Choudhury <di...@gmail.com>
wrote:
> Great timing for this thread!
>
> I have been working on this for the past few months and here is what I am
> doing and would be nice if docker was supported straight way in Mesos. So
> here goes the features that I would personally love to see in Mesos Core
> from the perspective of an user which I had to implement on my own -
>
> a. Mesos understanding docker metrics which should be straightforward
> because docker writes all its metrics in the following fashin for cpu,
> blkio, memory etc - /sys/fs/cgroup/cpu/docker/<conntainerid>
> I am sending all these metrics right now as a framework message back to my
> framework/scheduler but it would be cool if Mesos took care of them.
>
> b. Easier way to map tasks to docker containers, probably the external
> containerizer takes care of it to a large extent. It would be helpful if
> there was a blog about it's API and internals by the core committers
> explaining the design. Even a simple example in the mesos codebase using
> the external containerizer would help.
>
> c. stdout and stderr of docker containers in the Mesos task stdout and
> stderr logs. Does the external containerizer already takes care of it? I
> had to write a service which runs on every slave for exposing the container
> logs to an user.
>
> d. Mesos GC of tasks taking care of cleaning up docker containers which
> have terminated. Right now the way I implemented this is that the service
> which exposes the logs of a container also listens to docker events and
> when a container exits, it knows that this has to be cleaned up and so
> removes it after a fixed amount of time[configurable through a Rest
> API/config file]
>
> e. Translate all task constraints to docker run flags. This is probably
> the easiest and I know it's super easy to implement with the external
> containerizer.
>
>
> On Fri, Jun 20, 2014 at 3:40 PM, Tobias Knaup <to...@knaup.me> wrote:
>
>> Hi all,
>>
>> We've got a lot of feedback from folks who use Mesos to run Dockers at
>> scale via Deimos, and the main wish was to make Docker a first class
>> citizen in Mesos, instead of a plugin that needs to be installed
>> separately. Mesosphere wants to contribute this and I already chatted with
>> Ben H about what an implementation could look like.
>>
>> I'd love for folks on here that are working with Docker to chime in!
>> I created a JIRA here: https://issues.apache.org/jira/browse/MESOS-1524
>>
>> Cheers,
>>
>> Tobi
>>
>
>
>
> --
> Thanks,
> Diptanu Choudhury
> Web - www.linkedin.com/in/diptanu
> Twitter - @diptanu <http://twitter.com/diptanu>
>
Re: Docker support in Mesos core
Posted by Diptanu Choudhury <di...@gmail.com>.
Great timing for this thread!
I have been working on this for the past few months and here is what I am
doing and would be nice if docker was supported straight way in Mesos. So
here goes the features that I would personally love to see in Mesos Core
from the perspective of an user which I had to implement on my own -
a. Mesos understanding docker metrics which should be straightforward
because docker writes all its metrics in the following fashin for cpu,
blkio, memory etc - /sys/fs/cgroup/cpu/docker/<conntainerid>
I am sending all these metrics right now as a framework message back to my
framework/scheduler but it would be cool if Mesos took care of them.
b. Easier way to map tasks to docker containers, probably the external
containerizer takes care of it to a large extent. It would be helpful if
there was a blog about it's API and internals by the core committers
explaining the design. Even a simple example in the mesos codebase using
the external containerizer would help.
c. stdout and stderr of docker containers in the Mesos task stdout and
stderr logs. Does the external containerizer already takes care of it? I
had to write a service which runs on every slave for exposing the container
logs to an user.
d. Mesos GC of tasks taking care of cleaning up docker containers which
have terminated. Right now the way I implemented this is that the service
which exposes the logs of a container also listens to docker events and
when a container exits, it knows that this has to be cleaned up and so
removes it after a fixed amount of time[configurable through a Rest
API/config file]
e. Translate all task constraints to docker run flags. This is probably the
easiest and I know it's super easy to implement with the external
containerizer.
On Fri, Jun 20, 2014 at 3:40 PM, Tobias Knaup <to...@knaup.me> wrote:
> Hi all,
>
> We've got a lot of feedback from folks who use Mesos to run Dockers at
> scale via Deimos, and the main wish was to make Docker a first class
> citizen in Mesos, instead of a plugin that needs to be installed
> separately. Mesosphere wants to contribute this and I already chatted with
> Ben H about what an implementation could look like.
>
> I'd love for folks on here that are working with Docker to chime in!
> I created a JIRA here: https://issues.apache.org/jira/browse/MESOS-1524
>
> Cheers,
>
> Tobi
>
--
Thanks,
Diptanu Choudhury
Web - www.linkedin.com/in/diptanu
Twitter - @diptanu <http://twitter.com/diptanu>
Re: Docker support in Mesos core
Posted by Dick Davies <di...@hellooperator.net>.
That's fantastic news, really good to see some integration happening
between chocolate and peanut butter
here. Deimos has been pretty difficult for us to deploy on our
platforms (largely down to the python implementation,
which has problems on the ancient python EL6 ships with).
On 20 June 2014 23:40, Tobias Knaup <to...@knaup.me> wrote:
> Hi all,
>
> We've got a lot of feedback from folks who use Mesos to run Dockers at scale
> via Deimos, and the main wish was to make Docker a first class citizen in
> Mesos, instead of a plugin that needs to be installed separately. Mesosphere
> wants to contribute this and I already chatted with Ben H about what an
> implementation could look like.
>
> I'd love for folks on here that are working with Docker to chime in!
> I created a JIRA here: https://issues.apache.org/jira/browse/MESOS-1524
>
> Cheers,
>
> Tobi