Mesos Docker design question

[Andy Grove <an...@codefutures.com>]

We've made good progress deploying our product with Mesos but feel like we
may need to move away from using the mesos docker executor and roll our own
but at the same time I am wondering if I am just looking at the problem in
the wrong way, not having that much experience with mesos.

The issue is that as well as being able to launch a docker container on a
slave, we also then want to be able to get information about the container
once it starts (like its ID or IP address) and write that information to
zookeeper.

Our current approach is:

1. Scheduler asks mesos to execute container (e.g. use mesos docker support
to issue the "docker run" command)
2. Have some code inside the container that gets the containers IP address
on startup and writes it to zookeeper

This works but the downside is each container/image must have this extra
step added.

There is a potential way of doing this in mesos instead but there are some
pieces missing:

1. Scheduler asks mesos to execute container (e.g. use mesos docker support
to issue the "docker run" command)
2. Scheduler receives statusUpdate() saying that the task is running (but
we don't know the container ID or container name)
3. Scheduler requests that the same slave now runs another task (custom
code in our product) that will get the container details and register them
with ZK

There is no way for the scheduler to know the container ID which means we
can't schedule the follow up task.

Even if we could do this, my concern would then be that step 3 might fail
if the slave no longer has spare resource.

I'd appreciate any feedback on best practices to achieve this.

Thanks,

Andy.

--
Andy Grove
VP Engineering
CodeFutures Corporation

Re: Mesos Docker design question

[Connor Doyle <co...@gmail.com>]

Andy, passing the "sidekick" container ID is one issue.  But aside
from that, if you have written a custom framework what's to stop you
from waiting for a resource offer that accommodates both containers
you want to schedule and then submitting two TaskInfos in the same
call to SchedulerDriver.launchTasks?
--
Connor

On Wed, Oct 15, 2014 at 5:36 PM, Tim Chen <ti...@mesosphere.io> wrote:
> Hi Andy,
>
> I've definitely been seeing similar use cases popping up, and you're right
> that nothing in Mesos right out of the box has any support for co-locating
> tasks for you.
>
> For your potential solution, I don't see why you will need the container
> name or ID? TaskStatus also has slaveId so you do know which slave you want
> to launch your second task on. You will need to keep a mapping yourself that
> for your given TaskId you can now launch your 2nd task locating in the same
> slave id once you have a offer from that slave.
>
> And yes your concern is correct about can not always gurantee you can either
> launch or when to launch your 2nd docker task.
>
> I believe we will be thinking about more how to launch a collection of tasks
> co-located together, and what that looks like in the near future. If you
> have more requirements and thoughts of how to do so please share them as
> well.
>
> Thanks!
>
> Tim
>
> On Tue, Oct 14, 2014 at 4:57 PM, Andy Grove <an...@codefutures.com>
> wrote:
>>
>> We've made good progress deploying our product with Mesos but feel like we
>> may need to move away from using the mesos docker executor and roll our own
>> but at the same time I am wondering if I am just looking at the problem in
>> the wrong way, not having that much experience with mesos.
>>
>> The issue is that as well as being able to launch a docker container on a
>> slave, we also then want to be able to get information about the container
>> once it starts (like its ID or IP address) and write that information to
>> zookeeper.
>>
>> Our current approach is:
>>
>> 1. Scheduler asks mesos to execute container (e.g. use mesos docker
>> support to issue the "docker run" command)
>> 2. Have some code inside the container that gets the containers IP address
>> on startup and writes it to zookeeper
>>
>> This works but the downside is each container/image must have this extra
>> step added.
>>
>> There is a potential way of doing this in mesos instead but there are some
>> pieces missing:
>>
>> 1. Scheduler asks mesos to execute container (e.g. use mesos docker
>> support to issue the "docker run" command)
>> 2. Scheduler receives statusUpdate() saying that the task is running (but
>> we don't know the container ID or container name)
>> 3. Scheduler requests that the same slave now runs another task (custom
>> code in our product) that will get the container details and register them
>> with ZK
>>
>> There is no way for the scheduler to know the container ID which means we
>> can't schedule the follow up task.
>>
>> Even if we could do this, my concern would then be that step 3 might fail
>> if the slave no longer has spare resource.
>>
>> I'd appreciate any feedback on best practices to achieve this.
>>
>> Thanks,
>>
>> Andy.
>>
>> --
>> Andy Grove
>> VP Engineering
>> CodeFutures Corporation
>>
>>
>



-- 
connor

Re: Mesos Docker design question

[Tim Chen <ti...@mesosphere.io>]

Hi Andy,

I've definitely been seeing similar use cases popping up, and you're right
that nothing in Mesos right out of the box has any support for co-locating
tasks for you.

For your potential solution, I don't see why you will need the container
name or ID? TaskStatus also has slaveId so you do know which slave you want
to launch your second task on. You will need to keep a mapping yourself
that for your given TaskId you can now launch your 2nd task locating in the
same slave id once you have a offer from that slave.

And yes your concern is correct about can not always gurantee you can
either launch or when to launch your 2nd docker task.

I believe we will be thinking about more how to launch a collection of
tasks co-located together, and what that looks like in the near future. If
you have more requirements and thoughts of how to do so please share them
as well.

Thanks!

Tim

On Tue, Oct 14, 2014 at 4:57 PM, Andy Grove <an...@codefutures.com>
wrote:

> We've made good progress deploying our product with Mesos but feel like we
> may need to move away from using the mesos docker executor and roll our own
> but at the same time I am wondering if I am just looking at the problem in
> the wrong way, not having that much experience with mesos.
>
> The issue is that as well as being able to launch a docker container on a
> slave, we also then want to be able to get information about the container
> once it starts (like its ID or IP address) and write that information to
> zookeeper.
>
> Our current approach is:
>
> 1. Scheduler asks mesos to execute container (e.g. use mesos docker
> support to issue the "docker run" command)
> 2. Have some code inside the container that gets the containers IP address
> on startup and writes it to zookeeper
>
> This works but the downside is each container/image must have this extra
> step added.
>
> There is a potential way of doing this in mesos instead but there are some
> pieces missing:
>
> 1. Scheduler asks mesos to execute container (e.g. use mesos docker
> support to issue the "docker run" command)
> 2. Scheduler receives statusUpdate() saying that the task is running (but
> we don't know the container ID or container name)
> 3. Scheduler requests that the same slave now runs another task (custom
> code in our product) that will get the container details and register them
> with ZK
>
> There is no way for the scheduler to know the container ID which means we
> can't schedule the follow up task.
>
> Even if we could do this, my concern would then be that step 3 might fail
> if the slave no longer has spare resource.
>
> I'd appreciate any feedback on best practices to achieve this.
>
> Thanks,
>
> Andy.
>
> --
> Andy Grove
> VP Engineering
> CodeFutures Corporation
>
>
>