You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@mesos.apache.org by Janet Borschowa <ja...@codefutures.com> on 2014/11/19 23:16:08 UTC
Implementing an Executor
Hi,
I'm implementing an executor which is used by the mesos slave to launch
tasks. The tasks are to launch a docker container - this is because I need
more info about the launched container than what the docker containerizer
returns.
Is it OK to block in the executor's launchTask method until the task
completes? If not, how does the framework discover when that task
completes? I could spawn a process which notifies my executor when the task
completes and then have my executor send a status update. Or is there some
other recommended way to deal with this when the task could run for an
indefinite period of time before completing its work?
Thanks!
Janet
--
Janet Borschowa
CodeFutures Corporation
Re: Implementing an Executor
Posted by Janet Borschowa <ja...@codefutures.com>.
Hi Tom,
Thank you - this helps a lot!
Janet
On Wed, Nov 19, 2014 at 5:22 PM, Tom Arnfeld <to...@duedil.com> wrote:
> Hi Janet,
>
> Oh sorry my mistake, I didn't read your email correctly, I thought you
> were using the containerizer. What you're doing here is actually going to
> be quite difficult to do, the mesos docker containerizer has some quite
> complex logic implemented to ensure the slave stays in sync with the
> containers that are running, and kills anything that goes rogue.
>
> It's going to be non-trivial for you to do that from the executor, though
> I guess you could make use of the docker events API or poll other endpoints
> in the API to check the status of your containers, and off the back of that
> send status updates to the cluster. Doing this however brings no guarantees
> that if your executor dies exceptionally (perhaps OOMd) the containers
> spawned will die... they'll keep running in the background and it'll be
> hard for you to know the state of your containers on the cluster.
>
> You probably want to be aware (if you don't know already) that the
> resource limits assigned to your tasks aren't going to be enforced by mesos
> because docker is running outside of its control. You'll need to pass the
> correct CPU/Memory limit parameters to your docker containers to ensure
> this happens correctly.
>
> Here are the docker API docs;
> https://docs.docker.com/reference/api/docker_remote_api_v1.15/
>
> Something you might want to consider, if all you're trying to do is allow
> your container access to details about itself (e.g `docker inspect`) is to
> open up the docker remote API to be queried by your containers on the
> slave, and switch to using the mesos docker containerizer.
>
> I hope that helps somewhat!
>
> Tom.
>
> --
>
> Tom Arnfeld
> Developer // DueDil
>
> (+44) 7525940046
> 25 Christopher Street, London, EC2A 2BS
>
>
> On Wed, Nov 19, 2014 at 10:16 PM, Janet Borschowa <
> janet.borschowa@codefutures.com> wrote:
>
>> Hi,
>> I'm implementing an executor which is used by the mesos slave to launch
>> tasks. The tasks are to launch a docker container - this is because I need
>> more info about the launched container than what the docker containerizer
>> returns.
>>
>> Is it OK to block in the executor's launchTask method until the task
>> completes? If not, how does the framework discover when that task
>> completes? I could spawn a process which notifies my executor when the task
>> completes and then have my executor send a status update. Or is there some
>> other recommended way to deal with this when the task could run for an
>> indefinite period of time before completing its work?
>>
>> Thanks!
>>
>> Janet
>>
>> --
>> Janet Borschowa
>> CodeFutures Corporation
>>
>>
>>
>
Re: Implementing an Executor
Posted by Janet Borschowa <ja...@codefutures.com>.
Hi Tim,
I'd like to get the NetworkSetting's IPAddress and Ports from the launched
docker container right now. But there is other information that I'll
probably need in the future such as the Volumes. What would be ideal is if
one could get the information about the docker container that is available
via docker's "inspect" command.
Thanks,
Janet
On Thu, Nov 20, 2014 at 1:23 AM, Tim Chen <ti...@mesosphere.io> wrote:
> Hi Janet,
>
> Can you elaborate more what you like to get back from the docker container
> that you launched?
>
> Thanks,
>
> Tim
>
> On Wed, Nov 19, 2014 at 5:22 PM, Tom Arnfeld <to...@duedil.com> wrote:
>
>> Hi Janet,
>>
>> Oh sorry my mistake, I didn't read your email correctly, I thought you
>> were using the containerizer. What you're doing here is actually going to
>> be quite difficult to do, the mesos docker containerizer has some quite
>> complex logic implemented to ensure the slave stays in sync with the
>> containers that are running, and kills anything that goes rogue.
>>
>> It's going to be non-trivial for you to do that from the executor, though
>> I guess you could make use of the docker events API or poll other endpoints
>> in the API to check the status of your containers, and off the back of that
>> send status updates to the cluster. Doing this however brings no guarantees
>> that if your executor dies exceptionally (perhaps OOMd) the containers
>> spawned will die... they'll keep running in the background and it'll be
>> hard for you to know the state of your containers on the cluster.
>>
>> You probably want to be aware (if you don't know already) that the
>> resource limits assigned to your tasks aren't going to be enforced by mesos
>> because docker is running outside of its control. You'll need to pass the
>> correct CPU/Memory limit parameters to your docker containers to ensure
>> this happens correctly.
>>
>> Here are the docker API docs;
>> https://docs.docker.com/reference/api/docker_remote_api_v1.15/
>>
>> Something you might want to consider, if all you're trying to do is allow
>> your container access to details about itself (e.g `docker inspect`) is to
>> open up the docker remote API to be queried by your containers on the
>> slave, and switch to using the mesos docker containerizer.
>>
>> I hope that helps somewhat!
>>
>> Tom.
>>
>> --
>>
>> Tom Arnfeld
>> Developer // DueDil
>>
>> (+44) 7525940046
>> 25 Christopher Street, London, EC2A 2BS
>>
>>
>> On Wed, Nov 19, 2014 at 10:16 PM, Janet Borschowa <
>> janet.borschowa@codefutures.com> wrote:
>>
>>> Hi,
>>> I'm implementing an executor which is used by the mesos slave to launch
>>> tasks. The tasks are to launch a docker container - this is because I need
>>> more info about the launched container than what the docker containerizer
>>> returns.
>>>
>>> Is it OK to block in the executor's launchTask method until the task
>>> completes? If not, how does the framework discover when that task
>>> completes? I could spawn a process which notifies my executor when the task
>>> completes and then have my executor send a status update. Or is there some
>>> other recommended way to deal with this when the task could run for an
>>> indefinite period of time before completing its work?
>>>
>>> Thanks!
>>>
>>> Janet
>>>
>>> --
>>> Janet Borschowa
>>> CodeFutures Corporation
>>>
>>>
>>>
>>
>
Re: Implementing an Executor
Posted by Benjamin Mahler <be...@gmail.com>.
Got it, thanks for clarifying. :)
On Mon, Dec 1, 2014 at 1:40 PM, Diptanu Choudhury <di...@gmail.com>
wrote:
> Hey Ben,
>
> We are not using the Docker containerizer from Mesos Core yet, we just
> have a plain executor. And by "container" I meant the docker container
> here. The docker containers in this case are started by the executor [which
> uses Docker's remote API]. Mesos does kill the container that it creates to
> run the executor but however the docker containers are not controlled by
> Mesos Slave.
>
> Also because Mesos doesn't know the existence of those containers because
> the executor was monitoring them, task reconciliation won't help either.
> This is another downside of this approach. So we use an out-of-band
> communication here to kill containers that the Framework and Mesos Master
> thinks should not be running.
>
> Our main problem is - we want the docker containers to be setup in a
> certain way, for example attach elastic network interfaces to containers,
> mount ZFS volumes etc, and that's why we had to write a custom executor in
> the first place.
>
>
> On Mon, Dec 1, 2014 at 1:01 PM, Benjamin Mahler <benjamin.mahler@gmail.com
> > wrote:
>
>> Sorry this is a bit of a tangent to the thread:
>>
>> > For ex, if the executor crashes Mesos would get a TASK_LOST while the
>> container might still be running.
>>
>> Mesos should destroy the container when the executor exits, are you
>> seeing otherwise?
>>
>> > So we are doing something similar to Aurora's GC Executor to clean up
>> containers behind us.
>>
>> Just to make sure you're aware, you don't need to do this with 0.21.0+,
>> now that we provide reconciliation support. :)
>>
>> On Mon, Dec 1, 2014 at 12:56 PM, Diptanu Choudhury <di...@gmail.com>
>> wrote:
>>
>>> There are some downsides to this as well - For ex, if the executor
>>> crashes Mesos would get a TASK_LOST while the container might still be
>>> running. So we are doing something similar to Aurora's GC Executor to clean
>>> up containers behind us.
>>>
>>> On Mon, Dec 1, 2014 at 10:19 AM, Janet Borschowa <
>>> janet.borschowa@codefutures.com> wrote:
>>>
>>>> Hi Diptanu,
>>>> Thanks for your feedback. This sounds like the approach we'll be
>>>> taking, too.
>>>>
>>>>
>>>> On Sat, Nov 29, 2014 at 11:34 AM, Diptanu Choudhury <diptanuc@gmail.com
>>>> > wrote:
>>>>
>>>>> Hi Janet,
>>>>>
>>>>> We implemented the same in our Titan Mesos Executor. We have an
>>>>> executor manage multiple containers. We didn't want the overhead of running
>>>>> multiple executors to manage multiple containers. We are using RxJava
>>>>> heavily in our codebase, so whenever a launchTask callback is invoked in
>>>>> the MesosExecutor we publish an event and return immediately from the
>>>>> callback so that our executor can handle more callbacks from Mesos Slave.
>>>>>
>>>>> But the event which is published is picked up by the subscriber and
>>>>> that starts the workflow of downloading a container, mount volumes, sets up
>>>>> networking, sets up logging, and finally starts a monitoring process for
>>>>> the container. The monitoring process has the reference to the executor
>>>>> driver and we send back Mesos status updates whenever the state of the
>>>>> process changes.
>>>>>
>>>>>
>>>>> On Thu, Nov 20, 2014 at 8:40 AM, Janet Borschowa <
>>>>> janet.borschowa@codefutures.com> wrote:
>>>>>
>>>>>> Hi David,
>>>>>> Thanks for your suggestion. This approach sounds promising for what I
>>>>>> need to do. I'm going to have to try this out.
>>>>>>
>>>>>> Thanks!
>>>>>> Janet
>>>>>>
>>>>>> --
>>>>>>
>>>>>> Janet Borschowa
>>>>>> CodeFutures Corporation
>>>>>>
>>>>>> On Thu, Nov 20, 2014 at 5:04 AM, David Greenberg <
>>>>>> dsg123456789@gmail.com> wrote:
>>>>>>
>>>>>>> One cool feature of the docker containerizer is that you can
>>>>>>> actually launch your executor inside the docker image, so that you can just
>>>>>>> layer the executor's custom logic on top of whatever container you desire.
>>>>>>> This way, you can more easily control what's happening in the docker image,
>>>>>>> and still leverage the containerizer.
>>>>>>>
>>>>>>> On Thursday, November 20, 2014, Tim Chen <ti...@mesosphere.io> wrote:
>>>>>>>
>>>>>>>> Hi Janet,
>>>>>>>>
>>>>>>>> Can you elaborate more what you like to get back from the docker
>>>>>>>> container that you launched?
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>>
>>>>>>>> Tim
>>>>>>>>
>>>>>>>> On Wed, Nov 19, 2014 at 5:22 PM, Tom Arnfeld <to...@duedil.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hi Janet,
>>>>>>>>>
>>>>>>>>> Oh sorry my mistake, I didn't read your email correctly, I thought
>>>>>>>>> you were using the containerizer. What you're doing here is actually going
>>>>>>>>> to be quite difficult to do, the mesos docker containerizer has some quite
>>>>>>>>> complex logic implemented to ensure the slave stays in sync with the
>>>>>>>>> containers that are running, and kills anything that goes rogue.
>>>>>>>>>
>>>>>>>>> It's going to be non-trivial for you to do that from the executor,
>>>>>>>>> though I guess you could make use of the docker events API or poll other
>>>>>>>>> endpoints in the API to check the status of your containers, and off the
>>>>>>>>> back of that send status updates to the cluster. Doing this however brings
>>>>>>>>> no guarantees that if your executor dies exceptionally (perhaps OOMd) the
>>>>>>>>> containers spawned will die... they'll keep running in the background and
>>>>>>>>> it'll be hard for you to know the state of your containers on the cluster.
>>>>>>>>>
>>>>>>>>> You probably want to be aware (if you don't know already) that the
>>>>>>>>> resource limits assigned to your tasks aren't going to be enforced by mesos
>>>>>>>>> because docker is running outside of its control. You'll need to pass the
>>>>>>>>> correct CPU/Memory limit parameters to your docker containers to ensure
>>>>>>>>> this happens correctly.
>>>>>>>>>
>>>>>>>>> Here are the docker API docs;
>>>>>>>>> https://docs.docker.com/reference/api/docker_remote_api_v1.15/
>>>>>>>>>
>>>>>>>>> Something you might want to consider, if all you're trying to do
>>>>>>>>> is allow your container access to details about itself (e.g `docker
>>>>>>>>> inspect`) is to open up the docker remote API to be queried by your
>>>>>>>>> containers on the slave, and switch to using the mesos docker containerizer.
>>>>>>>>>
>>>>>>>>> I hope that helps somewhat!
>>>>>>>>>
>>>>>>>>> Tom.
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>>
>>>>>>>>> Tom Arnfeld
>>>>>>>>> Developer // DueDil
>>>>>>>>>
>>>>>>>>> (+44) 7525940046
>>>>>>>>> 25 Christopher Street, London, EC2A 2BS
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Wed, Nov 19, 2014 at 10:16 PM, Janet Borschowa <
>>>>>>>>> janet.borschowa@codefutures.com> wrote:
>>>>>>>>>
>>>>>>>>>> Hi,
>>>>>>>>>> I'm implementing an executor which is used by the mesos slave to
>>>>>>>>>> launch tasks. The tasks are to launch a docker container - this is because
>>>>>>>>>> I need more info about the launched container than what the docker
>>>>>>>>>> containerizer returns.
>>>>>>>>>>
>>>>>>>>>> Is it OK to block in the executor's launchTask method until the
>>>>>>>>>> task completes? If not, how does the framework discover when that task
>>>>>>>>>> completes? I could spawn a process which notifies my executor when the task
>>>>>>>>>> completes and then have my executor send a status update. Or is there some
>>>>>>>>>> other recommended way to deal with this when the task could run for an
>>>>>>>>>> indefinite period of time before completing its work?
>>>>>>>>>>
>>>>>>>>>> Thanks!
>>>>>>>>>>
>>>>>>>>>> Janet
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Janet Borschowa
>>>>>>>>>> CodeFutures Corporation
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Thanks,
>>>>> Diptanu Choudhury
>>>>> Web - www.linkedin.com/in/diptanu
>>>>> Twitter - @diptanu <http://twitter.com/diptanu>
>>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Thanks,
>>> Diptanu Choudhury
>>> Web - www.linkedin.com/in/diptanu
>>> Twitter - @diptanu <http://twitter.com/diptanu>
>>>
>>
>>
>
>
> --
> Thanks,
> Diptanu Choudhury
> Web - www.linkedin.com/in/diptanu
> Twitter - @diptanu <http://twitter.com/diptanu>
>
Re: Implementing an Executor
Posted by Diptanu Choudhury <di...@gmail.com>.
Hey Ben,
We are not using the Docker containerizer from Mesos Core yet, we just have
a plain executor. And by "container" I meant the docker container here. The
docker containers in this case are started by the executor [which uses
Docker's remote API]. Mesos does kill the container that it creates to run
the executor but however the docker containers are not controlled by Mesos
Slave.
Also because Mesos doesn't know the existence of those containers because
the executor was monitoring them, task reconciliation won't help either.
This is another downside of this approach. So we use an out-of-band
communication here to kill containers that the Framework and Mesos Master
thinks should not be running.
Our main problem is - we want the docker containers to be setup in a
certain way, for example attach elastic network interfaces to containers,
mount ZFS volumes etc, and that's why we had to write a custom executor in
the first place.
On Mon, Dec 1, 2014 at 1:01 PM, Benjamin Mahler <be...@gmail.com>
wrote:
> Sorry this is a bit of a tangent to the thread:
>
> > For ex, if the executor crashes Mesos would get a TASK_LOST while the
> container might still be running.
>
> Mesos should destroy the container when the executor exits, are you seeing
> otherwise?
>
> > So we are doing something similar to Aurora's GC Executor to clean up
> containers behind us.
>
> Just to make sure you're aware, you don't need to do this with 0.21.0+,
> now that we provide reconciliation support. :)
>
> On Mon, Dec 1, 2014 at 12:56 PM, Diptanu Choudhury <di...@gmail.com>
> wrote:
>
>> There are some downsides to this as well - For ex, if the executor
>> crashes Mesos would get a TASK_LOST while the container might still be
>> running. So we are doing something similar to Aurora's GC Executor to clean
>> up containers behind us.
>>
>> On Mon, Dec 1, 2014 at 10:19 AM, Janet Borschowa <
>> janet.borschowa@codefutures.com> wrote:
>>
>>> Hi Diptanu,
>>> Thanks for your feedback. This sounds like the approach we'll be taking,
>>> too.
>>>
>>>
>>> On Sat, Nov 29, 2014 at 11:34 AM, Diptanu Choudhury <di...@gmail.com>
>>> wrote:
>>>
>>>> Hi Janet,
>>>>
>>>> We implemented the same in our Titan Mesos Executor. We have an
>>>> executor manage multiple containers. We didn't want the overhead of running
>>>> multiple executors to manage multiple containers. We are using RxJava
>>>> heavily in our codebase, so whenever a launchTask callback is invoked in
>>>> the MesosExecutor we publish an event and return immediately from the
>>>> callback so that our executor can handle more callbacks from Mesos Slave.
>>>>
>>>> But the event which is published is picked up by the subscriber and
>>>> that starts the workflow of downloading a container, mount volumes, sets up
>>>> networking, sets up logging, and finally starts a monitoring process for
>>>> the container. The monitoring process has the reference to the executor
>>>> driver and we send back Mesos status updates whenever the state of the
>>>> process changes.
>>>>
>>>>
>>>> On Thu, Nov 20, 2014 at 8:40 AM, Janet Borschowa <
>>>> janet.borschowa@codefutures.com> wrote:
>>>>
>>>>> Hi David,
>>>>> Thanks for your suggestion. This approach sounds promising for what I
>>>>> need to do. I'm going to have to try this out.
>>>>>
>>>>> Thanks!
>>>>> Janet
>>>>>
>>>>> --
>>>>>
>>>>> Janet Borschowa
>>>>> CodeFutures Corporation
>>>>>
>>>>> On Thu, Nov 20, 2014 at 5:04 AM, David Greenberg <
>>>>> dsg123456789@gmail.com> wrote:
>>>>>
>>>>>> One cool feature of the docker containerizer is that you can actually
>>>>>> launch your executor inside the docker image, so that you can just layer
>>>>>> the executor's custom logic on top of whatever container you desire. This
>>>>>> way, you can more easily control what's happening in the docker image, and
>>>>>> still leverage the containerizer.
>>>>>>
>>>>>> On Thursday, November 20, 2014, Tim Chen <ti...@mesosphere.io> wrote:
>>>>>>
>>>>>>> Hi Janet,
>>>>>>>
>>>>>>> Can you elaborate more what you like to get back from the docker
>>>>>>> container that you launched?
>>>>>>>
>>>>>>> Thanks,
>>>>>>>
>>>>>>> Tim
>>>>>>>
>>>>>>> On Wed, Nov 19, 2014 at 5:22 PM, Tom Arnfeld <to...@duedil.com> wrote:
>>>>>>>
>>>>>>>> Hi Janet,
>>>>>>>>
>>>>>>>> Oh sorry my mistake, I didn't read your email correctly, I thought
>>>>>>>> you were using the containerizer. What you're doing here is actually going
>>>>>>>> to be quite difficult to do, the mesos docker containerizer has some quite
>>>>>>>> complex logic implemented to ensure the slave stays in sync with the
>>>>>>>> containers that are running, and kills anything that goes rogue.
>>>>>>>>
>>>>>>>> It's going to be non-trivial for you to do that from the executor,
>>>>>>>> though I guess you could make use of the docker events API or poll other
>>>>>>>> endpoints in the API to check the status of your containers, and off the
>>>>>>>> back of that send status updates to the cluster. Doing this however brings
>>>>>>>> no guarantees that if your executor dies exceptionally (perhaps OOMd) the
>>>>>>>> containers spawned will die... they'll keep running in the background and
>>>>>>>> it'll be hard for you to know the state of your containers on the cluster.
>>>>>>>>
>>>>>>>> You probably want to be aware (if you don't know already) that the
>>>>>>>> resource limits assigned to your tasks aren't going to be enforced by mesos
>>>>>>>> because docker is running outside of its control. You'll need to pass the
>>>>>>>> correct CPU/Memory limit parameters to your docker containers to ensure
>>>>>>>> this happens correctly.
>>>>>>>>
>>>>>>>> Here are the docker API docs;
>>>>>>>> https://docs.docker.com/reference/api/docker_remote_api_v1.15/
>>>>>>>>
>>>>>>>> Something you might want to consider, if all you're trying to do is
>>>>>>>> allow your container access to details about itself (e.g `docker inspect`)
>>>>>>>> is to open up the docker remote API to be queried by your containers on the
>>>>>>>> slave, and switch to using the mesos docker containerizer.
>>>>>>>>
>>>>>>>> I hope that helps somewhat!
>>>>>>>>
>>>>>>>> Tom.
>>>>>>>>
>>>>>>>> --
>>>>>>>>
>>>>>>>> Tom Arnfeld
>>>>>>>> Developer // DueDil
>>>>>>>>
>>>>>>>> (+44) 7525940046
>>>>>>>> 25 Christopher Street, London, EC2A 2BS
>>>>>>>>
>>>>>>>>
>>>>>>>> On Wed, Nov 19, 2014 at 10:16 PM, Janet Borschowa <
>>>>>>>> janet.borschowa@codefutures.com> wrote:
>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>> I'm implementing an executor which is used by the mesos slave to
>>>>>>>>> launch tasks. The tasks are to launch a docker container - this is because
>>>>>>>>> I need more info about the launched container than what the docker
>>>>>>>>> containerizer returns.
>>>>>>>>>
>>>>>>>>> Is it OK to block in the executor's launchTask method until the
>>>>>>>>> task completes? If not, how does the framework discover when that task
>>>>>>>>> completes? I could spawn a process which notifies my executor when the task
>>>>>>>>> completes and then have my executor send a status update. Or is there some
>>>>>>>>> other recommended way to deal with this when the task could run for an
>>>>>>>>> indefinite period of time before completing its work?
>>>>>>>>>
>>>>>>>>> Thanks!
>>>>>>>>>
>>>>>>>>> Janet
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Janet Borschowa
>>>>>>>>> CodeFutures Corporation
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Thanks,
>>>> Diptanu Choudhury
>>>> Web - www.linkedin.com/in/diptanu
>>>> Twitter - @diptanu <http://twitter.com/diptanu>
>>>>
>>>
>>>
>>
>>
>> --
>> Thanks,
>> Diptanu Choudhury
>> Web - www.linkedin.com/in/diptanu
>> Twitter - @diptanu <http://twitter.com/diptanu>
>>
>
>
--
Thanks,
Diptanu Choudhury
Web - www.linkedin.com/in/diptanu
Twitter - @diptanu <http://twitter.com/diptanu>
Re: Implementing an Executor
Posted by Benjamin Mahler <be...@gmail.com>.
Sorry this is a bit of a tangent to the thread:
> For ex, if the executor crashes Mesos would get a TASK_LOST while the
container might still be running.
Mesos should destroy the container when the executor exits, are you seeing
otherwise?
> So we are doing something similar to Aurora's GC Executor to clean up
containers behind us.
Just to make sure you're aware, you don't need to do this with 0.21.0+, now
that we provide reconciliation support. :)
On Mon, Dec 1, 2014 at 12:56 PM, Diptanu Choudhury <di...@gmail.com>
wrote:
> There are some downsides to this as well - For ex, if the executor crashes
> Mesos would get a TASK_LOST while the container might still be running. So
> we are doing something similar to Aurora's GC Executor to clean up
> containers behind us.
>
> On Mon, Dec 1, 2014 at 10:19 AM, Janet Borschowa <
> janet.borschowa@codefutures.com> wrote:
>
>> Hi Diptanu,
>> Thanks for your feedback. This sounds like the approach we'll be taking,
>> too.
>>
>>
>> On Sat, Nov 29, 2014 at 11:34 AM, Diptanu Choudhury <di...@gmail.com>
>> wrote:
>>
>>> Hi Janet,
>>>
>>> We implemented the same in our Titan Mesos Executor. We have an executor
>>> manage multiple containers. We didn't want the overhead of running multiple
>>> executors to manage multiple containers. We are using RxJava heavily in our
>>> codebase, so whenever a launchTask callback is invoked in the MesosExecutor
>>> we publish an event and return immediately from the callback so that our
>>> executor can handle more callbacks from Mesos Slave.
>>>
>>> But the event which is published is picked up by the subscriber and that
>>> starts the workflow of downloading a container, mount volumes, sets up
>>> networking, sets up logging, and finally starts a monitoring process for
>>> the container. The monitoring process has the reference to the executor
>>> driver and we send back Mesos status updates whenever the state of the
>>> process changes.
>>>
>>>
>>> On Thu, Nov 20, 2014 at 8:40 AM, Janet Borschowa <
>>> janet.borschowa@codefutures.com> wrote:
>>>
>>>> Hi David,
>>>> Thanks for your suggestion. This approach sounds promising for what I
>>>> need to do. I'm going to have to try this out.
>>>>
>>>> Thanks!
>>>> Janet
>>>>
>>>> --
>>>>
>>>> Janet Borschowa
>>>> CodeFutures Corporation
>>>>
>>>> On Thu, Nov 20, 2014 at 5:04 AM, David Greenberg <
>>>> dsg123456789@gmail.com> wrote:
>>>>
>>>>> One cool feature of the docker containerizer is that you can actually
>>>>> launch your executor inside the docker image, so that you can just layer
>>>>> the executor's custom logic on top of whatever container you desire. This
>>>>> way, you can more easily control what's happening in the docker image, and
>>>>> still leverage the containerizer.
>>>>>
>>>>> On Thursday, November 20, 2014, Tim Chen <ti...@mesosphere.io> wrote:
>>>>>
>>>>>> Hi Janet,
>>>>>>
>>>>>> Can you elaborate more what you like to get back from the docker
>>>>>> container that you launched?
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> Tim
>>>>>>
>>>>>> On Wed, Nov 19, 2014 at 5:22 PM, Tom Arnfeld <to...@duedil.com> wrote:
>>>>>>
>>>>>>> Hi Janet,
>>>>>>>
>>>>>>> Oh sorry my mistake, I didn't read your email correctly, I thought
>>>>>>> you were using the containerizer. What you're doing here is actually going
>>>>>>> to be quite difficult to do, the mesos docker containerizer has some quite
>>>>>>> complex logic implemented to ensure the slave stays in sync with the
>>>>>>> containers that are running, and kills anything that goes rogue.
>>>>>>>
>>>>>>> It's going to be non-trivial for you to do that from the executor,
>>>>>>> though I guess you could make use of the docker events API or poll other
>>>>>>> endpoints in the API to check the status of your containers, and off the
>>>>>>> back of that send status updates to the cluster. Doing this however brings
>>>>>>> no guarantees that if your executor dies exceptionally (perhaps OOMd) the
>>>>>>> containers spawned will die... they'll keep running in the background and
>>>>>>> it'll be hard for you to know the state of your containers on the cluster.
>>>>>>>
>>>>>>> You probably want to be aware (if you don't know already) that the
>>>>>>> resource limits assigned to your tasks aren't going to be enforced by mesos
>>>>>>> because docker is running outside of its control. You'll need to pass the
>>>>>>> correct CPU/Memory limit parameters to your docker containers to ensure
>>>>>>> this happens correctly.
>>>>>>>
>>>>>>> Here are the docker API docs;
>>>>>>> https://docs.docker.com/reference/api/docker_remote_api_v1.15/
>>>>>>>
>>>>>>> Something you might want to consider, if all you're trying to do is
>>>>>>> allow your container access to details about itself (e.g `docker inspect`)
>>>>>>> is to open up the docker remote API to be queried by your containers on the
>>>>>>> slave, and switch to using the mesos docker containerizer.
>>>>>>>
>>>>>>> I hope that helps somewhat!
>>>>>>>
>>>>>>> Tom.
>>>>>>>
>>>>>>> --
>>>>>>>
>>>>>>> Tom Arnfeld
>>>>>>> Developer // DueDil
>>>>>>>
>>>>>>> (+44) 7525940046
>>>>>>> 25 Christopher Street, London, EC2A 2BS
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Nov 19, 2014 at 10:16 PM, Janet Borschowa <
>>>>>>> janet.borschowa@codefutures.com> wrote:
>>>>>>>
>>>>>>>> Hi,
>>>>>>>> I'm implementing an executor which is used by the mesos slave to
>>>>>>>> launch tasks. The tasks are to launch a docker container - this is because
>>>>>>>> I need more info about the launched container than what the docker
>>>>>>>> containerizer returns.
>>>>>>>>
>>>>>>>> Is it OK to block in the executor's launchTask method until the
>>>>>>>> task completes? If not, how does the framework discover when that task
>>>>>>>> completes? I could spawn a process which notifies my executor when the task
>>>>>>>> completes and then have my executor send a status update. Or is there some
>>>>>>>> other recommended way to deal with this when the task could run for an
>>>>>>>> indefinite period of time before completing its work?
>>>>>>>>
>>>>>>>> Thanks!
>>>>>>>>
>>>>>>>> Janet
>>>>>>>>
>>>>>>>> --
>>>>>>>> Janet Borschowa
>>>>>>>> CodeFutures Corporation
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>
>>>
>>>
>>> --
>>> Thanks,
>>> Diptanu Choudhury
>>> Web - www.linkedin.com/in/diptanu
>>> Twitter - @diptanu <http://twitter.com/diptanu>
>>>
>>
>>
>
>
> --
> Thanks,
> Diptanu Choudhury
> Web - www.linkedin.com/in/diptanu
> Twitter - @diptanu <http://twitter.com/diptanu>
>
Re: Implementing an Executor
Posted by Diptanu Choudhury <di...@gmail.com>.
There are some downsides to this as well - For ex, if the executor crashes
Mesos would get a TASK_LOST while the container might still be running. So
we are doing something similar to Aurora's GC Executor to clean up
containers behind us.
On Mon, Dec 1, 2014 at 10:19 AM, Janet Borschowa <
janet.borschowa@codefutures.com> wrote:
> Hi Diptanu,
> Thanks for your feedback. This sounds like the approach we'll be taking,
> too.
>
>
> On Sat, Nov 29, 2014 at 11:34 AM, Diptanu Choudhury <di...@gmail.com>
> wrote:
>
>> Hi Janet,
>>
>> We implemented the same in our Titan Mesos Executor. We have an executor
>> manage multiple containers. We didn't want the overhead of running multiple
>> executors to manage multiple containers. We are using RxJava heavily in our
>> codebase, so whenever a launchTask callback is invoked in the MesosExecutor
>> we publish an event and return immediately from the callback so that our
>> executor can handle more callbacks from Mesos Slave.
>>
>> But the event which is published is picked up by the subscriber and that
>> starts the workflow of downloading a container, mount volumes, sets up
>> networking, sets up logging, and finally starts a monitoring process for
>> the container. The monitoring process has the reference to the executor
>> driver and we send back Mesos status updates whenever the state of the
>> process changes.
>>
>>
>> On Thu, Nov 20, 2014 at 8:40 AM, Janet Borschowa <
>> janet.borschowa@codefutures.com> wrote:
>>
>>> Hi David,
>>> Thanks for your suggestion. This approach sounds promising for what I
>>> need to do. I'm going to have to try this out.
>>>
>>> Thanks!
>>> Janet
>>>
>>> --
>>>
>>> Janet Borschowa
>>> CodeFutures Corporation
>>>
>>> On Thu, Nov 20, 2014 at 5:04 AM, David Greenberg <dsg123456789@gmail.com
>>> > wrote:
>>>
>>>> One cool feature of the docker containerizer is that you can actually
>>>> launch your executor inside the docker image, so that you can just layer
>>>> the executor's custom logic on top of whatever container you desire. This
>>>> way, you can more easily control what's happening in the docker image, and
>>>> still leverage the containerizer.
>>>>
>>>> On Thursday, November 20, 2014, Tim Chen <ti...@mesosphere.io> wrote:
>>>>
>>>>> Hi Janet,
>>>>>
>>>>> Can you elaborate more what you like to get back from the docker
>>>>> container that you launched?
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Tim
>>>>>
>>>>> On Wed, Nov 19, 2014 at 5:22 PM, Tom Arnfeld <to...@duedil.com> wrote:
>>>>>
>>>>>> Hi Janet,
>>>>>>
>>>>>> Oh sorry my mistake, I didn't read your email correctly, I thought
>>>>>> you were using the containerizer. What you're doing here is actually going
>>>>>> to be quite difficult to do, the mesos docker containerizer has some quite
>>>>>> complex logic implemented to ensure the slave stays in sync with the
>>>>>> containers that are running, and kills anything that goes rogue.
>>>>>>
>>>>>> It's going to be non-trivial for you to do that from the executor,
>>>>>> though I guess you could make use of the docker events API or poll other
>>>>>> endpoints in the API to check the status of your containers, and off the
>>>>>> back of that send status updates to the cluster. Doing this however brings
>>>>>> no guarantees that if your executor dies exceptionally (perhaps OOMd) the
>>>>>> containers spawned will die... they'll keep running in the background and
>>>>>> it'll be hard for you to know the state of your containers on the cluster.
>>>>>>
>>>>>> You probably want to be aware (if you don't know already) that the
>>>>>> resource limits assigned to your tasks aren't going to be enforced by mesos
>>>>>> because docker is running outside of its control. You'll need to pass the
>>>>>> correct CPU/Memory limit parameters to your docker containers to ensure
>>>>>> this happens correctly.
>>>>>>
>>>>>> Here are the docker API docs;
>>>>>> https://docs.docker.com/reference/api/docker_remote_api_v1.15/
>>>>>>
>>>>>> Something you might want to consider, if all you're trying to do is
>>>>>> allow your container access to details about itself (e.g `docker inspect`)
>>>>>> is to open up the docker remote API to be queried by your containers on the
>>>>>> slave, and switch to using the mesos docker containerizer.
>>>>>>
>>>>>> I hope that helps somewhat!
>>>>>>
>>>>>> Tom.
>>>>>>
>>>>>> --
>>>>>>
>>>>>> Tom Arnfeld
>>>>>> Developer // DueDil
>>>>>>
>>>>>> (+44) 7525940046
>>>>>> 25 Christopher Street, London, EC2A 2BS
>>>>>>
>>>>>>
>>>>>> On Wed, Nov 19, 2014 at 10:16 PM, Janet Borschowa <
>>>>>> janet.borschowa@codefutures.com> wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>> I'm implementing an executor which is used by the mesos slave to
>>>>>>> launch tasks. The tasks are to launch a docker container - this is because
>>>>>>> I need more info about the launched container than what the docker
>>>>>>> containerizer returns.
>>>>>>>
>>>>>>> Is it OK to block in the executor's launchTask method until the task
>>>>>>> completes? If not, how does the framework discover when that task
>>>>>>> completes? I could spawn a process which notifies my executor when the task
>>>>>>> completes and then have my executor send a status update. Or is there some
>>>>>>> other recommended way to deal with this when the task could run for an
>>>>>>> indefinite period of time before completing its work?
>>>>>>>
>>>>>>> Thanks!
>>>>>>>
>>>>>>> Janet
>>>>>>>
>>>>>>> --
>>>>>>> Janet Borschowa
>>>>>>> CodeFutures Corporation
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>
>>
>>
>> --
>> Thanks,
>> Diptanu Choudhury
>> Web - www.linkedin.com/in/diptanu
>> Twitter - @diptanu <http://twitter.com/diptanu>
>>
>
>
--
Thanks,
Diptanu Choudhury
Web - www.linkedin.com/in/diptanu
Twitter - @diptanu <http://twitter.com/diptanu>
Re: Implementing an Executor
Posted by Janet Borschowa <ja...@codefutures.com>.
Hi Diptanu,
Thanks for your feedback. This sounds like the approach we'll be taking,
too.
On Sat, Nov 29, 2014 at 11:34 AM, Diptanu Choudhury <di...@gmail.com>
wrote:
> Hi Janet,
>
> We implemented the same in our Titan Mesos Executor. We have an executor
> manage multiple containers. We didn't want the overhead of running multiple
> executors to manage multiple containers. We are using RxJava heavily in our
> codebase, so whenever a launchTask callback is invoked in the MesosExecutor
> we publish an event and return immediately from the callback so that our
> executor can handle more callbacks from Mesos Slave.
>
> But the event which is published is picked up by the subscriber and that
> starts the workflow of downloading a container, mount volumes, sets up
> networking, sets up logging, and finally starts a monitoring process for
> the container. The monitoring process has the reference to the executor
> driver and we send back Mesos status updates whenever the state of the
> process changes.
>
>
> On Thu, Nov 20, 2014 at 8:40 AM, Janet Borschowa <
> janet.borschowa@codefutures.com> wrote:
>
>> Hi David,
>> Thanks for your suggestion. This approach sounds promising for what I
>> need to do. I'm going to have to try this out.
>>
>> Thanks!
>> Janet
>>
>> --
>>
>> Janet Borschowa
>> CodeFutures Corporation
>>
>> On Thu, Nov 20, 2014 at 5:04 AM, David Greenberg <ds...@gmail.com>
>> wrote:
>>
>>> One cool feature of the docker containerizer is that you can actually
>>> launch your executor inside the docker image, so that you can just layer
>>> the executor's custom logic on top of whatever container you desire. This
>>> way, you can more easily control what's happening in the docker image, and
>>> still leverage the containerizer.
>>>
>>> On Thursday, November 20, 2014, Tim Chen <ti...@mesosphere.io> wrote:
>>>
>>>> Hi Janet,
>>>>
>>>> Can you elaborate more what you like to get back from the docker
>>>> container that you launched?
>>>>
>>>> Thanks,
>>>>
>>>> Tim
>>>>
>>>> On Wed, Nov 19, 2014 at 5:22 PM, Tom Arnfeld <to...@duedil.com> wrote:
>>>>
>>>>> Hi Janet,
>>>>>
>>>>> Oh sorry my mistake, I didn't read your email correctly, I thought you
>>>>> were using the containerizer. What you're doing here is actually going to
>>>>> be quite difficult to do, the mesos docker containerizer has some quite
>>>>> complex logic implemented to ensure the slave stays in sync with the
>>>>> containers that are running, and kills anything that goes rogue.
>>>>>
>>>>> It's going to be non-trivial for you to do that from the executor,
>>>>> though I guess you could make use of the docker events API or poll other
>>>>> endpoints in the API to check the status of your containers, and off the
>>>>> back of that send status updates to the cluster. Doing this however brings
>>>>> no guarantees that if your executor dies exceptionally (perhaps OOMd) the
>>>>> containers spawned will die... they'll keep running in the background and
>>>>> it'll be hard for you to know the state of your containers on the cluster.
>>>>>
>>>>> You probably want to be aware (if you don't know already) that the
>>>>> resource limits assigned to your tasks aren't going to be enforced by mesos
>>>>> because docker is running outside of its control. You'll need to pass the
>>>>> correct CPU/Memory limit parameters to your docker containers to ensure
>>>>> this happens correctly.
>>>>>
>>>>> Here are the docker API docs;
>>>>> https://docs.docker.com/reference/api/docker_remote_api_v1.15/
>>>>>
>>>>> Something you might want to consider, if all you're trying to do is
>>>>> allow your container access to details about itself (e.g `docker inspect`)
>>>>> is to open up the docker remote API to be queried by your containers on the
>>>>> slave, and switch to using the mesos docker containerizer.
>>>>>
>>>>> I hope that helps somewhat!
>>>>>
>>>>> Tom.
>>>>>
>>>>> --
>>>>>
>>>>> Tom Arnfeld
>>>>> Developer // DueDil
>>>>>
>>>>> (+44) 7525940046
>>>>> 25 Christopher Street, London, EC2A 2BS
>>>>>
>>>>>
>>>>> On Wed, Nov 19, 2014 at 10:16 PM, Janet Borschowa <
>>>>> janet.borschowa@codefutures.com> wrote:
>>>>>
>>>>>> Hi,
>>>>>> I'm implementing an executor which is used by the mesos slave to
>>>>>> launch tasks. The tasks are to launch a docker container - this is because
>>>>>> I need more info about the launched container than what the docker
>>>>>> containerizer returns.
>>>>>>
>>>>>> Is it OK to block in the executor's launchTask method until the task
>>>>>> completes? If not, how does the framework discover when that task
>>>>>> completes? I could spawn a process which notifies my executor when the task
>>>>>> completes and then have my executor send a status update. Or is there some
>>>>>> other recommended way to deal with this when the task could run for an
>>>>>> indefinite period of time before completing its work?
>>>>>>
>>>>>> Thanks!
>>>>>>
>>>>>> Janet
>>>>>>
>>>>>> --
>>>>>> Janet Borschowa
>>>>>> CodeFutures Corporation
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>
>
>
> --
> Thanks,
> Diptanu Choudhury
> Web - www.linkedin.com/in/diptanu
> Twitter - @diptanu <http://twitter.com/diptanu>
>
Re: Implementing an Executor
Posted by Diptanu Choudhury <di...@gmail.com>.
Hi Janet,
We implemented the same in our Titan Mesos Executor. We have an executor
manage multiple containers. We didn't want the overhead of running multiple
executors to manage multiple containers. We are using RxJava heavily in our
codebase, so whenever a launchTask callback is invoked in the MesosExecutor
we publish an event and return immediately from the callback so that our
executor can handle more callbacks from Mesos Slave.
But the event which is published is picked up by the subscriber and that
starts the workflow of downloading a container, mount volumes, sets up
networking, sets up logging, and finally starts a monitoring process for
the container. The monitoring process has the reference to the executor
driver and we send back Mesos status updates whenever the state of the
process changes.
On Thu, Nov 20, 2014 at 8:40 AM, Janet Borschowa <
janet.borschowa@codefutures.com> wrote:
> Hi David,
> Thanks for your suggestion. This approach sounds promising for what I need
> to do. I'm going to have to try this out.
>
> Thanks!
> Janet
>
> --
>
> Janet Borschowa
> CodeFutures Corporation
>
> On Thu, Nov 20, 2014 at 5:04 AM, David Greenberg <ds...@gmail.com>
> wrote:
>
>> One cool feature of the docker containerizer is that you can actually
>> launch your executor inside the docker image, so that you can just layer
>> the executor's custom logic on top of whatever container you desire. This
>> way, you can more easily control what's happening in the docker image, and
>> still leverage the containerizer.
>>
>> On Thursday, November 20, 2014, Tim Chen <ti...@mesosphere.io> wrote:
>>
>>> Hi Janet,
>>>
>>> Can you elaborate more what you like to get back from the docker
>>> container that you launched?
>>>
>>> Thanks,
>>>
>>> Tim
>>>
>>> On Wed, Nov 19, 2014 at 5:22 PM, Tom Arnfeld <to...@duedil.com> wrote:
>>>
>>>> Hi Janet,
>>>>
>>>> Oh sorry my mistake, I didn't read your email correctly, I thought you
>>>> were using the containerizer. What you're doing here is actually going to
>>>> be quite difficult to do, the mesos docker containerizer has some quite
>>>> complex logic implemented to ensure the slave stays in sync with the
>>>> containers that are running, and kills anything that goes rogue.
>>>>
>>>> It's going to be non-trivial for you to do that from the executor,
>>>> though I guess you could make use of the docker events API or poll other
>>>> endpoints in the API to check the status of your containers, and off the
>>>> back of that send status updates to the cluster. Doing this however brings
>>>> no guarantees that if your executor dies exceptionally (perhaps OOMd) the
>>>> containers spawned will die... they'll keep running in the background and
>>>> it'll be hard for you to know the state of your containers on the cluster.
>>>>
>>>> You probably want to be aware (if you don't know already) that the
>>>> resource limits assigned to your tasks aren't going to be enforced by mesos
>>>> because docker is running outside of its control. You'll need to pass the
>>>> correct CPU/Memory limit parameters to your docker containers to ensure
>>>> this happens correctly.
>>>>
>>>> Here are the docker API docs;
>>>> https://docs.docker.com/reference/api/docker_remote_api_v1.15/
>>>>
>>>> Something you might want to consider, if all you're trying to do is
>>>> allow your container access to details about itself (e.g `docker inspect`)
>>>> is to open up the docker remote API to be queried by your containers on the
>>>> slave, and switch to using the mesos docker containerizer.
>>>>
>>>> I hope that helps somewhat!
>>>>
>>>> Tom.
>>>>
>>>> --
>>>>
>>>> Tom Arnfeld
>>>> Developer // DueDil
>>>>
>>>> (+44) 7525940046
>>>> 25 Christopher Street, London, EC2A 2BS
>>>>
>>>>
>>>> On Wed, Nov 19, 2014 at 10:16 PM, Janet Borschowa <
>>>> janet.borschowa@codefutures.com> wrote:
>>>>
>>>>> Hi,
>>>>> I'm implementing an executor which is used by the mesos slave to
>>>>> launch tasks. The tasks are to launch a docker container - this is because
>>>>> I need more info about the launched container than what the docker
>>>>> containerizer returns.
>>>>>
>>>>> Is it OK to block in the executor's launchTask method until the task
>>>>> completes? If not, how does the framework discover when that task
>>>>> completes? I could spawn a process which notifies my executor when the task
>>>>> completes and then have my executor send a status update. Or is there some
>>>>> other recommended way to deal with this when the task could run for an
>>>>> indefinite period of time before completing its work?
>>>>>
>>>>> Thanks!
>>>>>
>>>>> Janet
>>>>>
>>>>> --
>>>>> Janet Borschowa
>>>>> CodeFutures Corporation
>>>>>
>>>>>
>>>>>
>>>>
>>>
>
--
Thanks,
Diptanu Choudhury
Web - www.linkedin.com/in/diptanu
Twitter - @diptanu <http://twitter.com/diptanu>
Re: Implementing an Executor
Posted by Janet Borschowa <ja...@codefutures.com>.
Hi David,
Thanks for your suggestion. This approach sounds promising for what I need
to do. I'm going to have to try this out.
Thanks!
Janet
--
Janet Borschowa
CodeFutures Corporation
On Thu, Nov 20, 2014 at 5:04 AM, David Greenberg <ds...@gmail.com>
wrote:
> One cool feature of the docker containerizer is that you can actually
> launch your executor inside the docker image, so that you can just layer
> the executor's custom logic on top of whatever container you desire. This
> way, you can more easily control what's happening in the docker image, and
> still leverage the containerizer.
>
> On Thursday, November 20, 2014, Tim Chen <ti...@mesosphere.io> wrote:
>
>> Hi Janet,
>>
>> Can you elaborate more what you like to get back from the docker
>> container that you launched?
>>
>> Thanks,
>>
>> Tim
>>
>> On Wed, Nov 19, 2014 at 5:22 PM, Tom Arnfeld <to...@duedil.com> wrote:
>>
>>> Hi Janet,
>>>
>>> Oh sorry my mistake, I didn't read your email correctly, I thought you
>>> were using the containerizer. What you're doing here is actually going to
>>> be quite difficult to do, the mesos docker containerizer has some quite
>>> complex logic implemented to ensure the slave stays in sync with the
>>> containers that are running, and kills anything that goes rogue.
>>>
>>> It's going to be non-trivial for you to do that from the executor,
>>> though I guess you could make use of the docker events API or poll other
>>> endpoints in the API to check the status of your containers, and off the
>>> back of that send status updates to the cluster. Doing this however brings
>>> no guarantees that if your executor dies exceptionally (perhaps OOMd) the
>>> containers spawned will die... they'll keep running in the background and
>>> it'll be hard for you to know the state of your containers on the cluster.
>>>
>>> You probably want to be aware (if you don't know already) that the
>>> resource limits assigned to your tasks aren't going to be enforced by mesos
>>> because docker is running outside of its control. You'll need to pass the
>>> correct CPU/Memory limit parameters to your docker containers to ensure
>>> this happens correctly.
>>>
>>> Here are the docker API docs;
>>> https://docs.docker.com/reference/api/docker_remote_api_v1.15/
>>>
>>> Something you might want to consider, if all you're trying to do is
>>> allow your container access to details about itself (e.g `docker inspect`)
>>> is to open up the docker remote API to be queried by your containers on the
>>> slave, and switch to using the mesos docker containerizer.
>>>
>>> I hope that helps somewhat!
>>>
>>> Tom.
>>>
>>> --
>>>
>>> Tom Arnfeld
>>> Developer // DueDil
>>>
>>> (+44) 7525940046
>>> 25 Christopher Street, London, EC2A 2BS
>>>
>>>
>>> On Wed, Nov 19, 2014 at 10:16 PM, Janet Borschowa <
>>> janet.borschowa@codefutures.com> wrote:
>>>
>>>> Hi,
>>>> I'm implementing an executor which is used by the mesos slave to launch
>>>> tasks. The tasks are to launch a docker container - this is because I need
>>>> more info about the launched container than what the docker containerizer
>>>> returns.
>>>>
>>>> Is it OK to block in the executor's launchTask method until the task
>>>> completes? If not, how does the framework discover when that task
>>>> completes? I could spawn a process which notifies my executor when the task
>>>> completes and then have my executor send a status update. Or is there some
>>>> other recommended way to deal with this when the task could run for an
>>>> indefinite period of time before completing its work?
>>>>
>>>> Thanks!
>>>>
>>>> Janet
>>>>
>>>> --
>>>> Janet Borschowa
>>>> CodeFutures Corporation
>>>>
>>>>
>>>>
>>>
>>
Re: Implementing an Executor
Posted by David Greenberg <ds...@gmail.com>.
One cool feature of the docker containerizer is that you can actually
launch your executor inside the docker image, so that you can just layer
the executor's custom logic on top of whatever container you desire. This
way, you can more easily control what's happening in the docker image, and
still leverage the containerizer.
On Thursday, November 20, 2014, Tim Chen <ti...@mesosphere.io> wrote:
> Hi Janet,
>
> Can you elaborate more what you like to get back from the docker container
> that you launched?
>
> Thanks,
>
> Tim
>
> On Wed, Nov 19, 2014 at 5:22 PM, Tom Arnfeld <tom@duedil.com
> <javascript:_e(%7B%7D,'cvml','tom@duedil.com');>> wrote:
>
>> Hi Janet,
>>
>> Oh sorry my mistake, I didn't read your email correctly, I thought you
>> were using the containerizer. What you're doing here is actually going to
>> be quite difficult to do, the mesos docker containerizer has some quite
>> complex logic implemented to ensure the slave stays in sync with the
>> containers that are running, and kills anything that goes rogue.
>>
>> It's going to be non-trivial for you to do that from the executor, though
>> I guess you could make use of the docker events API or poll other endpoints
>> in the API to check the status of your containers, and off the back of that
>> send status updates to the cluster. Doing this however brings no guarantees
>> that if your executor dies exceptionally (perhaps OOMd) the containers
>> spawned will die... they'll keep running in the background and it'll be
>> hard for you to know the state of your containers on the cluster.
>>
>> You probably want to be aware (if you don't know already) that the
>> resource limits assigned to your tasks aren't going to be enforced by mesos
>> because docker is running outside of its control. You'll need to pass the
>> correct CPU/Memory limit parameters to your docker containers to ensure
>> this happens correctly.
>>
>> Here are the docker API docs;
>> https://docs.docker.com/reference/api/docker_remote_api_v1.15/
>>
>> Something you might want to consider, if all you're trying to do is allow
>> your container access to details about itself (e.g `docker inspect`) is to
>> open up the docker remote API to be queried by your containers on the
>> slave, and switch to using the mesos docker containerizer.
>>
>> I hope that helps somewhat!
>>
>> Tom.
>>
>> --
>>
>> Tom Arnfeld
>> Developer // DueDil
>>
>> (+44) 7525940046
>> 25 Christopher Street, London, EC2A 2BS
>>
>>
>> On Wed, Nov 19, 2014 at 10:16 PM, Janet Borschowa <
>> janet.borschowa@codefutures.com
>> <javascript:_e(%7B%7D,'cvml','janet.borschowa@codefutures.com');>> wrote:
>>
>>> Hi,
>>> I'm implementing an executor which is used by the mesos slave to launch
>>> tasks. The tasks are to launch a docker container - this is because I need
>>> more info about the launched container than what the docker containerizer
>>> returns.
>>>
>>> Is it OK to block in the executor's launchTask method until the task
>>> completes? If not, how does the framework discover when that task
>>> completes? I could spawn a process which notifies my executor when the task
>>> completes and then have my executor send a status update. Or is there some
>>> other recommended way to deal with this when the task could run for an
>>> indefinite period of time before completing its work?
>>>
>>> Thanks!
>>>
>>> Janet
>>>
>>> --
>>> Janet Borschowa
>>> CodeFutures Corporation
>>>
>>>
>>>
>>
>
Re: Implementing an Executor
Posted by Tim Chen <ti...@mesosphere.io>.
Hi Janet,
Can you elaborate more what you like to get back from the docker container
that you launched?
Thanks,
Tim
On Wed, Nov 19, 2014 at 5:22 PM, Tom Arnfeld <to...@duedil.com> wrote:
> Hi Janet,
>
> Oh sorry my mistake, I didn't read your email correctly, I thought you
> were using the containerizer. What you're doing here is actually going to
> be quite difficult to do, the mesos docker containerizer has some quite
> complex logic implemented to ensure the slave stays in sync with the
> containers that are running, and kills anything that goes rogue.
>
> It's going to be non-trivial for you to do that from the executor, though
> I guess you could make use of the docker events API or poll other endpoints
> in the API to check the status of your containers, and off the back of that
> send status updates to the cluster. Doing this however brings no guarantees
> that if your executor dies exceptionally (perhaps OOMd) the containers
> spawned will die... they'll keep running in the background and it'll be
> hard for you to know the state of your containers on the cluster.
>
> You probably want to be aware (if you don't know already) that the
> resource limits assigned to your tasks aren't going to be enforced by mesos
> because docker is running outside of its control. You'll need to pass the
> correct CPU/Memory limit parameters to your docker containers to ensure
> this happens correctly.
>
> Here are the docker API docs;
> https://docs.docker.com/reference/api/docker_remote_api_v1.15/
>
> Something you might want to consider, if all you're trying to do is allow
> your container access to details about itself (e.g `docker inspect`) is to
> open up the docker remote API to be queried by your containers on the
> slave, and switch to using the mesos docker containerizer.
>
> I hope that helps somewhat!
>
> Tom.
>
> --
>
> Tom Arnfeld
> Developer // DueDil
>
> (+44) 7525940046
> 25 Christopher Street, London, EC2A 2BS
>
>
> On Wed, Nov 19, 2014 at 10:16 PM, Janet Borschowa <
> janet.borschowa@codefutures.com> wrote:
>
>> Hi,
>> I'm implementing an executor which is used by the mesos slave to launch
>> tasks. The tasks are to launch a docker container - this is because I need
>> more info about the launched container than what the docker containerizer
>> returns.
>>
>> Is it OK to block in the executor's launchTask method until the task
>> completes? If not, how does the framework discover when that task
>> completes? I could spawn a process which notifies my executor when the task
>> completes and then have my executor send a status update. Or is there some
>> other recommended way to deal with this when the task could run for an
>> indefinite period of time before completing its work?
>>
>> Thanks!
>>
>> Janet
>>
>> --
>> Janet Borschowa
>> CodeFutures Corporation
>>
>>
>>
>
Re: Implementing an Executor
Posted by Tom Arnfeld <to...@duedil.com>.
Hi Janet,
Oh sorry my mistake, I didn't read your email correctly, I thought you were using the containerizer. What you're doing here is actually going to be quite difficult to do, the mesos docker containerizer has some quite complex logic implemented to ensure the slave stays in sync with the containers that are running, and kills anything that goes rogue.
It's going to be non-trivial for you to do that from the executor, though I guess you could make use of the docker events API or poll other endpoints in the API to check the status of your containers, and off the back of that send status updates to the cluster. Doing this however brings no guarantees that if your executor dies exceptionally (perhaps OOMd) the containers spawned will die... they'll keep running in the background and it'll be hard for you to know the state of your containers on the cluster.
You probably want to be aware (if you don't know already) that the resource limits assigned to your tasks aren't going to be enforced by mesos because docker is running outside of its control. You'll need to pass the correct CPU/Memory limit parameters to your docker containers to ensure this happens correctly.
Here are the docker API docs; https://docs.docker.com/reference/api/docker_remote_api_v1.15/
Something you might want to consider, if all you're trying to do is allow your container access to details about itself (e.g `docker inspect`) is to open up the docker remote API to be queried by your containers on the slave, and switch to using the mesos docker containerizer.
I hope that helps somewhat!
Tom.
--
Tom Arnfeld
Developer // DueDil
(+44) 7525940046
25 Christopher Street, London, EC2A 2BS
On Wed, Nov 19, 2014 at 10:16 PM, Janet Borschowa
<ja...@codefutures.com> wrote:
> Hi,
> I'm implementing an executor which is used by the mesos slave to launch
> tasks. The tasks are to launch a docker container - this is because I need
> more info about the launched container than what the docker containerizer
> returns.
> Is it OK to block in the executor's launchTask method until the task
> completes? If not, how does the framework discover when that task
> completes? I could spawn a process which notifies my executor when the task
> completes and then have my executor send a status update. Or is there some
> other recommended way to deal with this when the task could run for an
> indefinite period of time before completing its work?
> Thanks!
> Janet
> --
> Janet Borschowa
> CodeFutures Corporation
Re: Implementing an Executor
Posted by Janet Borschowa <ja...@codefutures.com>.
Hi Tom,
Thanks for your reply. To be sure I understand correctly, my executor
launches my task. (Which is actually launching a docker container without
using the docker containerizer at all. I'm directly communicating with the
docker daemon in my executor.) When the task has been launched, I send a
status update TASK_RUNNING and the call to the executor returns so that it
isn't blocking. Now, I need some way of watching the status of this task
outside of mesos (in my case, the task is a docker container) and then,
when I see it is done I notify the framework with another status update,
e.g. TASK_FINISHED or TASK_FAILED.
I was hoping there was missing something. Even if I was not launching a
container but just some other task like a short running app, it seems
there's not a way to have Mesos to keep an eye on the tasks' status to tell
when/if it completes when using a custom Executor. Instead I need another
process to do that.
Thanks for your help!
Janet
--
On Wed, Nov 19, 2014 at 2:29 PM, Tom Arnfeld <to...@duedil.com> wrote:
> Hi Janet,
>
> Great to hear you're using Mesos! It's not the best idea to block in
> callbacks from the mesos drivers, either in the Executor or Framework, this
> is because you won't be notified correctly when other events happen (master
> failover, shutdown request, kill task request) as I believe the driver will
> guarantee you only get one callback at once and uses a blocking call into
> userland code.
>
> The slave and master become aware of task status by the executor correctly
> sending the TaskStatus message with TASK_STARTING, TASK_RUNNING, TASK_LOST,
> TASK_FAILED and TASK_FINISHED.
>
> Your executor is guaranteed to be alive for your task to be alive (unless
> there are any failures cases i'm not aware of) so you it's easier to
> monitor your task from the outside (if it's a subprocess, for example).
> Your use of the docker containerizer will also contribute to this behaviour
> as the mesos slave is going to kill the container ASAP after the executor
> disconnects.
>
> Sending task status updates should do the trick for you here.
>
> Tom.
>
> --
>
> Tom Arnfeld
> Developer // DueDil
>
> (+44) 7525940046
> 25 Christopher Street, London, EC2A 2BS
>
>
> On Wed, Nov 19, 2014 at 10:16 PM, Janet Borschowa <
> janet.borschowa@codefutures.com> wrote:
>
>> Hi,
>> I'm implementing an executor which is used by the mesos slave to launch
>> tasks. The tasks are to launch a docker container - this is because I need
>> more info about the launched container than what the docker containerizer
>> returns.
>>
>> Is it OK to block in the executor's launchTask method until the task
>> completes? If not, how does the framework discover when that task
>> completes? I could spawn a process which notifies my executor when the task
>> completes and then have my executor send a status update. Or is there some
>> other recommended way to deal with this when the task could run for an
>> indefinite period of time before completing its work?
>>
>> Thanks!
>>
>> Janet
>>
>> --
>> Janet Borschowa
>> CodeFutures Corporation
>>
>>
>>
>
Re: Implementing an Executor
Posted by Tom Arnfeld <to...@duedil.com>.
Hi Janet,
Great to hear you're using Mesos! It's not the best idea to block in callbacks from the mesos drivers, either in the Executor or Framework, this is because you won't be notified correctly when other events happen (master failover, shutdown request, kill task request) as I believe the driver will guarantee you only get one callback at once and uses a blocking call into userland code.
The slave and master become aware of task status by the executor correctly sending the TaskStatus message with TASK_STARTING, TASK_RUNNING, TASK_LOST, TASK_FAILED and TASK_FINISHED.
Your executor is guaranteed to be alive for your task to be alive (unless there are any failures cases i'm not aware of) so you it's easier to monitor your task from the outside (if it's a subprocess, for example). Your use of the docker containerizer will also contribute to this behaviour as the mesos slave is going to kill the container ASAP after the executor disconnects.
Sending task status updates should do the trick for you here.
Tom.
--
Tom Arnfeld
Developer // DueDil
(+44) 7525940046
25 Christopher Street, London, EC2A 2BS
On Wed, Nov 19, 2014 at 10:16 PM, Janet Borschowa
<ja...@codefutures.com> wrote:
> Hi,
> I'm implementing an executor which is used by the mesos slave to launch
> tasks. The tasks are to launch a docker container - this is because I need
> more info about the launched container than what the docker containerizer
> returns.
> Is it OK to block in the executor's launchTask method until the task
> completes? If not, how does the framework discover when that task
> completes? I could spawn a process which notifies my executor when the task
> completes and then have my executor send a status update. Or is there some
> other recommended way to deal with this when the task could run for an
> indefinite period of time before completing its work?
> Thanks!
> Janet
> --
> Janet Borschowa
> CodeFutures Corporation