You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@mesos.apache.org by Tom Fordon <to...@gmail.com> on 2015/04/01 16:27:47 UTC
Custom python executor with Docker
Hi. I'm trying to understand using docker within a custom executor. For
each of my tasks, I would like to perform some steps on the node before
launching a docker container. I was planning on writing a custom python
executor for this, but I wasn't sure how to launch docker from within this
executor.
Can I just call docker in a subprocess using the ContainerInfo from the
Task? If I do this, how does the Containerizer fit in?
Thank you,
Tom Fordon
Re: Custom python executor with Docker
Posted by James DeFelice <ja...@gmail.com>.
Hi Tom,
Docker cgroup parenting recently landed, so recovering from such a tricky
situation may no longer be as difficult if docker containers are parented
to the executor's cgroup at run-time.
On Tue, Apr 7, 2015 at 7:15 PM, Tom Arnfeld <to...@duedil.com> wrote:
> It's not possible to invoke the docker containerizer from outside of
> Mesos, as far as I know.
>
> If you persue this route, you can run into issues with orphaned containers
> as your executor may die for some unknown reason, and the container is
> still running. Recovering from this can be tricky business, so it's better
> if you can adapt your framework design to fit within the Mesos
> Task/Executor pattern.
>
> --
>
> Tom Arnfeld
> Developer // DueDil
>
> (+44) 7525940046
> 25 Christopher Street, London, EC2A 2BS
>
>
> On Mon, Apr 6, 2015 at 7:00 PM, Vinod Kone <vi...@apache.org> wrote:
>
>> Tim, do you want answer this?
>>
>> On Wed, Apr 1, 2015 at 7:27 AM, Tom Fordon <to...@gmail.com> wrote:
>>
>>> Hi. I'm trying to understand using docker within a custom executor. For
>>> each of my tasks, I would like to perform some steps on the node before
>>> launching a docker container. I was planning on writing a custom python
>>> executor for this, but I wasn't sure how to launch docker from within this
>>> executor.
>>>
>>> Can I just call docker in a subprocess using the ContainerInfo from the
>>> Task? If I do this, how does the Containerizer fit in?
>>> Thank you,
>>> Tom Fordon
>>>
>>
>>
>
--
James DeFelice
585.241.9488 (voice)
650.649.6071 (fax)
Re: Custom python executor with Docker
Posted by Weitao <zh...@gmail.com>.
Customized framework with Mesos persistent volumn can come to your desired feature, IMO. Which will recover the volumn after framework deregistered.
> 在 2015年11月10日，23:43，Aaron Carey <ac...@ilm.com> 写道：
>
> Yeah.. it'd be nice to do it natively though :)
>
> From: Plotka, Bartlomiej [bartlomiej.plotka@intel.com]
> Sent: 10 November 2015 15:38
> To: user@mesos.apache.org
> Subject: RE: Custom python executor with Docker
>
> This is somehow possible using Kubernetes over Mesos: https://github.com/kubernetes/kubernetes/blob/master/docs/getting-started-guides/mesos.md
>
> Kind Regards,
> Bartek Plotka
>
> From: Aaron Carey [mailto:acarey@ilm.com]
> Sent: Tuesday, November 10, 2015 4:33 PM
> To: user@mesos.apache.org
> Subject: RE: Custom python executor with Docker
>
> We would also be interested in some sort of standardised DockerExecutor which would allow us to add pre and post launch steps.
>
> Also having the ability to run two containers together as one task would be very useful (ie on the same host and linked together)
>
> From: Tom Fordon [tom.fordon@gmail.com]
> Sent: 12 August 2015 00:28
> To: user@mesos.apache.org
> Subject: Re: Custom python executor with Docker
>
> We ended up implementing a solution where we did the pre/post steps as separate mesos tasks and adding logic to our scheduler to ensure they were run on the same machine. If anybody knows of a standard / openly available DockerExecutor like what is described below, my team would be greatly interested.
>
> On Fri, Aug 7, 2015 at 4:01 AM, Kapil Malik <km...@adobe.com> wrote:
> Hi,
>
> We have a similar usecase while running multi-user workloads on mesos. Users provide docker images encapsulating application logic, which we (we = say some “Central API”) schedule on Chronos / Marathon. However, we need to run some standard pre / post steps for every docker submitted by users. We have following options –
>
> 1. Ask every user to embed their logic inside a pre-defined docker template which will perform pre/post steps.
> è This is error prone, makes us dependent on whether the users followed template, and not very popular with users either.
>
> 2. Extend every user docker (FROM <>) and find a way to add pre-post steps in our docker. Refer this docker when scheduling on chronos / marathon.
> è Building new dockers does not scale as users and applications grow
>
> 3. Write a custom executor which will perform the pre-post steps and manage the user docker lifetime.
> è Deals with user docker lifetime and is obviously complex.
>
> Is there a standard / openly available DockerExecutor which manages the docker lifetime and which I can extend to build my custom executor? This way I will be concerned only with my custom logic (pre/post steps) and still get benefits of a standard way to manage docker containers.
>
> Btw, thanks for the meaningful discussion below, it is very helpful.
>
> Thanks and regards,
>
> Kapil Malik | kmalik@adobe.com | 33430 / 8800836581
>
> From: James DeFelice [mailto:james.defelice@gmail.com]
> Sent: 09 April 2015 18:12
> To: user@mesos.apache.org
> Subject: Re: Custom python executor with Docker
>
> If you can run the pre/post steps in a container then I'd recommend building a Docker image that includes your pre/post step scripting + your algorithm and launching it using the built-in mesos Docker containerizer. It's much simpler than managing the lifetime of the Docker container yourself.
>
> On Thu, Apr 9, 2015 at 8:29 AM, Tom Fordon <to...@gmail.com> wrote:
> Thanks for all the responses, I really appreciate the help. Let me try to state my problem more clearly
>
> Our project is performing file-based data processing. I would like to keep the actual algorithm as contained as possible since we are in an R&D setting and will be getting untested code. We have some pre/post steps that need to be run on the same box as the actual algorithm: downloading/uploading files and database calls.
>
> We can run the pre/post steps and algorithm within the same container. The algorithm will be a little less contained, but it will work.
>
> Docker letting you specify a cgroup parent is really exciting. If I invoke a docker container with the executor as the cgroup-parent are there any other steps I need to perform? Would I need to do anything special to make mesos aware of the resource usage, or is that handled since the docker process would be in the executors cgroup?
>
> Thanks again,
> Tom
>
> On Tue, Apr 7, 2015 at 8:10 PM, Timothy Chen <tn...@gmail.com> wrote:
> Hi Tom(s),
>
> Tom Arnfeld is right, if you want to launch your own docker container
> in your custom executor you will have to handle all the issues
> yourself and not able to use the Docker containerizer at all.
>
> Alternatively, you can actually launch your custom executor in a
> Docker container by Mesos, by specifying the ContainerInfo in the
> ExecutorInfo.
> What this means is that your custom executor is already running in a
> docker container, and you can do your custom logic afterwards. This
> does means you can simply just launch multiple containers in the
> executor anymore.
>
> If there is something you want to do and doesnt' fit these let us know
> what you're trying to achieve and we can see what we can do.
>
> Tim
>
> On Tue, Apr 7, 2015 at 4:15 PM, Tom Arnfeld <to...@duedil.com> wrote:
> > It's not possible to invoke the docker containerizer from outside of Mesos,
> > as far as I know.
> >
> > If you persue this route, you can run into issues with orphaned containers
> > as your executor may die for some unknown reason, and the container is still
> > running. Recovering from this can be tricky business, so it's better if you
> > can adapt your framework design to fit within the Mesos Task/Executor
> > pattern.
> >
> > --
> >
> > Tom Arnfeld
> > Developer // DueDil
> >
> > (+44) 7525940046
> > 25 Christopher Street, London, EC2A 2BS
> >
> >
> > On Mon, Apr 6, 2015 at 7:00 PM, Vinod Kone <vi...@apache.org> wrote:
> >>
> >> Tim, do you want answer this?
> >>
> >> On Wed, Apr 1, 2015 at 7:27 AM, Tom Fordon <to...@gmail.com> wrote:
> >>>
> >>> Hi. I'm trying to understand using docker within a custom executor. For
> >>> each of my tasks, I would like to perform some steps on the node before
> >>> launching a docker container. I was planning on writing a custom python
> >>> executor for this, but I wasn't sure how to launch docker from within this
> >>> executor.
> >>>
> >>> Can I just call docker in a subprocess using the ContainerInfo from the
> >>> Task? If I do this, how does the Containerizer fit in?
> >>>
> >>> Thank you,
> >>> Tom Fordon
> >>
> >>
> >
>
>
>
>
> --
> James DeFelice
> 585.241.9488 (voice)
> 650.649.6071 (fax)
>
> ---------------------------------------------------------------------
> Intel Technology Poland sp. z o.o.
> ul. Słowackiego 173 | 80-298 Gdańsk | Sąd Rejonowy Gdańsk Północ | VII Wydział Gospodarczy Krajowego Rejestru Sądowego - KRS 101882 | NIP 957-07-52-316 | Kapitał zakładowy 200.000 PLN.
> Ta wiadomość wraz z załącznikami jest przeznaczona dla określonego adresata i może zawierać informacje poufne. W razie przypadkowego otrzymania tej wiadomości, prosimy o powiadomienie nadawcy oraz trwałe jej usunięcie; jakiekolwiek przeglądanie lub rozpowszechnianie jest zabronione.
> This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). If you are not the intended recipient, please contact the sender and delete all copies; any review or distribution by others is strictly prohibited.
RE: Custom python executor with Docker
Posted by Aaron Carey <ac...@ilm.com>.
Yeah.. it'd be nice to do it natively though :)
________________________________
From: Plotka, Bartlomiej [bartlomiej.plotka@intel.com]
Sent: 10 November 2015 15:38
To: user@mesos.apache.org
Subject: RE: Custom python executor with Docker
This is somehow possible using Kubernetes over Mesos: https://github.com/kubernetes/kubernetes/blob/master/docs/getting-started-guides/mesos.md
Kind Regards,
Bartek Plotka
From: Aaron Carey [mailto:acarey@ilm.com]
Sent: Tuesday, November 10, 2015 4:33 PM
To: user@mesos.apache.org
Subject: RE: Custom python executor with Docker
We would also be interested in some sort of standardised DockerExecutor which would allow us to add pre and post launch steps.
Also having the ability to run two containers together as one task would be very useful (ie on the same host and linked together)
________________________________
From: Tom Fordon [tom.fordon@gmail.com]
Sent: 12 August 2015 00:28
To: user@mesos.apache.org<ma...@mesos.apache.org>
Subject: Re: Custom python executor with Docker
We ended up implementing a solution where we did the pre/post steps as separate mesos tasks and adding logic to our scheduler to ensure they were run on the same machine. If anybody knows of a standard / openly available DockerExecutor like what is described below, my team would be greatly interested.
On Fri, Aug 7, 2015 at 4:01 AM, Kapil Malik <km...@adobe.com>> wrote:
Hi,
We have a similar usecase while running multi-user workloads on mesos. Users provide docker images encapsulating application logic, which we (we = say some “Central API”) schedule on Chronos / Marathon. However, we need to run some standard pre / post steps for every docker submitted by users. We have following options –
1. Ask every user to embed their logic inside a pre-defined docker template which will perform pre/post steps.
==> This is error prone, makes us dependent on whether the users followed template, and not very popular with users either.
2. Extend every user docker (FROM <>) and find a way to add pre-post steps in our docker. Refer this docker when scheduling on chronos / marathon.
==> Building new dockers does not scale as users and applications grow
3. Write a custom executor which will perform the pre-post steps and manage the user docker lifetime.
==> Deals with user docker lifetime and is obviously complex.
Is there a standard / openly available DockerExecutor which manages the docker lifetime and which I can extend to build my custom executor? This way I will be concerned only with my custom logic (pre/post steps) and still get benefits of a standard way to manage docker containers.
Btw, thanks for the meaningful discussion below, it is very helpful.
Thanks and regards,
Kapil Malik | kmalik@adobe.com<ma...@adobe.com> | 33430 / 8800836581
From: James DeFelice [mailto:james.defelice@gmail.com<ma...@gmail.com>]
Sent: 09 April 2015 18:12
To: user@mesos.apache.org<ma...@mesos.apache.org>
Subject: Re: Custom python executor with Docker
If you can run the pre/post steps in a container then I'd recommend building a Docker image that includes your pre/post step scripting + your algorithm and launching it using the built-in mesos Docker containerizer. It's much simpler than managing the lifetime of the Docker container yourself.
On Thu, Apr 9, 2015 at 8:29 AM, Tom Fordon <to...@gmail.com>> wrote:
Thanks for all the responses, I really appreciate the help. Let me try to state my problem more clearly
Our project is performing file-based data processing. I would like to keep the actual algorithm as contained as possible since we are in an R&D setting and will be getting untested code. We have some pre/post steps that need to be run on the same box as the actual algorithm: downloading/uploading files and database calls.
We can run the pre/post steps and algorithm within the same container. The algorithm will be a little less contained, but it will work.
Docker letting you specify a cgroup parent is really exciting. If I invoke a docker container with the executor as the cgroup-parent are there any other steps I need to perform? Would I need to do anything special to make mesos aware of the resource usage, or is that handled since the docker process would be in the executors cgroup?
Thanks again,
Tom
On Tue, Apr 7, 2015 at 8:10 PM, Timothy Chen <tn...@gmail.com>> wrote:
Hi Tom(s),
Tom Arnfeld is right, if you want to launch your own docker container
in your custom executor you will have to handle all the issues
yourself and not able to use the Docker containerizer at all.
Alternatively, you can actually launch your custom executor in a
Docker container by Mesos, by specifying the ContainerInfo in the
ExecutorInfo.
What this means is that your custom executor is already running in a
docker container, and you can do your custom logic afterwards. This
does means you can simply just launch multiple containers in the
executor anymore.
If there is something you want to do and doesnt' fit these let us know
what you're trying to achieve and we can see what we can do.
Tim
On Tue, Apr 7, 2015 at 4:15 PM, Tom Arnfeld <to...@duedil.com>> wrote:
> It's not possible to invoke the docker containerizer from outside of Mesos,
> as far as I know.
>
> If you persue this route, you can run into issues with orphaned containers
> as your executor may die for some unknown reason, and the container is still
> running. Recovering from this can be tricky business, so it's better if you
> can adapt your framework design to fit within the Mesos Task/Executor
> pattern.
>
> --
>
> Tom Arnfeld
> Developer // DueDil
>
> (+44) 7525940046<tel:%28%2B44%29%207525940046>
> 25 Christopher Street, London, EC2A 2BS
>
>
> On Mon, Apr 6, 2015 at 7:00 PM, Vinod Kone <vi...@apache.org>> wrote:
>>
>> Tim, do you want answer this?
>>
>> On Wed, Apr 1, 2015 at 7:27 AM, Tom Fordon <to...@gmail.com>> wrote:
>>>
>>> Hi. I'm trying to understand using docker within a custom executor. For
>>> each of my tasks, I would like to perform some steps on the node before
>>> launching a docker container. I was planning on writing a custom python
>>> executor for this, but I wasn't sure how to launch docker from within this
>>> executor.
>>>
>>> Can I just call docker in a subprocess using the ContainerInfo from the
>>> Task? If I do this, how does the Containerizer fit in?
>>>
>>> Thank you,
>>> Tom Fordon
>>
>>
>
--
James DeFelice
585.241.9488<tel:585.241.9488> (voice)
650.649.6071<tel:650.649.6071> (fax)
---------------------------------------------------------------------
Intel Technology Poland sp. z o.o.
ul. Słowackiego 173 | 80-298 Gdańsk | Sąd Rejonowy Gdańsk Północ | VII Wydział Gospodarczy Krajowego Rejestru Sądowego - KRS 101882 | NIP 957-07-52-316 | Kapitał zakładowy 200.000 PLN.
Ta wiadomość wraz z załącznikami jest przeznaczona dla określonego adresata i może zawierać informacje poufne. W razie przypadkowego otrzymania tej wiadomości, prosimy o powiadomienie nadawcy oraz trwałe jej usunięcie; jakiekolwiek przeglądanie lub rozpowszechnianie jest zabronione.
This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). If you are not the intended recipient, please contact the sender and delete all copies; any review or distribution by others is strictly prohibited.
RE: Custom python executor with Docker
Posted by "Plotka, Bartlomiej" <ba...@intel.com>.
This is somehow possible using Kubernetes over Mesos: https://github.com/kubernetes/kubernetes/blob/master/docs/getting-started-guides/mesos.md
Kind Regards,
Bartek Plotka
From: Aaron Carey [mailto:acarey@ilm.com]
Sent: Tuesday, November 10, 2015 4:33 PM
To: user@mesos.apache.org
Subject: RE: Custom python executor with Docker
We would also be interested in some sort of standardised DockerExecutor which would allow us to add pre and post launch steps.
Also having the ability to run two containers together as one task would be very useful (ie on the same host and linked together)
________________________________
From: Tom Fordon [tom.fordon@gmail.com]
Sent: 12 August 2015 00:28
To: user@mesos.apache.org<ma...@mesos.apache.org>
Subject: Re: Custom python executor with Docker
We ended up implementing a solution where we did the pre/post steps as separate mesos tasks and adding logic to our scheduler to ensure they were run on the same machine. If anybody knows of a standard / openly available DockerExecutor like what is described below, my team would be greatly interested.
On Fri, Aug 7, 2015 at 4:01 AM, Kapil Malik <km...@adobe.com>> wrote:
Hi,
We have a similar usecase while running multi-user workloads on mesos. Users provide docker images encapsulating application logic, which we (we = say some "Central API") schedule on Chronos / Marathon. However, we need to run some standard pre / post steps for every docker submitted by users. We have following options -
1. Ask every user to embed their logic inside a pre-defined docker template which will perform pre/post steps.
==> This is error prone, makes us dependent on whether the users followed template, and not very popular with users either.
2. Extend every user docker (FROM <>) and find a way to add pre-post steps in our docker. Refer this docker when scheduling on chronos / marathon.
==> Building new dockers does not scale as users and applications grow
3. Write a custom executor which will perform the pre-post steps and manage the user docker lifetime.
==> Deals with user docker lifetime and is obviously complex.
Is there a standard / openly available DockerExecutor which manages the docker lifetime and which I can extend to build my custom executor? This way I will be concerned only with my custom logic (pre/post steps) and still get benefits of a standard way to manage docker containers.
Btw, thanks for the meaningful discussion below, it is very helpful.
Thanks and regards,
Kapil Malik | kmalik@adobe.com<ma...@adobe.com> | 33430 / 8800836581
From: James DeFelice [mailto:james.defelice@gmail.com<ma...@gmail.com>]
Sent: 09 April 2015 18:12
To: user@mesos.apache.org<ma...@mesos.apache.org>
Subject: Re: Custom python executor with Docker
If you can run the pre/post steps in a container then I'd recommend building a Docker image that includes your pre/post step scripting + your algorithm and launching it using the built-in mesos Docker containerizer. It's much simpler than managing the lifetime of the Docker container yourself.
On Thu, Apr 9, 2015 at 8:29 AM, Tom Fordon <to...@gmail.com>> wrote:
Thanks for all the responses, I really appreciate the help. Let me try to state my problem more clearly
Our project is performing file-based data processing. I would like to keep the actual algorithm as contained as possible since we are in an R&D setting and will be getting untested code. We have some pre/post steps that need to be run on the same box as the actual algorithm: downloading/uploading files and database calls.
We can run the pre/post steps and algorithm within the same container. The algorithm will be a little less contained, but it will work.
Docker letting you specify a cgroup parent is really exciting. If I invoke a docker container with the executor as the cgroup-parent are there any other steps I need to perform? Would I need to do anything special to make mesos aware of the resource usage, or is that handled since the docker process would be in the executors cgroup?
Thanks again,
Tom
On Tue, Apr 7, 2015 at 8:10 PM, Timothy Chen <tn...@gmail.com>> wrote:
Hi Tom(s),
Tom Arnfeld is right, if you want to launch your own docker container
in your custom executor you will have to handle all the issues
yourself and not able to use the Docker containerizer at all.
Alternatively, you can actually launch your custom executor in a
Docker container by Mesos, by specifying the ContainerInfo in the
ExecutorInfo.
What this means is that your custom executor is already running in a
docker container, and you can do your custom logic afterwards. This
does means you can simply just launch multiple containers in the
executor anymore.
If there is something you want to do and doesnt' fit these let us know
what you're trying to achieve and we can see what we can do.
Tim
On Tue, Apr 7, 2015 at 4:15 PM, Tom Arnfeld <to...@duedil.com>> wrote:
> It's not possible to invoke the docker containerizer from outside of Mesos,
> as far as I know.
>
> If you persue this route, you can run into issues with orphaned containers
> as your executor may die for some unknown reason, and the container is still
> running. Recovering from this can be tricky business, so it's better if you
> can adapt your framework design to fit within the Mesos Task/Executor
> pattern.
>
> --
>
> Tom Arnfeld
> Developer // DueDil
>
> (+44) 7525940046<tel:%28%2B44%29%207525940046>
> 25 Christopher Street, London, EC2A 2BS
>
>
> On Mon, Apr 6, 2015 at 7:00 PM, Vinod Kone <vi...@apache.org>> wrote:
>>
>> Tim, do you want answer this?
>>
>> On Wed, Apr 1, 2015 at 7:27 AM, Tom Fordon <to...@gmail.com>> wrote:
>>>
>>> Hi. I'm trying to understand using docker within a custom executor. For
>>> each of my tasks, I would like to perform some steps on the node before
>>> launching a docker container. I was planning on writing a custom python
>>> executor for this, but I wasn't sure how to launch docker from within this
>>> executor.
>>>
>>> Can I just call docker in a subprocess using the ContainerInfo from the
>>> Task? If I do this, how does the Containerizer fit in?
>>>
>>> Thank you,
>>> Tom Fordon
>>
>>
>
--
James DeFelice
585.241.9488<tel:585.241.9488> (voice)
650.649.6071<tel:650.649.6071> (fax)
--------------------------------------------------------------------
Intel Technology Poland sp. z o.o.
ul. Slowackiego 173 | 80-298 Gdansk | Sad Rejonowy Gdansk Polnoc | VII Wydzial Gospodarczy Krajowego Rejestru Sadowego - KRS 101882 | NIP 957-07-52-316 | Kapital zakladowy 200.000 PLN.
Ta wiadomosc wraz z zalacznikami jest przeznaczona dla okreslonego adresata i moze zawierac informacje poufne. W razie przypadkowego otrzymania tej wiadomosci, prosimy o powiadomienie nadawcy oraz trwale jej usuniecie; jakiekolwiek
przegladanie lub rozpowszechnianie jest zabronione.
This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). If you are not the intended recipient, please contact the sender and delete all copies; any review or distribution by
others is strictly prohibited.
RE: Custom python executor with Docker
Posted by Aaron Carey <ac...@ilm.com>.
We would also be interested in some sort of standardised DockerExecutor which would allow us to add pre and post launch steps.
Also having the ability to run two containers together as one task would be very useful (ie on the same host and linked together)
________________________________
From: Tom Fordon [tom.fordon@gmail.com]
Sent: 12 August 2015 00:28
To: user@mesos.apache.org
Subject: Re: Custom python executor with Docker
We ended up implementing a solution where we did the pre/post steps as separate mesos tasks and adding logic to our scheduler to ensure they were run on the same machine. If anybody knows of a standard / openly available DockerExecutor like what is described below, my team would be greatly interested.
On Fri, Aug 7, 2015 at 4:01 AM, Kapil Malik <km...@adobe.com>> wrote:
Hi,
We have a similar usecase while running multi-user workloads on mesos. Users provide docker images encapsulating application logic, which we (we = say some “Central API”) schedule on Chronos / Marathon. However, we need to run some standard pre / post steps for every docker submitted by users. We have following options –
1. Ask every user to embed their logic inside a pre-defined docker template which will perform pre/post steps.
==> This is error prone, makes us dependent on whether the users followed template, and not very popular with users either.
2. Extend every user docker (FROM <>) and find a way to add pre-post steps in our docker. Refer this docker when scheduling on chronos / marathon.
==> Building new dockers does not scale as users and applications grow
3. Write a custom executor which will perform the pre-post steps and manage the user docker lifetime.
==> Deals with user docker lifetime and is obviously complex.
Is there a standard / openly available DockerExecutor which manages the docker lifetime and which I can extend to build my custom executor? This way I will be concerned only with my custom logic (pre/post steps) and still get benefits of a standard way to manage docker containers.
Btw, thanks for the meaningful discussion below, it is very helpful.
Thanks and regards,
Kapil Malik | kmalik@adobe.com<ma...@adobe.com> | 33430 / 8800836581
From: James DeFelice [mailto:james.defelice@gmail.com<ma...@gmail.com>]
Sent: 09 April 2015 18:12
To: user@mesos.apache.org<ma...@mesos.apache.org>
Subject: Re: Custom python executor with Docker
If you can run the pre/post steps in a container then I'd recommend building a Docker image that includes your pre/post step scripting + your algorithm and launching it using the built-in mesos Docker containerizer. It's much simpler than managing the lifetime of the Docker container yourself.
On Thu, Apr 9, 2015 at 8:29 AM, Tom Fordon <to...@gmail.com>> wrote:
Thanks for all the responses, I really appreciate the help. Let me try to state my problem more clearly
Our project is performing file-based data processing. I would like to keep the actual algorithm as contained as possible since we are in an R&D setting and will be getting untested code. We have some pre/post steps that need to be run on the same box as the actual algorithm: downloading/uploading files and database calls.
We can run the pre/post steps and algorithm within the same container. The algorithm will be a little less contained, but it will work.
Docker letting you specify a cgroup parent is really exciting. If I invoke a docker container with the executor as the cgroup-parent are there any other steps I need to perform? Would I need to do anything special to make mesos aware of the resource usage, or is that handled since the docker process would be in the executors cgroup?
Thanks again,
Tom
On Tue, Apr 7, 2015 at 8:10 PM, Timothy Chen <tn...@gmail.com>> wrote:
Hi Tom(s),
Tom Arnfeld is right, if you want to launch your own docker container
in your custom executor you will have to handle all the issues
yourself and not able to use the Docker containerizer at all.
Alternatively, you can actually launch your custom executor in a
Docker container by Mesos, by specifying the ContainerInfo in the
ExecutorInfo.
What this means is that your custom executor is already running in a
docker container, and you can do your custom logic afterwards. This
does means you can simply just launch multiple containers in the
executor anymore.
If there is something you want to do and doesnt' fit these let us know
what you're trying to achieve and we can see what we can do.
Tim
On Tue, Apr 7, 2015 at 4:15 PM, Tom Arnfeld <to...@duedil.com>> wrote:
> It's not possible to invoke the docker containerizer from outside of Mesos,
> as far as I know.
>
> If you persue this route, you can run into issues with orphaned containers
> as your executor may die for some unknown reason, and the container is still
> running. Recovering from this can be tricky business, so it's better if you
> can adapt your framework design to fit within the Mesos Task/Executor
> pattern.
>
> --
>
> Tom Arnfeld
> Developer // DueDil
>
> (+44) 7525940046<tel:%28%2B44%29%207525940046>
> 25 Christopher Street, London, EC2A 2BS
>
>
> On Mon, Apr 6, 2015 at 7:00 PM, Vinod Kone <vi...@apache.org>> wrote:
>>
>> Tim, do you want answer this?
>>
>> On Wed, Apr 1, 2015 at 7:27 AM, Tom Fordon <to...@gmail.com>> wrote:
>>>
>>> Hi. I'm trying to understand using docker within a custom executor. For
>>> each of my tasks, I would like to perform some steps on the node before
>>> launching a docker container. I was planning on writing a custom python
>>> executor for this, but I wasn't sure how to launch docker from within this
>>> executor.
>>>
>>> Can I just call docker in a subprocess using the ContainerInfo from the
>>> Task? If I do this, how does the Containerizer fit in?
>>>
>>> Thank you,
>>> Tom Fordon
>>
>>
>
--
James DeFelice
585.241.9488<tel:585.241.9488> (voice)
650.649.6071<tel:650.649.6071> (fax)
RE: Custom python executor with Docker
Posted by "Plotka, Bartlomiej" <ba...@intel.com>.
Hi All,
I can see there are many use cases for extending Docker tasks with some pre/post work.
In our solution we want to prepare and attach some volumes (e.g ceph volumes) to Docker. As you know we cannot (easily) attach a volume to the running Docker, so we need pre script for preparing & attaching in one task. Unfortunately, we also need to clean-up these volumes and it seems there is not any “slavePostLaunchDockerHook” available now in Mesos.
We’ve developed short script for running such tasks using common Mesos Containerizer and without any hook module in Mesos. We can just put it in the Marathon “command” parameter. Do you think such created Docker container will have limited cpu shares & memory (using Mesos resources)?
Here is pseudo-bash code:
#!/bin/bash
function cleanup {{
docker rm <docker_name>;
<clean up volumes>
}}
trap cleanup SIGKILL SIGSTOP SIGHUP SIGINT SIGTERM;
<prepare volumes>
docker run --name <docker-name> -p <port>:22 <volumes> <docker image> bash -c "<docker cmd>";
We are aware that there wouldn’t be any sandbox support and any other Docker Containerizer features. However, our main concern is about cgroups… We are not sure if our newly created Docker (via this script) will inherit our task’s cgroups and as a result will limit memory, cpus via default Mesos Task’s resource parameters.
Anyone knows the answer?
As far as I understand without special configuration ‘Docker run’ somehow pass the ‘run’ command to the docker daemon so we don’t have control of container cgroup…
Kind Regards,
Bartek Plotka (bplotka)
From: Kevin Sweeney [mailto:kevints@apache.org]
Sent: Wednesday, August 12, 2015 1:52 AM
To: user@mesos.apache.org
Subject: Re: Custom python executor with Docker
Apache Aurora [1] uses a custom Python executor and supports Docker via the containerizer. There's just one problem - the container has to have a Python2.7 runtime inside that can run the executor PEX file [2]. So if you're okay with that restriction you're in business (and you can use the Aurora configuration DSL to describe setup/teardown steps).
[1] https://aurora.apache.org
[2] https://pex.readthedocs.org/en/latest/
On Tue, Aug 11, 2015 at 4:42 PM, Tim Chen <ti...@mesosphere.io>> wrote:
So currently there is a review out for pre-hooks (https://reviews.apache.org/r/36185/) before a docker container launches.
We can also add a post hook, but like to see if the specified hook satifies what you guys are looking for.
Tim
On Tue, Aug 11, 2015 at 4:28 PM, Tom Fordon <to...@gmail.com>> wrote:
We ended up implementing a solution where we did the pre/post steps as separate mesos tasks and adding logic to our scheduler to ensure they were run on the same machine. If anybody knows of a standard / openly available DockerExecutor like what is described below, my team would be greatly interested.
On Fri, Aug 7, 2015 at 4:01 AM, Kapil Malik <km...@adobe.com>> wrote:
Hi,
We have a similar usecase while running multi-user workloads on mesos. Users provide docker images encapsulating application logic, which we (we = say some “Central API”) schedule on Chronos / Marathon. However, we need to run some standard pre / post steps for every docker submitted by users. We have following options –
1. Ask every user to embed their logic inside a pre-defined docker template which will perform pre/post steps.
==> This is error prone, makes us dependent on whether the users followed template, and not very popular with users either.
2. Extend every user docker (FROM <>) and find a way to add pre-post steps in our docker. Refer this docker when scheduling on chronos / marathon.
==> Building new dockers does not scale as users and applications grow
3. Write a custom executor which will perform the pre-post steps and manage the user docker lifetime.
==> Deals with user docker lifetime and is obviously complex.
Is there a standard / openly available DockerExecutor which manages the docker lifetime and which I can extend to build my custom executor? This way I will be concerned only with my custom logic (pre/post steps) and still get benefits of a standard way to manage docker containers.
Btw, thanks for the meaningful discussion below, it is very helpful.
Thanks and regards,
Kapil Malik | kmalik@adobe.com<ma...@adobe.com> | 33430 / 8800836581
From: James DeFelice [mailto:james.defelice@gmail.com<ma...@gmail.com>]
Sent: 09 April 2015 18:12
To: user@mesos.apache.org<ma...@mesos.apache.org>
Subject: Re: Custom python executor with Docker
If you can run the pre/post steps in a container then I'd recommend building a Docker image that includes your pre/post step scripting + your algorithm and launching it using the built-in mesos Docker containerizer. It's much simpler than managing the lifetime of the Docker container yourself.
On Thu, Apr 9, 2015 at 8:29 AM, Tom Fordon <to...@gmail.com>> wrote:
Thanks for all the responses, I really appreciate the help. Let me try to state my problem more clearly
Our project is performing file-based data processing. I would like to keep the actual algorithm as contained as possible since we are in an R&D setting and will be getting untested code. We have some pre/post steps that need to be run on the same box as the actual algorithm: downloading/uploading files and database calls.
We can run the pre/post steps and algorithm within the same container. The algorithm will be a little less contained, but it will work.
Docker letting you specify a cgroup parent is really exciting. If I invoke a docker container with the executor as the cgroup-parent are there any other steps I need to perform? Would I need to do anything special to make mesos aware of the resource usage, or is that handled since the docker process would be in the executors cgroup?
Thanks again,
Tom
On Tue, Apr 7, 2015 at 8:10 PM, Timothy Chen <tn...@gmail.com>> wrote:
Hi Tom(s),
Tom Arnfeld is right, if you want to launch your own docker container
in your custom executor you will have to handle all the issues
yourself and not able to use the Docker containerizer at all.
Alternatively, you can actually launch your custom executor in a
Docker container by Mesos, by specifying the ContainerInfo in the
ExecutorInfo.
What this means is that your custom executor is already running in a
docker container, and you can do your custom logic afterwards. This
does means you can simply just launch multiple containers in the
executor anymore.
If there is something you want to do and doesnt' fit these let us know
what you're trying to achieve and we can see what we can do.
Tim
On Tue, Apr 7, 2015 at 4:15 PM, Tom Arnfeld <to...@duedil.com>> wrote:
> It's not possible to invoke the docker containerizer from outside of Mesos,
> as far as I know.
>
> If you persue this route, you can run into issues with orphaned containers
> as your executor may die for some unknown reason, and the container is still
> running. Recovering from this can be tricky business, so it's better if you
> can adapt your framework design to fit within the Mesos Task/Executor
> pattern.
>
> --
>
> Tom Arnfeld
> Developer // DueDil
>
> (+44) 7525940046<tel:%28%2B44%29%207525940046>
> 25 Christopher Street, London, EC2A 2BS
>
>
> On Mon, Apr 6, 2015 at 7:00 PM, Vinod Kone <vi...@apache.org>> wrote:
>>
>> Tim, do you want answer this?
>>
>> On Wed, Apr 1, 2015 at 7:27 AM, Tom Fordon <to...@gmail.com>> wrote:
>>>
>>> Hi. I'm trying to understand using docker within a custom executor. For
>>> each of my tasks, I would like to perform some steps on the node before
>>> launching a docker container. I was planning on writing a custom python
>>> executor for this, but I wasn't sure how to launch docker from within this
>>> executor.
>>>
>>> Can I just call docker in a subprocess using the ContainerInfo from the
>>> Task? If I do this, how does the Containerizer fit in?
>>>
>>> Thank you,
>>> Tom Fordon
>>
>>
>
--
James DeFelice
585.241.9488<tel:585.241.9488> (voice)
650.649.6071<tel:650.649.6071> (fax)
--------------------------------------------------------------------
Intel Technology Poland sp. z o.o.
ul. Slowackiego 173 | 80-298 Gdansk | Sad Rejonowy Gdansk Polnoc | VII Wydzial Gospodarczy Krajowego Rejestru Sadowego - KRS 101882 | NIP 957-07-52-316 | Kapital zakladowy 200.000 PLN.
Ta wiadomosc wraz z zalacznikami jest przeznaczona dla okreslonego adresata i moze zawierac informacje poufne. W razie przypadkowego otrzymania tej wiadomosci, prosimy o powiadomienie nadawcy oraz trwale jej usuniecie; jakiekolwiek
przegladanie lub rozpowszechnianie jest zabronione.
This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). If you are not the intended recipient, please contact the sender and delete all copies; any review or distribution by
others is strictly prohibited.
Re: Custom python executor with Docker
Posted by Kevin Sweeney <ke...@apache.org>.
Apache Aurora [1] uses a custom Python executor and supports Docker via the
containerizer. There's just one problem - the container has to have a
Python2.7 runtime inside that can run the executor PEX file [2]. So if
you're okay with that restriction you're in business (and you can use the
Aurora configuration DSL to describe setup/teardown steps).
[1] https://aurora.apache.org
[2] https://pex.readthedocs.org/en/latest/
On Tue, Aug 11, 2015 at 4:42 PM, Tim Chen <ti...@mesosphere.io> wrote:
> So currently there is a review out for pre-hooks (
> https://reviews.apache.org/r/36185/) before a docker container launches.
>
> We can also add a post hook, but like to see if the specified hook
> satifies what you guys are looking for.
>
> Tim
>
> On Tue, Aug 11, 2015 at 4:28 PM, Tom Fordon <to...@gmail.com> wrote:
>
>> We ended up implementing a solution where we did the pre/post steps as
>> separate mesos tasks and adding logic to our scheduler to ensure they were
>> run on the same machine. If anybody knows of a standard / openly available
>> DockerExecutor like what is described below, my team would be greatly
>> interested.
>>
>>
>> On Fri, Aug 7, 2015 at 4:01 AM, Kapil Malik <km...@adobe.com> wrote:
>>
>>> Hi,
>>>
>>>
>>>
>>> We have a similar usecase while running multi-user workloads on mesos.
>>> Users provide docker images encapsulating application logic, which we (we =
>>> say some “Central API”) schedule on Chronos / Marathon. However, we need to
>>> run some standard pre / post steps for every docker submitted by users. We
>>> have following options –
>>>
>>>
>>>
>>> 1. Ask every user to embed their logic inside a pre-defined
>>> docker template which will perform pre/post steps.
>>>
>>> è This is error prone, makes us dependent on whether the users followed
>>> template, and not very popular with users either.
>>>
>>>
>>>
>>> 2. Extend every user docker (FROM <>) and find a way to add
>>> pre-post steps in our docker. Refer this docker when scheduling on chronos
>>> / marathon.
>>>
>>> è Building new dockers does not scale as users and applications grow
>>>
>>>
>>>
>>> 3. Write a custom executor which will perform the pre-post steps
>>> and manage the user docker lifetime.
>>>
>>> è Deals with user docker lifetime and is obviously complex.
>>>
>>>
>>>
>>> Is there a standard / openly available DockerExecutor which manages the
>>> docker lifetime and which I can extend to build my custom executor? This
>>> way I will be concerned only with my custom logic (pre/post steps) and
>>> still get benefits of a standard way to manage docker containers.
>>>
>>>
>>>
>>> Btw, thanks for the meaningful discussion below, it is very helpful.
>>>
>>>
>>>
>>> Thanks and regards,
>>>
>>>
>>>
>>> Kapil Malik | kmalik@adobe.com | 33430 / 8800836581
>>>
>>>
>>>
>>> *From:* James DeFelice [mailto:james.defelice@gmail.com]
>>> *Sent:* 09 April 2015 18:12
>>> *To:* user@mesos.apache.org
>>> *Subject:* Re: Custom python executor with Docker
>>>
>>>
>>>
>>> If you can run the pre/post steps in a container then I'd recommend
>>> building a Docker image that includes your pre/post step scripting + your
>>> algorithm and launching it using the built-in mesos Docker containerizer.
>>> It's much simpler than managing the lifetime of the Docker container
>>> yourself.
>>>
>>>
>>>
>>> On Thu, Apr 9, 2015 at 8:29 AM, Tom Fordon <to...@gmail.com> wrote:
>>>
>>> Thanks for all the responses, I really appreciate the help. Let me try
>>> to state my problem more clearly
>>>
>>>
>>>
>>> Our project is performing file-based data processing. I would like to
>>> keep the actual algorithm as contained as possible since we are in an R&D
>>> setting and will be getting untested code. We have some pre/post steps
>>> that need to be run on the same box as the actual algorithm:
>>> downloading/uploading files and database calls.
>>>
>>>
>>>
>>> We can run the pre/post steps and algorithm within the same container.
>>> The algorithm will be a little less contained, but it will work.
>>>
>>>
>>>
>>> Docker letting you specify a cgroup parent is really exciting. If I
>>> invoke a docker container with the executor as the cgroup-parent are there
>>> any other steps I need to perform? Would I need to do anything special to
>>> make mesos aware of the resource usage, or is that handled since the docker
>>> process would be in the executors cgroup?
>>>
>>>
>>>
>>> Thanks again,
>>>
>>> Tom
>>>
>>>
>>>
>>> On Tue, Apr 7, 2015 at 8:10 PM, Timothy Chen <tn...@gmail.com> wrote:
>>>
>>> Hi Tom(s),
>>>
>>> Tom Arnfeld is right, if you want to launch your own docker container
>>> in your custom executor you will have to handle all the issues
>>> yourself and not able to use the Docker containerizer at all.
>>>
>>> Alternatively, you can actually launch your custom executor in a
>>> Docker container by Mesos, by specifying the ContainerInfo in the
>>> ExecutorInfo.
>>> What this means is that your custom executor is already running in a
>>> docker container, and you can do your custom logic afterwards. This
>>> does means you can simply just launch multiple containers in the
>>> executor anymore.
>>>
>>> If there is something you want to do and doesnt' fit these let us know
>>> what you're trying to achieve and we can see what we can do.
>>>
>>> Tim
>>>
>>> On Tue, Apr 7, 2015 at 4:15 PM, Tom Arnfeld <to...@duedil.com> wrote:
>>>
>>> > It's not possible to invoke the docker containerizer from outside of
>>> Mesos,
>>> > as far as I know.
>>> >
>>> > If you persue this route, you can run into issues with orphaned
>>> containers
>>> > as your executor may die for some unknown reason, and the container is
>>> still
>>> > running. Recovering from this can be tricky business, so it's better
>>> if you
>>> > can adapt your framework design to fit within the Mesos Task/Executor
>>> > pattern.
>>> >
>>> > --
>>> >
>>> > Tom Arnfeld
>>> > Developer // DueDil
>>> >
>>> > (+44) 7525940046
>>> > 25 Christopher Street, London, EC2A 2BS
>>> >
>>> >
>>> > On Mon, Apr 6, 2015 at 7:00 PM, Vinod Kone <vi...@apache.org>
>>> wrote:
>>> >>
>>> >> Tim, do you want answer this?
>>> >>
>>> >> On Wed, Apr 1, 2015 at 7:27 AM, Tom Fordon <to...@gmail.com>
>>> wrote:
>>> >>>
>>> >>> Hi. I'm trying to understand using docker within a custom executor.
>>> For
>>> >>> each of my tasks, I would like to perform some steps on the node
>>> before
>>> >>> launching a docker container. I was planning on writing a custom
>>> python
>>> >>> executor for this, but I wasn't sure how to launch docker from
>>> within this
>>> >>> executor.
>>> >>>
>>> >>> Can I just call docker in a subprocess using the ContainerInfo from
>>> the
>>> >>> Task? If I do this, how does the Containerizer fit in?
>>> >>>
>>> >>> Thank you,
>>> >>> Tom Fordon
>>> >>
>>> >>
>>> >
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> --
>>>
>>> James DeFelice
>>> 585.241.9488 (voice)
>>> 650.649.6071 (fax)
>>>
>>
>>
>
Re: Custom python executor with Docker
Posted by Tim Chen <ti...@mesosphere.io>.
So currently there is a review out for pre-hooks (
https://reviews.apache.org/r/36185/) before a docker container launches.
We can also add a post hook, but like to see if the specified hook satifies
what you guys are looking for.
Tim
On Tue, Aug 11, 2015 at 4:28 PM, Tom Fordon <to...@gmail.com> wrote:
> We ended up implementing a solution where we did the pre/post steps as
> separate mesos tasks and adding logic to our scheduler to ensure they were
> run on the same machine. If anybody knows of a standard / openly available
> DockerExecutor like what is described below, my team would be greatly
> interested.
>
>
> On Fri, Aug 7, 2015 at 4:01 AM, Kapil Malik <km...@adobe.com> wrote:
>
>> Hi,
>>
>>
>>
>> We have a similar usecase while running multi-user workloads on mesos.
>> Users provide docker images encapsulating application logic, which we (we =
>> say some “Central API”) schedule on Chronos / Marathon. However, we need to
>> run some standard pre / post steps for every docker submitted by users. We
>> have following options –
>>
>>
>>
>> 1. Ask every user to embed their logic inside a pre-defined docker
>> template which will perform pre/post steps.
>>
>> è This is error prone, makes us dependent on whether the users followed
>> template, and not very popular with users either.
>>
>>
>>
>> 2. Extend every user docker (FROM <>) and find a way to add
>> pre-post steps in our docker. Refer this docker when scheduling on chronos
>> / marathon.
>>
>> è Building new dockers does not scale as users and applications grow
>>
>>
>>
>> 3. Write a custom executor which will perform the pre-post steps
>> and manage the user docker lifetime.
>>
>> è Deals with user docker lifetime and is obviously complex.
>>
>>
>>
>> Is there a standard / openly available DockerExecutor which manages the
>> docker lifetime and which I can extend to build my custom executor? This
>> way I will be concerned only with my custom logic (pre/post steps) and
>> still get benefits of a standard way to manage docker containers.
>>
>>
>>
>> Btw, thanks for the meaningful discussion below, it is very helpful.
>>
>>
>>
>> Thanks and regards,
>>
>>
>>
>> Kapil Malik | kmalik@adobe.com | 33430 / 8800836581
>>
>>
>>
>> *From:* James DeFelice [mailto:james.defelice@gmail.com]
>> *Sent:* 09 April 2015 18:12
>> *To:* user@mesos.apache.org
>> *Subject:* Re: Custom python executor with Docker
>>
>>
>>
>> If you can run the pre/post steps in a container then I'd recommend
>> building a Docker image that includes your pre/post step scripting + your
>> algorithm and launching it using the built-in mesos Docker containerizer.
>> It's much simpler than managing the lifetime of the Docker container
>> yourself.
>>
>>
>>
>> On Thu, Apr 9, 2015 at 8:29 AM, Tom Fordon <to...@gmail.com> wrote:
>>
>> Thanks for all the responses, I really appreciate the help. Let me try
>> to state my problem more clearly
>>
>>
>>
>> Our project is performing file-based data processing. I would like to
>> keep the actual algorithm as contained as possible since we are in an R&D
>> setting and will be getting untested code. We have some pre/post steps
>> that need to be run on the same box as the actual algorithm:
>> downloading/uploading files and database calls.
>>
>>
>>
>> We can run the pre/post steps and algorithm within the same container.
>> The algorithm will be a little less contained, but it will work.
>>
>>
>>
>> Docker letting you specify a cgroup parent is really exciting. If I
>> invoke a docker container with the executor as the cgroup-parent are there
>> any other steps I need to perform? Would I need to do anything special to
>> make mesos aware of the resource usage, or is that handled since the docker
>> process would be in the executors cgroup?
>>
>>
>>
>> Thanks again,
>>
>> Tom
>>
>>
>>
>> On Tue, Apr 7, 2015 at 8:10 PM, Timothy Chen <tn...@gmail.com> wrote:
>>
>> Hi Tom(s),
>>
>> Tom Arnfeld is right, if you want to launch your own docker container
>> in your custom executor you will have to handle all the issues
>> yourself and not able to use the Docker containerizer at all.
>>
>> Alternatively, you can actually launch your custom executor in a
>> Docker container by Mesos, by specifying the ContainerInfo in the
>> ExecutorInfo.
>> What this means is that your custom executor is already running in a
>> docker container, and you can do your custom logic afterwards. This
>> does means you can simply just launch multiple containers in the
>> executor anymore.
>>
>> If there is something you want to do and doesnt' fit these let us know
>> what you're trying to achieve and we can see what we can do.
>>
>> Tim
>>
>> On Tue, Apr 7, 2015 at 4:15 PM, Tom Arnfeld <to...@duedil.com> wrote:
>>
>> > It's not possible to invoke the docker containerizer from outside of
>> Mesos,
>> > as far as I know.
>> >
>> > If you persue this route, you can run into issues with orphaned
>> containers
>> > as your executor may die for some unknown reason, and the container is
>> still
>> > running. Recovering from this can be tricky business, so it's better if
>> you
>> > can adapt your framework design to fit within the Mesos Task/Executor
>> > pattern.
>> >
>> > --
>> >
>> > Tom Arnfeld
>> > Developer // DueDil
>> >
>> > (+44) 7525940046
>> > 25 Christopher Street, London, EC2A 2BS
>> >
>> >
>> > On Mon, Apr 6, 2015 at 7:00 PM, Vinod Kone <vi...@apache.org>
>> wrote:
>> >>
>> >> Tim, do you want answer this?
>> >>
>> >> On Wed, Apr 1, 2015 at 7:27 AM, Tom Fordon <to...@gmail.com>
>> wrote:
>> >>>
>> >>> Hi. I'm trying to understand using docker within a custom executor.
>> For
>> >>> each of my tasks, I would like to perform some steps on the node
>> before
>> >>> launching a docker container. I was planning on writing a custom
>> python
>> >>> executor for this, but I wasn't sure how to launch docker from within
>> this
>> >>> executor.
>> >>>
>> >>> Can I just call docker in a subprocess using the ContainerInfo from
>> the
>> >>> Task? If I do this, how does the Containerizer fit in?
>> >>>
>> >>> Thank you,
>> >>> Tom Fordon
>> >>
>> >>
>> >
>>
>>
>>
>>
>>
>>
>>
>> --
>>
>> James DeFelice
>> 585.241.9488 (voice)
>> 650.649.6071 (fax)
>>
>
>
Re: Custom python executor with Docker
Posted by Tom Fordon <to...@gmail.com>.
We ended up implementing a solution where we did the pre/post steps as
separate mesos tasks and adding logic to our scheduler to ensure they were
run on the same machine. If anybody knows of a standard / openly available
DockerExecutor like what is described below, my team would be greatly
interested.
On Fri, Aug 7, 2015 at 4:01 AM, Kapil Malik <km...@adobe.com> wrote:
> Hi,
>
>
>
> We have a similar usecase while running multi-user workloads on mesos.
> Users provide docker images encapsulating application logic, which we (we =
> say some “Central API”) schedule on Chronos / Marathon. However, we need to
> run some standard pre / post steps for every docker submitted by users. We
> have following options –
>
>
>
> 1. Ask every user to embed their logic inside a pre-defined docker
> template which will perform pre/post steps.
>
> è This is error prone, makes us dependent on whether the users followed
> template, and not very popular with users either.
>
>
>
> 2. Extend every user docker (FROM <>) and find a way to add
> pre-post steps in our docker. Refer this docker when scheduling on chronos
> / marathon.
>
> è Building new dockers does not scale as users and applications grow
>
>
>
> 3. Write a custom executor which will perform the pre-post steps
> and manage the user docker lifetime.
>
> è Deals with user docker lifetime and is obviously complex.
>
>
>
> Is there a standard / openly available DockerExecutor which manages the
> docker lifetime and which I can extend to build my custom executor? This
> way I will be concerned only with my custom logic (pre/post steps) and
> still get benefits of a standard way to manage docker containers.
>
>
>
> Btw, thanks for the meaningful discussion below, it is very helpful.
>
>
>
> Thanks and regards,
>
>
>
> Kapil Malik | kmalik@adobe.com | 33430 / 8800836581
>
>
>
> *From:* James DeFelice [mailto:james.defelice@gmail.com]
> *Sent:* 09 April 2015 18:12
> *To:* user@mesos.apache.org
> *Subject:* Re: Custom python executor with Docker
>
>
>
> If you can run the pre/post steps in a container then I'd recommend
> building a Docker image that includes your pre/post step scripting + your
> algorithm and launching it using the built-in mesos Docker containerizer.
> It's much simpler than managing the lifetime of the Docker container
> yourself.
>
>
>
> On Thu, Apr 9, 2015 at 8:29 AM, Tom Fordon <to...@gmail.com> wrote:
>
> Thanks for all the responses, I really appreciate the help. Let me try to
> state my problem more clearly
>
>
>
> Our project is performing file-based data processing. I would like to
> keep the actual algorithm as contained as possible since we are in an R&D
> setting and will be getting untested code. We have some pre/post steps
> that need to be run on the same box as the actual algorithm:
> downloading/uploading files and database calls.
>
>
>
> We can run the pre/post steps and algorithm within the same container.
> The algorithm will be a little less contained, but it will work.
>
>
>
> Docker letting you specify a cgroup parent is really exciting. If I
> invoke a docker container with the executor as the cgroup-parent are there
> any other steps I need to perform? Would I need to do anything special to
> make mesos aware of the resource usage, or is that handled since the docker
> process would be in the executors cgroup?
>
>
>
> Thanks again,
>
> Tom
>
>
>
> On Tue, Apr 7, 2015 at 8:10 PM, Timothy Chen <tn...@gmail.com> wrote:
>
> Hi Tom(s),
>
> Tom Arnfeld is right, if you want to launch your own docker container
> in your custom executor you will have to handle all the issues
> yourself and not able to use the Docker containerizer at all.
>
> Alternatively, you can actually launch your custom executor in a
> Docker container by Mesos, by specifying the ContainerInfo in the
> ExecutorInfo.
> What this means is that your custom executor is already running in a
> docker container, and you can do your custom logic afterwards. This
> does means you can simply just launch multiple containers in the
> executor anymore.
>
> If there is something you want to do and doesnt' fit these let us know
> what you're trying to achieve and we can see what we can do.
>
> Tim
>
> On Tue, Apr 7, 2015 at 4:15 PM, Tom Arnfeld <to...@duedil.com> wrote:
>
> > It's not possible to invoke the docker containerizer from outside of
> Mesos,
> > as far as I know.
> >
> > If you persue this route, you can run into issues with orphaned
> containers
> > as your executor may die for some unknown reason, and the container is
> still
> > running. Recovering from this can be tricky business, so it's better if
> you
> > can adapt your framework design to fit within the Mesos Task/Executor
> > pattern.
> >
> > --
> >
> > Tom Arnfeld
> > Developer // DueDil
> >
> > (+44) 7525940046
> > 25 Christopher Street, London, EC2A 2BS
> >
> >
> > On Mon, Apr 6, 2015 at 7:00 PM, Vinod Kone <vi...@apache.org> wrote:
> >>
> >> Tim, do you want answer this?
> >>
> >> On Wed, Apr 1, 2015 at 7:27 AM, Tom Fordon <to...@gmail.com>
> wrote:
> >>>
> >>> Hi. I'm trying to understand using docker within a custom executor.
> For
> >>> each of my tasks, I would like to perform some steps on the node before
> >>> launching a docker container. I was planning on writing a custom python
> >>> executor for this, but I wasn't sure how to launch docker from within
> this
> >>> executor.
> >>>
> >>> Can I just call docker in a subprocess using the ContainerInfo from the
> >>> Task? If I do this, how does the Containerizer fit in?
> >>>
> >>> Thank you,
> >>> Tom Fordon
> >>
> >>
> >
>
>
>
>
>
>
>
> --
>
> James DeFelice
> 585.241.9488 (voice)
> 650.649.6071 (fax)
>
RE: Custom python executor with Docker
Posted by Kapil Malik <km...@adobe.com>.
Hi,
We have a similar usecase while running multi-user workloads on mesos. Users provide docker images encapsulating application logic, which we (we = say some “Central API”) schedule on Chronos / Marathon. However, we need to run some standard pre / post steps for every docker submitted by users. We have following options –
1. Ask every user to embed their logic inside a pre-defined docker template which will perform pre/post steps.
è This is error prone, makes us dependent on whether the users followed template, and not very popular with users either.
2. Extend every user docker (FROM <>) and find a way to add pre-post steps in our docker. Refer this docker when scheduling on chronos / marathon.
è Building new dockers does not scale as users and applications grow
3. Write a custom executor which will perform the pre-post steps and manage the user docker lifetime.
è Deals with user docker lifetime and is obviously complex.
Is there a standard / openly available DockerExecutor which manages the docker lifetime and which I can extend to build my custom executor? This way I will be concerned only with my custom logic (pre/post steps) and still get benefits of a standard way to manage docker containers.
Btw, thanks for the meaningful discussion below, it is very helpful.
Thanks and regards,
Kapil Malik | kmalik@adobe.com<ma...@adobe.com> | 33430 / 8800836581
From: James DeFelice [mailto:james.defelice@gmail.com]
Sent: 09 April 2015 18:12
To: user@mesos.apache.org
Subject: Re: Custom python executor with Docker
If you can run the pre/post steps in a container then I'd recommend building a Docker image that includes your pre/post step scripting + your algorithm and launching it using the built-in mesos Docker containerizer. It's much simpler than managing the lifetime of the Docker container yourself.
On Thu, Apr 9, 2015 at 8:29 AM, Tom Fordon <to...@gmail.com>> wrote:
Thanks for all the responses, I really appreciate the help. Let me try to state my problem more clearly
Our project is performing file-based data processing. I would like to keep the actual algorithm as contained as possible since we are in an R&D setting and will be getting untested code. We have some pre/post steps that need to be run on the same box as the actual algorithm: downloading/uploading files and database calls.
We can run the pre/post steps and algorithm within the same container. The algorithm will be a little less contained, but it will work.
Docker letting you specify a cgroup parent is really exciting. If I invoke a docker container with the executor as the cgroup-parent are there any other steps I need to perform? Would I need to do anything special to make mesos aware of the resource usage, or is that handled since the docker process would be in the executors cgroup?
Thanks again,
Tom
On Tue, Apr 7, 2015 at 8:10 PM, Timothy Chen <tn...@gmail.com>> wrote:
Hi Tom(s),
Tom Arnfeld is right, if you want to launch your own docker container
in your custom executor you will have to handle all the issues
yourself and not able to use the Docker containerizer at all.
Alternatively, you can actually launch your custom executor in a
Docker container by Mesos, by specifying the ContainerInfo in the
ExecutorInfo.
What this means is that your custom executor is already running in a
docker container, and you can do your custom logic afterwards. This
does means you can simply just launch multiple containers in the
executor anymore.
If there is something you want to do and doesnt' fit these let us know
what you're trying to achieve and we can see what we can do.
Tim
On Tue, Apr 7, 2015 at 4:15 PM, Tom Arnfeld <to...@duedil.com>> wrote:
> It's not possible to invoke the docker containerizer from outside of Mesos,
> as far as I know.
>
> If you persue this route, you can run into issues with orphaned containers
> as your executor may die for some unknown reason, and the container is still
> running. Recovering from this can be tricky business, so it's better if you
> can adapt your framework design to fit within the Mesos Task/Executor
> pattern.
>
> --
>
> Tom Arnfeld
> Developer // DueDil
>
> (+44) 7525940046<tel:%28%2B44%29%207525940046>
> 25 Christopher Street, London, EC2A 2BS
>
>
> On Mon, Apr 6, 2015 at 7:00 PM, Vinod Kone <vi...@apache.org>> wrote:
>>
>> Tim, do you want answer this?
>>
>> On Wed, Apr 1, 2015 at 7:27 AM, Tom Fordon <to...@gmail.com>> wrote:
>>>
>>> Hi. I'm trying to understand using docker within a custom executor. For
>>> each of my tasks, I would like to perform some steps on the node before
>>> launching a docker container. I was planning on writing a custom python
>>> executor for this, but I wasn't sure how to launch docker from within this
>>> executor.
>>>
>>> Can I just call docker in a subprocess using the ContainerInfo from the
>>> Task? If I do this, how does the Containerizer fit in?
>>>
>>> Thank you,
>>> Tom Fordon
>>
>>
>
--
James DeFelice
585.241.9488 (voice)
650.649.6071 (fax)
Re: Custom python executor with Docker
Posted by James DeFelice <ja...@gmail.com>.
If you can run the pre/post steps in a container then I'd recommend
building a Docker image that includes your pre/post step scripting + your
algorithm and launching it using the built-in mesos Docker containerizer.
It's much simpler than managing the lifetime of the Docker container
yourself.
On Thu, Apr 9, 2015 at 8:29 AM, Tom Fordon <to...@gmail.com> wrote:
> Thanks for all the responses, I really appreciate the help. Let me try to
> state my problem more clearly
>
> Our project is performing file-based data processing. I would like to
> keep the actual algorithm as contained as possible since we are in an R&D
> setting and will be getting untested code. We have some pre/post steps
> that need to be run on the same box as the actual algorithm:
> downloading/uploading files and database calls.
>
> We can run the pre/post steps and algorithm within the same container.
> The algorithm will be a little less contained, but it will work.
>
> Docker letting you specify a cgroup parent is really exciting. If I
> invoke a docker container with the executor as the cgroup-parent are there
> any other steps I need to perform? Would I need to do anything special to
> make mesos aware of the resource usage, or is that handled since the docker
> process would be in the executors cgroup?
>
> Thanks again,
> Tom
>
> On Tue, Apr 7, 2015 at 8:10 PM, Timothy Chen <tn...@gmail.com> wrote:
>
>> Hi Tom(s),
>>
>> Tom Arnfeld is right, if you want to launch your own docker container
>> in your custom executor you will have to handle all the issues
>> yourself and not able to use the Docker containerizer at all.
>>
>> Alternatively, you can actually launch your custom executor in a
>> Docker container by Mesos, by specifying the ContainerInfo in the
>> ExecutorInfo.
>> What this means is that your custom executor is already running in a
>> docker container, and you can do your custom logic afterwards. This
>> does means you can simply just launch multiple containers in the
>> executor anymore.
>>
>> If there is something you want to do and doesnt' fit these let us know
>> what you're trying to achieve and we can see what we can do.
>>
>> Tim
>>
>> On Tue, Apr 7, 2015 at 4:15 PM, Tom Arnfeld <to...@duedil.com> wrote:
>> > It's not possible to invoke the docker containerizer from outside of
>> Mesos,
>> > as far as I know.
>> >
>> > If you persue this route, you can run into issues with orphaned
>> containers
>> > as your executor may die for some unknown reason, and the container is
>> still
>> > running. Recovering from this can be tricky business, so it's better if
>> you
>> > can adapt your framework design to fit within the Mesos Task/Executor
>> > pattern.
>> >
>> > --
>> >
>> > Tom Arnfeld
>> > Developer // DueDil
>> >
>> > (+44) 7525940046
>> > 25 Christopher Street, London, EC2A 2BS
>> >
>> >
>> > On Mon, Apr 6, 2015 at 7:00 PM, Vinod Kone <vi...@apache.org>
>> wrote:
>> >>
>> >> Tim, do you want answer this?
>> >>
>> >> On Wed, Apr 1, 2015 at 7:27 AM, Tom Fordon <to...@gmail.com>
>> wrote:
>> >>>
>> >>> Hi. I'm trying to understand using docker within a custom executor.
>> For
>> >>> each of my tasks, I would like to perform some steps on the node
>> before
>> >>> launching a docker container. I was planning on writing a custom
>> python
>> >>> executor for this, but I wasn't sure how to launch docker from within
>> this
>> >>> executor.
>> >>>
>> >>> Can I just call docker in a subprocess using the ContainerInfo from
>> the
>> >>> Task? If I do this, how does the Containerizer fit in?
>> >>>
>> >>> Thank you,
>> >>> Tom Fordon
>> >>
>> >>
>> >
>>
>
>
--
James DeFelice
585.241.9488 (voice)
650.649.6071 (fax)
Re: Custom python executor with Docker
Posted by Tom Fordon <to...@gmail.com>.
Thanks for all the responses, I really appreciate the help. Let me try to
state my problem more clearly
Our project is performing file-based data processing. I would like to keep
the actual algorithm as contained as possible since we are in an R&D
setting and will be getting untested code. We have some pre/post steps
that need to be run on the same box as the actual algorithm:
downloading/uploading files and database calls.
We can run the pre/post steps and algorithm within the same container. The
algorithm will be a little less contained, but it will work.
Docker letting you specify a cgroup parent is really exciting. If I invoke
a docker container with the executor as the cgroup-parent are there any
other steps I need to perform? Would I need to do anything special to make
mesos aware of the resource usage, or is that handled since the docker
process would be in the executors cgroup?
Thanks again,
Tom
On Tue, Apr 7, 2015 at 8:10 PM, Timothy Chen <tn...@gmail.com> wrote:
> Hi Tom(s),
>
> Tom Arnfeld is right, if you want to launch your own docker container
> in your custom executor you will have to handle all the issues
> yourself and not able to use the Docker containerizer at all.
>
> Alternatively, you can actually launch your custom executor in a
> Docker container by Mesos, by specifying the ContainerInfo in the
> ExecutorInfo.
> What this means is that your custom executor is already running in a
> docker container, and you can do your custom logic afterwards. This
> does means you can simply just launch multiple containers in the
> executor anymore.
>
> If there is something you want to do and doesnt' fit these let us know
> what you're trying to achieve and we can see what we can do.
>
> Tim
>
> On Tue, Apr 7, 2015 at 4:15 PM, Tom Arnfeld <to...@duedil.com> wrote:
> > It's not possible to invoke the docker containerizer from outside of
> Mesos,
> > as far as I know.
> >
> > If you persue this route, you can run into issues with orphaned
> containers
> > as your executor may die for some unknown reason, and the container is
> still
> > running. Recovering from this can be tricky business, so it's better if
> you
> > can adapt your framework design to fit within the Mesos Task/Executor
> > pattern.
> >
> > --
> >
> > Tom Arnfeld
> > Developer // DueDil
> >
> > (+44) 7525940046
> > 25 Christopher Street, London, EC2A 2BS
> >
> >
> > On Mon, Apr 6, 2015 at 7:00 PM, Vinod Kone <vi...@apache.org> wrote:
> >>
> >> Tim, do you want answer this?
> >>
> >> On Wed, Apr 1, 2015 at 7:27 AM, Tom Fordon <to...@gmail.com>
> wrote:
> >>>
> >>> Hi. I'm trying to understand using docker within a custom executor.
> For
> >>> each of my tasks, I would like to perform some steps on the node before
> >>> launching a docker container. I was planning on writing a custom python
> >>> executor for this, but I wasn't sure how to launch docker from within
> this
> >>> executor.
> >>>
> >>> Can I just call docker in a subprocess using the ContainerInfo from the
> >>> Task? If I do this, how does the Containerizer fit in?
> >>>
> >>> Thank you,
> >>> Tom Fordon
> >>
> >>
> >
>
Re: Custom python executor with Docker
Posted by Timothy Chen <tn...@gmail.com>.
Hi Tom(s),
Tom Arnfeld is right, if you want to launch your own docker container
in your custom executor you will have to handle all the issues
yourself and not able to use the Docker containerizer at all.
Alternatively, you can actually launch your custom executor in a
Docker container by Mesos, by specifying the ContainerInfo in the
ExecutorInfo.
What this means is that your custom executor is already running in a
docker container, and you can do your custom logic afterwards. This
does means you can simply just launch multiple containers in the
executor anymore.
If there is something you want to do and doesnt' fit these let us know
what you're trying to achieve and we can see what we can do.
Tim
On Tue, Apr 7, 2015 at 4:15 PM, Tom Arnfeld <to...@duedil.com> wrote:
> It's not possible to invoke the docker containerizer from outside of Mesos,
> as far as I know.
>
> If you persue this route, you can run into issues with orphaned containers
> as your executor may die for some unknown reason, and the container is still
> running. Recovering from this can be tricky business, so it's better if you
> can adapt your framework design to fit within the Mesos Task/Executor
> pattern.
>
> --
>
> Tom Arnfeld
> Developer // DueDil
>
> (+44) 7525940046
> 25 Christopher Street, London, EC2A 2BS
>
>
> On Mon, Apr 6, 2015 at 7:00 PM, Vinod Kone <vi...@apache.org> wrote:
>>
>> Tim, do you want answer this?
>>
>> On Wed, Apr 1, 2015 at 7:27 AM, Tom Fordon <to...@gmail.com> wrote:
>>>
>>> Hi. I'm trying to understand using docker within a custom executor. For
>>> each of my tasks, I would like to perform some steps on the node before
>>> launching a docker container. I was planning on writing a custom python
>>> executor for this, but I wasn't sure how to launch docker from within this
>>> executor.
>>>
>>> Can I just call docker in a subprocess using the ContainerInfo from the
>>> Task? If I do this, how does the Containerizer fit in?
>>>
>>> Thank you,
>>> Tom Fordon
>>
>>
>
Re: Custom python executor with Docker
Posted by Tom Arnfeld <to...@duedil.com>.
It's not possible to invoke the docker containerizer from outside of Mesos, as far as I know.
If you persue this route, you can run into issues with orphaned containers as your executor may die for some unknown reason, and the container is still running. Recovering from this can be tricky business, so it's better if you can adapt your framework design to fit within the Mesos Task/Executor pattern.
--
Tom Arnfeld
Developer // DueDil
(+44) 7525940046
25 Christopher Street, London, EC2A 2BS
On Mon, Apr 6, 2015 at 7:00 PM, Vinod Kone <vi...@apache.org> wrote:
> Tim, do you want answer this?
> On Wed, Apr 1, 2015 at 7:27 AM, Tom Fordon <to...@gmail.com> wrote:
>> Hi. I'm trying to understand using docker within a custom executor. For
>> each of my tasks, I would like to perform some steps on the node before
>> launching a docker container. I was planning on writing a custom python
>> executor for this, but I wasn't sure how to launch docker from within this
>> executor.
>>
>> Can I just call docker in a subprocess using the ContainerInfo from the
>> Task? If I do this, how does the Containerizer fit in?
>> Thank you,
>> Tom Fordon
>>
Re: Custom python executor with Docker
Posted by Vinod Kone <vi...@apache.org>.
Tim, do you want answer this?
On Wed, Apr 1, 2015 at 7:27 AM, Tom Fordon <to...@gmail.com> wrote:
> Hi. I'm trying to understand using docker within a custom executor. For
> each of my tasks, I would like to perform some steps on the node before
> launching a docker container. I was planning on writing a custom python
> executor for this, but I wasn't sure how to launch docker from within this
> executor.
>
> Can I just call docker in a subprocess using the ContainerInfo from the
> Task? If I do this, how does the Containerizer fit in?
> Thank you,
> Tom Fordon
>