RE: Custom python executor with Docker

["Plotka, Bartlomiej" <ba...@intel.com>]

Hi All,

I can see there are many use cases for extending Docker tasks with some pre/post work.

In our solution we want to prepare and attach some volumes (e.g ceph volumes) to Docker. As you know we cannot (easily) attach a volume to the running Docker, so we need pre script for preparing & attaching in one task. Unfortunately, we also need to clean-up these volumes and it seems there is not any “slavePostLaunchDockerHook” available now in Mesos.

We’ve developed short script for running such tasks using common Mesos Containerizer and without any hook module in Mesos. We can just put it in the Marathon “command” parameter. Do you think such created Docker container will have limited cpu shares & memory (using Mesos resources)?

Here is pseudo-bash code:
#!/bin/bash

function cleanup {{
  docker rm <docker_name>;
 <clean up volumes>
}}

trap cleanup SIGKILL SIGSTOP SIGHUP SIGINT SIGTERM;

<prepare volumes>
docker run --name <docker-name> -p <port>:22 <volumes> <docker image> bash -c "<docker cmd>";

We are aware that there wouldn’t be any sandbox support and any other Docker Containerizer features. However, our main concern is about cgroups… We are not sure if our newly created Docker (via this script) will inherit our task’s cgroups and as a result will limit memory, cpus via default Mesos Task’s resource parameters.

Anyone knows the answer?

As far as I understand without  special configuration ‘Docker run’ somehow pass the ‘run’ command to the docker daemon so we don’t have control of container cgroup…

Kind Regards,
Bartek Plotka (bplotka)

From: Kevin Sweeney [mailto:kevints@apache.org]
Sent: Wednesday, August 12, 2015 1:52 AM
To: user@mesos.apache.org
Subject: Re: Custom python executor with Docker

Apache Aurora [1] uses a custom Python executor and supports Docker via the containerizer. There's just one problem - the container has to have a Python2.7 runtime inside that can run the executor PEX file [2]. So if you're okay with that restriction you're in business (and you can use the Aurora configuration DSL to describe setup/teardown steps).

[1] https://aurora.apache.org
[2] https://pex.readthedocs.org/en/latest/

On Tue, Aug 11, 2015 at 4:42 PM, Tim Chen <ti...@mesosphere.io>> wrote:
So currently there is a review out for pre-hooks (https://reviews.apache.org/r/36185/) before a docker container launches.

We can also add a post hook, but like to see if the specified hook satifies what you guys are looking for.

Tim

On Tue, Aug 11, 2015 at 4:28 PM, Tom Fordon <to...@gmail.com>> wrote:
We ended up implementing a solution where we did the pre/post steps as separate mesos tasks and adding logic to our scheduler to ensure they were run on the same machine.  If anybody knows of a standard / openly available DockerExecutor like what is described below, my team would be greatly interested.


On Fri, Aug 7, 2015 at 4:01 AM, Kapil Malik <km...@adobe.com>> wrote:
Hi,

We have a similar usecase while running multi-user workloads on mesos. Users provide docker images encapsulating application logic, which we (we = say some “Central API”) schedule on Chronos / Marathon. However, we need to run some standard pre / post steps for every docker submitted by users. We have following options –


1.       Ask every user to embed their logic inside a pre-defined docker template which will perform pre/post steps.

==> This is error prone, makes us dependent on whether the users followed template, and not very popular with users either.



2.       Extend every user docker (FROM <>) and find a way to add pre-post steps in our docker. Refer this docker when scheduling on chronos / marathon.

==> Building new dockers does not scale as users and applications grow



3.       Write a custom executor which will perform the pre-post steps and manage the user docker lifetime.

==> Deals with user docker lifetime and is obviously complex.

Is there a standard / openly available DockerExecutor which manages the docker lifetime and which I can extend to build my custom executor? This way I will be concerned only with my custom logic (pre/post steps) and still get benefits of a standard way to manage docker containers.

Btw, thanks for the meaningful discussion below, it is very helpful.

Thanks and regards,

Kapil Malik | kmalik@adobe.com<ma...@adobe.com> | 33430 / 8800836581

From: James DeFelice [mailto:james.defelice@gmail.com<ma...@gmail.com>]
Sent: 09 April 2015 18:12
To: user@mesos.apache.org<ma...@mesos.apache.org>
Subject: Re: Custom python executor with Docker

If you can run the pre/post steps in a container then I'd recommend building a Docker image that includes your pre/post step scripting + your algorithm and launching it using the built-in mesos Docker containerizer. It's much simpler than managing the lifetime of the Docker container yourself.

On Thu, Apr 9, 2015 at 8:29 AM, Tom Fordon <to...@gmail.com>> wrote:
Thanks for all the responses, I really appreciate the help.  Let me try to state my problem more clearly

Our project is performing file-based data processing.  I would like to keep the actual algorithm as contained as possible since we are in an R&D setting and will be getting untested code.  We have some pre/post steps that need to be run on the same box as the actual algorithm: downloading/uploading files and database calls.

We can run the pre/post steps and algorithm within the same container.  The algorithm will be a little less contained, but it will work.

Docker letting you specify a cgroup parent is really exciting.  If I invoke a docker container with the executor as the cgroup-parent are there any other steps I need to perform?  Would I need to do anything special to make mesos aware of the resource usage, or is that handled since the docker process would be in the executors cgroup?

Thanks again,
Tom

On Tue, Apr 7, 2015 at 8:10 PM, Timothy Chen <tn...@gmail.com>> wrote:
Hi Tom(s),

Tom Arnfeld is right, if you want to launch your own docker container
in your custom executor you will have to handle all the issues
yourself and not able to use the Docker containerizer at all.

Alternatively, you can actually launch your custom executor in a
Docker container by Mesos, by specifying the ContainerInfo in the
ExecutorInfo.
What this means is that your custom executor is already running in a
docker container, and you can do your custom logic afterwards. This
does means you can simply just launch multiple containers in the
executor anymore.

If there is something you want to do and doesnt' fit these let us know
what you're trying to achieve and we can see what we can do.

Tim

On Tue, Apr 7, 2015 at 4:15 PM, Tom Arnfeld <to...@duedil.com>> wrote:
> It's not possible to invoke the docker containerizer from outside of Mesos,
> as far as I know.
>
> If you persue this route, you can run into issues with orphaned containers
> as your executor may die for some unknown reason, and the container is still
> running. Recovering from this can be tricky business, so it's better if you
> can adapt your framework design to fit within the Mesos Task/Executor
> pattern.
>
> --
>
> Tom Arnfeld
> Developer // DueDil
>
> (+44) 7525940046<tel:%28%2B44%29%207525940046>
> 25 Christopher Street, London, EC2A 2BS
>
>
> On Mon, Apr 6, 2015 at 7:00 PM, Vinod Kone <vi...@apache.org>> wrote:
>>
>> Tim, do you want answer this?
>>
>> On Wed, Apr 1, 2015 at 7:27 AM, Tom Fordon <to...@gmail.com>> wrote:
>>>
>>> Hi.  I'm trying to understand using docker within a custom executor. For
>>> each of my tasks, I would like to perform some steps on the node before
>>> launching a docker container. I was planning on writing a custom python
>>> executor for this, but I wasn't sure how to launch docker from within this
>>> executor.
>>>
>>> Can I just call docker in a subprocess using the ContainerInfo from the
>>> Task? If I do this, how does the Containerizer fit in?
>>>
>>> Thank you,
>>> Tom Fordon
>>
>>
>




--
James DeFelice
585.241.9488<tel:585.241.9488> (voice)
650.649.6071<tel:650.649.6071> (fax)



--------------------------------------------------------------------

Intel Technology Poland sp. z o.o.
ul. Slowackiego 173 | 80-298 Gdansk | Sad Rejonowy Gdansk Polnoc | VII Wydzial Gospodarczy Krajowego Rejestru Sadowego - KRS 101882 | NIP 957-07-52-316 | Kapital zakladowy 200.000 PLN.

Ta wiadomosc wraz z zalacznikami jest przeznaczona dla okreslonego adresata i moze zawierac informacje poufne. W razie przypadkowego otrzymania tej wiadomosci, prosimy o powiadomienie nadawcy oraz trwale jej usuniecie; jakiekolwiek
przegladanie lub rozpowszechnianie jest zabronione.
This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). If you are not the intended recipient, please contact the sender and delete all copies; any review or distribution by
others is strictly prohibited.