You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@aurora.apache.org by Kevin Sweeney <ks...@twitter.com.INVALID> on 2015/08/14 20:39:17 UTC
Improvements to container support with custom executor
(cross-posting for a wider audience)
Hi folks,
With mesos-0.23.0 it looks like a new dependency made it in for TLS
support. While this is fine in theory it actually makes the Docker
Containerizer story very difficult to reason about.
Here's the situation: Aurora uses a custom Python executor for its tasks.
Because of the design of the containerizer the executor runs in the context
of the container. This means that the container has to be able to run the
executor (in practice this means it needs a python2.7 installation and some
shared libraries libmesos links to). Since most containers don't contain
Aurora's executor, Aurora hacked around this by using the executor in
$MESOS_SANDBOX plus a requirement that hosted containers be able to run the
executor. However, with the upgrade to mesos-0.23.0 the containers that
could run the 0.22.0 executor no longer work due to the new dependency on
libcurl-nss.
This is not a problem limited to Docker - I don't see how this design will
work with *any* container runtime - we can never upgrade the executor
without upgrading all the containers to contain its new dependencies, which
at a minimum means we must rebuild them whenever mesos gains a new
dependency.
Does anyone with more experience with these APIs have a suggestion here? It
seems we need to make the executor run in the context of the host OS and
aware of the container (or maybe we have mesos launch a container with the
executor+its dependencies and have it launch a child container).
--
Kevin Sweeney
@kts
Re: Improvements to container support with custom executor
Posted by Jie Yu <yu...@gmail.com>.
Steven,
To be clear, will this Unified Containerizer give us the ability to run
> Docker images *without* the Docker daemon?
Yes. Also, you'll be able to leverage all the isolators we developed for
Mesos containerizer (cgroups cpu/mem, perf, network, disk, etc.) while
still be able to use Docker images.
- Jie
On Fri, Aug 14, 2015 at 3:19 PM, Steven Schlansker <
sschlansker@opentable.com> wrote:
> To be clear, will this Unified Containerizer give us the ability to run
> Docker images *without* the Docker daemon?
>
> That would be fantastic! This would remove one of the least reliable
> components of the ecosystem (the Docker daemon) while preserving the
> illusion of Docker to our end users.
>
> On Aug 14, 2015, at 12:17 PM, Jie Yu <yu...@gmail.com> wrote:
>
> > Kevin,
> >
> > Thanks for bringing this up! This question is related to the work we are
> currently doing for Filesystem Isolation (MESOS-2386) and Unified Container
> (MESOS-2840).
> >
> > In fact, if you think about using Mesos containerizer and want to run a
> task (using command executor) under a specified image, we have the same
> issue for Mesos' own command executor (MESOS-3004).
> >
> > I've proposed a solution in this doc. The main idea is that we allow the
> custom executor to still run under the host filesystem (so that it does not
> have to deal with dependency issues). Mesos will provision the image (the
> image requested by the user), and mount in as a Volume in the container.
> The custom executor is responsible for chrooting (or pivot_root if it has
> root privilege) into the user specified image before execing the user
> process.
> >
> > The above proposal is for only Mesos containerizer currently. But with
> the Unified Container (MESOS-2840) work, Mesos containerizer is going to
> support Docker images as well.
> >
> > cc Tim Chen
> >
> > - Jie
> >
> >
> > On Fri, Aug 14, 2015 at 11:39 AM, Kevin Sweeney
> <ks...@twitter.com.invalid> wrote:
> > (cross-posting for a wider audience)
> >
> > Hi folks,
> >
> > With mesos-0.23.0 it looks like a new dependency made it in for TLS
> > support. While this is fine in theory it actually makes the Docker
> > Containerizer story very difficult to reason about.
> >
> > Here's the situation: Aurora uses a custom Python executor for its tasks.
> > Because of the design of the containerizer the executor runs in the
> context
> > of the container. This means that the container has to be able to run the
> > executor (in practice this means it needs a python2.7 installation and
> some
> > shared libraries libmesos links to). Since most containers don't contain
> > Aurora's executor, Aurora hacked around this by using the executor in
> > $MESOS_SANDBOX plus a requirement that hosted containers be able to run
> the
> > executor. However, with the upgrade to mesos-0.23.0 the containers that
> > could run the 0.22.0 executor no longer work due to the new dependency on
> > libcurl-nss.
> >
> > This is not a problem limited to Docker - I don't see how this design
> will
> > work with *any* container runtime - we can never upgrade the executor
> > without upgrading all the containers to contain its new dependencies,
> which
> > at a minimum means we must rebuild them whenever mesos gains a new
> > dependency.
> >
> > Does anyone with more experience with these APIs have a suggestion here?
> It
> > seems we need to make the executor run in the context of the host OS and
> > aware of the container (or maybe we have mesos launch a container with
> the
> > executor+its dependencies and have it launch a child container).
> > --
> > Kevin Sweeney
> > @kts
> >
>
>
Re: Improvements to container support with custom executor
Posted by Jie Yu <yu...@gmail.com>.
Steven,
To be clear, will this Unified Containerizer give us the ability to run
> Docker images *without* the Docker daemon?
Yes. Also, you'll be able to leverage all the isolators we developed for
Mesos containerizer (cgroups cpu/mem, perf, network, disk, etc.) while
still be able to use Docker images.
- Jie
On Fri, Aug 14, 2015 at 3:19 PM, Steven Schlansker <
sschlansker@opentable.com> wrote:
> To be clear, will this Unified Containerizer give us the ability to run
> Docker images *without* the Docker daemon?
>
> That would be fantastic! This would remove one of the least reliable
> components of the ecosystem (the Docker daemon) while preserving the
> illusion of Docker to our end users.
>
> On Aug 14, 2015, at 12:17 PM, Jie Yu <yu...@gmail.com> wrote:
>
> > Kevin,
> >
> > Thanks for bringing this up! This question is related to the work we are
> currently doing for Filesystem Isolation (MESOS-2386) and Unified Container
> (MESOS-2840).
> >
> > In fact, if you think about using Mesos containerizer and want to run a
> task (using command executor) under a specified image, we have the same
> issue for Mesos' own command executor (MESOS-3004).
> >
> > I've proposed a solution in this doc. The main idea is that we allow the
> custom executor to still run under the host filesystem (so that it does not
> have to deal with dependency issues). Mesos will provision the image (the
> image requested by the user), and mount in as a Volume in the container.
> The custom executor is responsible for chrooting (or pivot_root if it has
> root privilege) into the user specified image before execing the user
> process.
> >
> > The above proposal is for only Mesos containerizer currently. But with
> the Unified Container (MESOS-2840) work, Mesos containerizer is going to
> support Docker images as well.
> >
> > cc Tim Chen
> >
> > - Jie
> >
> >
> > On Fri, Aug 14, 2015 at 11:39 AM, Kevin Sweeney
> <ks...@twitter.com.invalid> wrote:
> > (cross-posting for a wider audience)
> >
> > Hi folks,
> >
> > With mesos-0.23.0 it looks like a new dependency made it in for TLS
> > support. While this is fine in theory it actually makes the Docker
> > Containerizer story very difficult to reason about.
> >
> > Here's the situation: Aurora uses a custom Python executor for its tasks.
> > Because of the design of the containerizer the executor runs in the
> context
> > of the container. This means that the container has to be able to run the
> > executor (in practice this means it needs a python2.7 installation and
> some
> > shared libraries libmesos links to). Since most containers don't contain
> > Aurora's executor, Aurora hacked around this by using the executor in
> > $MESOS_SANDBOX plus a requirement that hosted containers be able to run
> the
> > executor. However, with the upgrade to mesos-0.23.0 the containers that
> > could run the 0.22.0 executor no longer work due to the new dependency on
> > libcurl-nss.
> >
> > This is not a problem limited to Docker - I don't see how this design
> will
> > work with *any* container runtime - we can never upgrade the executor
> > without upgrading all the containers to contain its new dependencies,
> which
> > at a minimum means we must rebuild them whenever mesos gains a new
> > dependency.
> >
> > Does anyone with more experience with these APIs have a suggestion here?
> It
> > seems we need to make the executor run in the context of the host OS and
> > aware of the container (or maybe we have mesos launch a container with
> the
> > executor+its dependencies and have it launch a child container).
> > --
> > Kevin Sweeney
> > @kts
> >
>
>
Re: Improvements to container support with custom executor
Posted by Steven Schlansker <ss...@opentable.com>.
To be clear, will this Unified Containerizer give us the ability to run Docker images *without* the Docker daemon?
That would be fantastic! This would remove one of the least reliable components of the ecosystem (the Docker daemon) while preserving the illusion of Docker to our end users.
On Aug 14, 2015, at 12:17 PM, Jie Yu <yu...@gmail.com> wrote:
> Kevin,
>
> Thanks for bringing this up! This question is related to the work we are currently doing for Filesystem Isolation (MESOS-2386) and Unified Container (MESOS-2840).
>
> In fact, if you think about using Mesos containerizer and want to run a task (using command executor) under a specified image, we have the same issue for Mesos' own command executor (MESOS-3004).
>
> I've proposed a solution in this doc. The main idea is that we allow the custom executor to still run under the host filesystem (so that it does not have to deal with dependency issues). Mesos will provision the image (the image requested by the user), and mount in as a Volume in the container. The custom executor is responsible for chrooting (or pivot_root if it has root privilege) into the user specified image before execing the user process.
>
> The above proposal is for only Mesos containerizer currently. But with the Unified Container (MESOS-2840) work, Mesos containerizer is going to support Docker images as well.
>
> cc Tim Chen
>
> - Jie
>
>
> On Fri, Aug 14, 2015 at 11:39 AM, Kevin Sweeney <ks...@twitter.com.invalid> wrote:
> (cross-posting for a wider audience)
>
> Hi folks,
>
> With mesos-0.23.0 it looks like a new dependency made it in for TLS
> support. While this is fine in theory it actually makes the Docker
> Containerizer story very difficult to reason about.
>
> Here's the situation: Aurora uses a custom Python executor for its tasks.
> Because of the design of the containerizer the executor runs in the context
> of the container. This means that the container has to be able to run the
> executor (in practice this means it needs a python2.7 installation and some
> shared libraries libmesos links to). Since most containers don't contain
> Aurora's executor, Aurora hacked around this by using the executor in
> $MESOS_SANDBOX plus a requirement that hosted containers be able to run the
> executor. However, with the upgrade to mesos-0.23.0 the containers that
> could run the 0.22.0 executor no longer work due to the new dependency on
> libcurl-nss.
>
> This is not a problem limited to Docker - I don't see how this design will
> work with *any* container runtime - we can never upgrade the executor
> without upgrading all the containers to contain its new dependencies, which
> at a minimum means we must rebuild them whenever mesos gains a new
> dependency.
>
> Does anyone with more experience with these APIs have a suggestion here? It
> seems we need to make the executor run in the context of the host OS and
> aware of the container (or maybe we have mesos launch a container with the
> executor+its dependencies and have it launch a child container).
> --
> Kevin Sweeney
> @kts
>
Re: Improvements to container support with custom executor
Posted by Steven Schlansker <ss...@opentable.com>.
To be clear, will this Unified Containerizer give us the ability to run Docker images *without* the Docker daemon?
That would be fantastic! This would remove one of the least reliable components of the ecosystem (the Docker daemon) while preserving the illusion of Docker to our end users.
On Aug 14, 2015, at 12:17 PM, Jie Yu <yu...@gmail.com> wrote:
> Kevin,
>
> Thanks for bringing this up! This question is related to the work we are currently doing for Filesystem Isolation (MESOS-2386) and Unified Container (MESOS-2840).
>
> In fact, if you think about using Mesos containerizer and want to run a task (using command executor) under a specified image, we have the same issue for Mesos' own command executor (MESOS-3004).
>
> I've proposed a solution in this doc. The main idea is that we allow the custom executor to still run under the host filesystem (so that it does not have to deal with dependency issues). Mesos will provision the image (the image requested by the user), and mount in as a Volume in the container. The custom executor is responsible for chrooting (or pivot_root if it has root privilege) into the user specified image before execing the user process.
>
> The above proposal is for only Mesos containerizer currently. But with the Unified Container (MESOS-2840) work, Mesos containerizer is going to support Docker images as well.
>
> cc Tim Chen
>
> - Jie
>
>
> On Fri, Aug 14, 2015 at 11:39 AM, Kevin Sweeney <ks...@twitter.com.invalid> wrote:
> (cross-posting for a wider audience)
>
> Hi folks,
>
> With mesos-0.23.0 it looks like a new dependency made it in for TLS
> support. While this is fine in theory it actually makes the Docker
> Containerizer story very difficult to reason about.
>
> Here's the situation: Aurora uses a custom Python executor for its tasks.
> Because of the design of the containerizer the executor runs in the context
> of the container. This means that the container has to be able to run the
> executor (in practice this means it needs a python2.7 installation and some
> shared libraries libmesos links to). Since most containers don't contain
> Aurora's executor, Aurora hacked around this by using the executor in
> $MESOS_SANDBOX plus a requirement that hosted containers be able to run the
> executor. However, with the upgrade to mesos-0.23.0 the containers that
> could run the 0.22.0 executor no longer work due to the new dependency on
> libcurl-nss.
>
> This is not a problem limited to Docker - I don't see how this design will
> work with *any* container runtime - we can never upgrade the executor
> without upgrading all the containers to contain its new dependencies, which
> at a minimum means we must rebuild them whenever mesos gains a new
> dependency.
>
> Does anyone with more experience with these APIs have a suggestion here? It
> seems we need to make the executor run in the context of the host OS and
> aware of the container (or maybe we have mesos launch a container with the
> executor+its dependencies and have it launch a child container).
> --
> Kevin Sweeney
> @kts
>
Re: Improvements to container support with custom executor
Posted by Jie Yu <yu...@gmail.com>.
Kevin,
Thanks for bringing this up! This question is related to the work we are
currently doing for Filesystem Isolation (MESOS-2386
<https://issues.apache.org/jira/browse/MESOS-2386>) and Unified Container (
MESOS-2840 <https://issues.apache.org/jira/browse/MESOS-2840>).
In fact, if you think about using Mesos containerizer and want to run a
task (using command executor) under a specified image, we have the same
issue for Mesos' own command executor (MESOS-3004
<https://issues.apache.org/jira/browse/MESOS-3004>).
I've proposed a solution in this doc
<https://docs.google.com/document/d/16hyLVRL0nz-KBts1J5stGyxZPniFPbPbs7R-ZRQVCH4/edit?usp=sharing>.
The main idea is that we allow the custom executor to still run under the
host filesystem (so that it does not have to deal with dependency issues).
Mesos will provision the image (the image requested by the user), and mount
in as a Volume in the container. The custom executor is responsible for
chrooting (or pivot_root if it has root privilege) into the user specified
image before execing the user process.
The above proposal is for only Mesos containerizer currently. But with the
Unified Container (MESOS-2840
<https://issues.apache.org/jira/browse/MESOS-2840>) work, Mesos
containerizer is going to support Docker images as well.
cc Tim Chen
- Jie
On Fri, Aug 14, 2015 at 11:39 AM, Kevin Sweeney <
ksweeney@twitter.com.invalid> wrote:
> (cross-posting for a wider audience)
>
> Hi folks,
>
> With mesos-0.23.0 it looks like a new dependency made it in for TLS
> support. While this is fine in theory it actually makes the Docker
> Containerizer story very difficult to reason about.
>
> Here's the situation: Aurora uses a custom Python executor for its tasks.
> Because of the design of the containerizer the executor runs in the context
> of the container. This means that the container has to be able to run the
> executor (in practice this means it needs a python2.7 installation and some
> shared libraries libmesos links to). Since most containers don't contain
> Aurora's executor, Aurora hacked around this by using the executor in
> $MESOS_SANDBOX plus a requirement that hosted containers be able to run the
> executor. However, with the upgrade to mesos-0.23.0 the containers that
> could run the 0.22.0 executor no longer work due to the new dependency on
> libcurl-nss.
>
> This is not a problem limited to Docker - I don't see how this design will
> work with *any* container runtime - we can never upgrade the executor
> without upgrading all the containers to contain its new dependencies, which
> at a minimum means we must rebuild them whenever mesos gains a new
> dependency.
>
> Does anyone with more experience with these APIs have a suggestion here? It
> seems we need to make the executor run in the context of the host OS and
> aware of the container (or maybe we have mesos launch a container with the
> executor+its dependencies and have it launch a child container).
> --
> Kevin Sweeney
> @kts
>
Re: Improvements to container support with custom executor
Posted by Jie Yu <yu...@gmail.com>.
Kevin,
Thanks for bringing this up! This question is related to the work we are
currently doing for Filesystem Isolation (MESOS-2386
<https://issues.apache.org/jira/browse/MESOS-2386>) and Unified Container (
MESOS-2840 <https://issues.apache.org/jira/browse/MESOS-2840>).
In fact, if you think about using Mesos containerizer and want to run a
task (using command executor) under a specified image, we have the same
issue for Mesos' own command executor (MESOS-3004
<https://issues.apache.org/jira/browse/MESOS-3004>).
I've proposed a solution in this doc
<https://docs.google.com/document/d/16hyLVRL0nz-KBts1J5stGyxZPniFPbPbs7R-ZRQVCH4/edit?usp=sharing>.
The main idea is that we allow the custom executor to still run under the
host filesystem (so that it does not have to deal with dependency issues).
Mesos will provision the image (the image requested by the user), and mount
in as a Volume in the container. The custom executor is responsible for
chrooting (or pivot_root if it has root privilege) into the user specified
image before execing the user process.
The above proposal is for only Mesos containerizer currently. But with the
Unified Container (MESOS-2840
<https://issues.apache.org/jira/browse/MESOS-2840>) work, Mesos
containerizer is going to support Docker images as well.
cc Tim Chen
- Jie
On Fri, Aug 14, 2015 at 11:39 AM, Kevin Sweeney <
ksweeney@twitter.com.invalid> wrote:
> (cross-posting for a wider audience)
>
> Hi folks,
>
> With mesos-0.23.0 it looks like a new dependency made it in for TLS
> support. While this is fine in theory it actually makes the Docker
> Containerizer story very difficult to reason about.
>
> Here's the situation: Aurora uses a custom Python executor for its tasks.
> Because of the design of the containerizer the executor runs in the context
> of the container. This means that the container has to be able to run the
> executor (in practice this means it needs a python2.7 installation and some
> shared libraries libmesos links to). Since most containers don't contain
> Aurora's executor, Aurora hacked around this by using the executor in
> $MESOS_SANDBOX plus a requirement that hosted containers be able to run the
> executor. However, with the upgrade to mesos-0.23.0 the containers that
> could run the 0.22.0 executor no longer work due to the new dependency on
> libcurl-nss.
>
> This is not a problem limited to Docker - I don't see how this design will
> work with *any* container runtime - we can never upgrade the executor
> without upgrading all the containers to contain its new dependencies, which
> at a minimum means we must rebuild them whenever mesos gains a new
> dependency.
>
> Does anyone with more experience with these APIs have a suggestion here? It
> seems we need to make the executor run in the context of the host OS and
> aware of the container (or maybe we have mesos launch a container with the
> executor+its dependencies and have it launch a child container).
> --
> Kevin Sweeney
> @kts
>