You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@mesos.apache.org by Hendrik Haddorp <he...@gmx.net> on 2016/06/23 14:56:19 UTC
Mesos persistent volumes as Docker volumes
Hi,
I'm trying to write a Mesos framework that should create persistent
volumes and then start a Docker container that uses this. So far I was
able to dynamically reserve resources (cpu, memory and disk) and create
a persistent volume in the reserved disk space. I'm also able to launch
a Docker container. I just can't figure out how to connect these
correctly. I either get told that some fields are not set correctly, end
up with mounting a path on the host system or nothing at all seems to
happen.
Would be nice if somebody could show how a TaskInfo protobuf would need
to get filled to achieve this.
thanks,
Hendrik
Re: Mesos persistent volumes as Docker volumes
Posted by Jie Yu <yu...@gmail.com>.
Hendrik,
Sorry about the late response on that. I am glad that you figured it out
yourself.
Currently, for local persistent volumes (consumes 'disk' resources on the
agent), the container_path has to be relative. We made that decision based
on a couple of reasons:
1) not all the systems Mesos supports has bind mount (e.g., osx, windows).
Because of that, we don't want to introduce an API that works on some
system, but not on others. A relative container_path (e.g., "abc") means
that the volume can be accessed at $MESOS_SANDBOX/abc. This can be
supported on all systems (i.e., using bind mount on Linux, symlinks on
others).
2) Mesos allows the resources of a container to expand when executor
receives a new task from the framework. The new task might contains a local
persistent volume. Dynamically adding volumes to a running container is not
supported by Docker daemon until recently. Even though, one needs to setup
mount propagation properly before hand. Supporting an absolute
container_path will make it very hard to implement (you'll need to make
sure the parent mount of `container_path` is a shared mount). To solve that
issue, we only set mount propagation for mesos sandbox directory. That's
the reason why we enforce that container_path has to be relative for local
persistent volumes.
The workaround you have is perfectly fine for command tasks. We should
definitely document this. I'll make sure that we follow up on that.
Thanks!
- Jie
On Fri, Jun 24, 2016 at 1:25 PM, Hendrik Haddorp <he...@gmx.net>
wrote:
> Basically the same issue was actually reported recently in MESOS-3413
> [1]. The discussion in there resulted in a code change in the Mesos
> framework for ArangoDB [2]. One first has to create a volume with some
> container path and then add a volume to the ContainerInfo when launching
> the docker container that uses the same path as host path for the
> volume. Now the container path needs to be absolute, like /data. Files
> that are now being created below /data show up in the persistent volume
> and newly launched containers with the same mapping can reuse the data.
> Not that obvious in my opinion but it works :-)
>
> [1] https://issues.apache.org/jira/browse/MESOS-3413
> [2]
>
> https://github.com/arangodb/arangodb-mesos-framework/commit/98ccbdbaa5ae41f83b02ca42e7325746ad044099
>
> p.s.: looks like jira is currently down ...
>
> On 23/06/16 16:56, Hendrik Haddorp wrote:
> > Hi,
> >
> > I'm trying to write a Mesos framework that should create persistent
> > volumes and then start a Docker container that uses this. So far I was
> > able to dynamically reserve resources (cpu, memory and disk) and create
> > a persistent volume in the reserved disk space. I'm also able to launch
> > a Docker container. I just can't figure out how to connect these
> > correctly. I either get told that some fields are not set correctly, end
> > up with mounting a path on the host system or nothing at all seems to
> > happen.
> >
> > Would be nice if somebody could show how a TaskInfo protobuf would need
> > to get filled to achieve this.
> >
> > thanks,
> > Hendrik
>
>
Re: Mesos persistent volumes as Docker volumes
Posted by Hendrik Haddorp <he...@gmx.net>.
Basically the same issue was actually reported recently in MESOS-3413
[1]. The discussion in there resulted in a code change in the Mesos
framework for ArangoDB [2]. One first has to create a volume with some
container path and then add a volume to the ContainerInfo when launching
the docker container that uses the same path as host path for the
volume. Now the container path needs to be absolute, like /data. Files
that are now being created below /data show up in the persistent volume
and newly launched containers with the same mapping can reuse the data.
Not that obvious in my opinion but it works :-)
[1] https://issues.apache.org/jira/browse/MESOS-3413
[2]
https://github.com/arangodb/arangodb-mesos-framework/commit/98ccbdbaa5ae41f83b02ca42e7325746ad044099
p.s.: looks like jira is currently down ...
On 23/06/16 16:56, Hendrik Haddorp wrote:
> Hi,
>
> I'm trying to write a Mesos framework that should create persistent
> volumes and then start a Docker container that uses this. So far I was
> able to dynamically reserve resources (cpu, memory and disk) and create
> a persistent volume in the reserved disk space. I'm also able to launch
> a Docker container. I just can't figure out how to connect these
> correctly. I either get told that some fields are not set correctly, end
> up with mounting a path on the host system or nothing at all seems to
> happen.
>
> Would be nice if somebody could show how a TaskInfo protobuf would need
> to get filled to achieve this.
>
> thanks,
> Hendrik
Re: Mesos persistent volumes as Docker volumes
Posted by Hendrik Haddorp <he...@gmx.net>.
Hi Guangya,
that seems to be pretty much the same that Vaibhav pointed me to, that
is using Docker volume drivers to mount external storage. I would like
to leverage the build in Mesos persistent volumes but mount them at any
position, just like you can with normal docker volumes from the host
filesystem.
On 23/06/16 23:41, Guangya Liu wrote:
> Hi Hendrik,
>
> You can take a look for how Mesos 1.0 support docker volume driver
> integration with Mesos Containerizer from
> here https://github.com/apache/mesos/blob/master/docs/docker-volume.md
>
> Both Mesos Containerizer and Docker Containerizer support integration
> with docker volume driver now, you can take a look
> at https://reviews.apache.org/r/36440/ for how to test docker volume
> driver with Docker Containerizer.
>
> Thanks,
>
> Guangya
>
> On Fri, Jun 24, 2016 at 2:54 AM, Hendrik Haddorp
> <hendrik.haddorp@gmx.net <ma...@gmx.net>> wrote:
>
> Thanks for the tip, I did actually notice the project when trying
> to find a solution for my problem. This project seems to be about
> leveraging external Docker volume drivers, which is certainly also
> interesting but I'm trying to use the build in Mesos persistent
> storage.
>
> Actually I just noticed that I can mount the volumes that I
> created with Mesos it is just that when I specify a relative path
> for the container path in the volume the mount shows up below
> /mnt/mesos/sandbox and when I specify an absolute path it is being
> ignored as according to the Mesos log slashes are not allowed in
> the container path. So now problem is that I would like to have
> the mount at a different location.
>
>
> On 23/06/16 20:43, Vaibhav Khanduja wrote:
>> Hi Hendrik,
>>
>> If you want to run Docker jobs, it may be a good idea to get
>> volumes from “Docker” volume plugin.
>>
>> There is a project by EMC - mesos-dvdi, which abstracts the
>> volume creation. Please check this out and it should work with
>> your scheduler …
>>
>>
>> https://github.com/emccode/mesos-module-dvdi
>>
>> Thx
>>
>>
>>
>> On Thu, Jun 23, 2016 at 7:56 AM, Hendrik Haddorp
>> <hendrik.haddorp@gmx.net <ma...@gmx.net>> wrote:
>>
>> Hi,
>>
>> I'm trying to write a Mesos framework that should create
>> persistent
>> volumes and then start a Docker container that uses this. So
>> far I was
>> able to dynamically reserve resources (cpu, memory and disk)
>> and create
>> a persistent volume in the reserved disk space. I'm also able
>> to launch
>> a Docker container. I just can't figure out how to connect these
>> correctly. I either get told that some fields are not set
>> correctly, end
>> up with mounting a path on the host system or nothing at all
>> seems to
>> happen.
>>
>> Would be nice if somebody could show how a TaskInfo protobuf
>> would need
>> to get filled to achieve this.
>>
>> thanks,
>> Hendrik
>>
>>
>
>
Re: Mesos persistent volumes as Docker volumes
Posted by Guangya Liu <gy...@gmail.com>.
Hi Hendrik,
You can take a look for how Mesos 1.0 support docker volume driver
integration with Mesos Containerizer from here
https://github.com/apache/mesos/blob/master/docs/docker-volume.md
Both Mesos Containerizer and Docker Containerizer support integration with
docker volume driver now, you can take a look at
https://reviews.apache.org/r/36440/ for how to test docker volume driver
with Docker Containerizer.
Thanks,
Guangya
On Fri, Jun 24, 2016 at 2:54 AM, Hendrik Haddorp <he...@gmx.net>
wrote:
> Thanks for the tip, I did actually notice the project when trying to find
> a solution for my problem. This project seems to be about leveraging
> external Docker volume drivers, which is certainly also interesting but I'm
> trying to use the build in Mesos persistent storage.
>
> Actually I just noticed that I can mount the volumes that I created with
> Mesos it is just that when I specify a relative path for the container path
> in the volume the mount shows up below /mnt/mesos/sandbox and when I
> specify an absolute path it is being ignored as according to the Mesos log
> slashes are not allowed in the container path. So now problem is that I
> would like to have the mount at a different location.
>
>
> On 23/06/16 20:43, Vaibhav Khanduja wrote:
>
> Hi Hendrik,
>
> If you want to run Docker jobs, it may be a good idea to get volumes from
> “Docker” volume plugin.
>
> There is a project by EMC - mesos-dvdi, which abstracts the volume
> creation. Please check this out and it should work with your scheduler …
>
>
> https://github.com/emccode/mesos-module-dvdi
>
> Thx
>
>
>
> On Thu, Jun 23, 2016 at 7:56 AM, Hendrik Haddorp <he...@gmx.net>
> wrote:
>
>> Hi,
>>
>> I'm trying to write a Mesos framework that should create persistent
>> volumes and then start a Docker container that uses this. So far I was
>> able to dynamically reserve resources (cpu, memory and disk) and create
>> a persistent volume in the reserved disk space. I'm also able to launch
>> a Docker container. I just can't figure out how to connect these
>> correctly. I either get told that some fields are not set correctly, end
>> up with mounting a path on the host system or nothing at all seems to
>> happen.
>>
>> Would be nice if somebody could show how a TaskInfo protobuf would need
>> to get filled to achieve this.
>>
>> thanks,
>> Hendrik
>>
>
>
>
Re: Mesos persistent volumes as Docker volumes
Posted by Hendrik Haddorp <he...@gmx.net>.
Thanks for the tip, I did actually notice the project when trying to
find a solution for my problem. This project seems to be about
leveraging external Docker volume drivers, which is certainly also
interesting but I'm trying to use the build in Mesos persistent storage.
Actually I just noticed that I can mount the volumes that I created with
Mesos it is just that when I specify a relative path for the container
path in the volume the mount shows up below /mnt/mesos/sandbox and when
I specify an absolute path it is being ignored as according to the Mesos
log slashes are not allowed in the container path. So now problem is
that I would like to have the mount at a different location.
On 23/06/16 20:43, Vaibhav Khanduja wrote:
> Hi Hendrik,
>
> If you want to run Docker jobs, it may be a good idea to get volumes
> from “Docker” volume plugin.
>
> There is a project by EMC - mesos-dvdi, which abstracts the volume
> creation. Please check this out and it should work with your scheduler …
>
>
> https://github.com/emccode/mesos-module-dvdi
>
> Thx
>
>
>
> On Thu, Jun 23, 2016 at 7:56 AM, Hendrik Haddorp
> <hendrik.haddorp@gmx.net <ma...@gmx.net>> wrote:
>
> Hi,
>
> I'm trying to write a Mesos framework that should create persistent
> volumes and then start a Docker container that uses this. So far I was
> able to dynamically reserve resources (cpu, memory and disk) and
> create
> a persistent volume in the reserved disk space. I'm also able to
> launch
> a Docker container. I just can't figure out how to connect these
> correctly. I either get told that some fields are not set
> correctly, end
> up with mounting a path on the host system or nothing at all seems to
> happen.
>
> Would be nice if somebody could show how a TaskInfo protobuf would
> need
> to get filled to achieve this.
>
> thanks,
> Hendrik
>
>
Re: Mesos persistent volumes as Docker volumes
Posted by Vaibhav Khanduja <va...@gmail.com>.
Hi Hendrik,
If you want to run Docker jobs, it may be a good idea to get volumes from
“Docker” volume plugin.
There is a project by EMC - mesos-dvdi, which abstracts the volume
creation. Please check this out and it should work with your scheduler …
https://github.com/emccode/mesos-module-dvdi
Thx
<https://github.com/emccode/mesos-module-dvdi>
<https://github.com/emccode/mesos-module-dvdi>
<https://github.com/emccode/mesos-module-dvdi>
<https://github.com/emccode/mesos-module-dvdi>
On Thu, Jun 23, 2016 at 7:56 AM, Hendrik Haddorp <he...@gmx.net>
wrote:
> Hi,
>
> I'm trying to write a Mesos framework that should create persistent
> volumes and then start a Docker container that uses this. So far I was
> able to dynamically reserve resources (cpu, memory and disk) and create
> a persistent volume in the reserved disk space. I'm also able to launch
> a Docker container. I just can't figure out how to connect these
> correctly. I either get told that some fields are not set correctly, end
> up with mounting a path on the host system or nothing at all seems to
> happen.
>
> Would be nice if somebody could show how a TaskInfo protobuf would need
> to get filled to achieve this.
>
> thanks,
> Hendrik
>