You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mesos.apache.org by Olivier Sallou <ol...@irisa.fr> on 2016/05/18 10:24:00 UTC
volume / mount point error with Unified Containerizer
Hi,
I am trying unified containerizer on a single server (master/slave) on
mesos 0.28.1, to switch from docker containerizer to mesos+docker image
container.
I have setup slave config as suggested in documentation:
containerizers=docker,mesos
image_providers=docker \
isolation=filesystem/linux,docker/runtime
However, when I execute my task with a volume I have an error:
....
+ mount -n --rbind
/tmp/mesos/provisioner/containers/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/backends/copy/rootfses/f9f66bb2-308d-4555-ba77-49ec61cbeb4f
/tmp/mesos/slaves/2a296daf-7419-4659-ade1-763c792cd522-S0/frameworks/aef1b0e3-ea2d-4770-baac-96d673ab88f9-0000/executors/51/runs/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/.rootfs
+ mount -n --rbind
/home/osallou/Development/NOSAVE/go-docker/godshared/tasks/pairtree_root/us/er/_o/sa/ll/ou/task
/tmp/mesos/slaves/2a296daf-7419-4659-ade1-763c792cd522-S0/frameworks/aef1b0e3-ea2d-4770-baac-96d673ab88f9-0000/executors/51/runs/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/.rootfs/mnt/god-data
mount: mount point
/tmp/mesos/slaves/2a296daf-7419-4659-ade1-763c792cd522-S0/frameworks/aef1b0e3-ea2d-4770-baac-96d673ab88f9-0000/executors/51/runs/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/.rootfs/mnt/god-data
does not exist
Failed to execute a preparation shell command
Then, my task switches to FAILED.
I define a local volume to bind mount in my "container"
/home/osallou/Development/NOSAVE/go-docker/godshared/tasks/pairtree_root/us/er/_o/sa/ll/ou/task
=> /mnt/god-data
My directory exists on local server.
In mesos UI, I can see the .rootfs directory along stdout and stderr
files, and inside .rootfs, I can see /mnt/god-data (empty).
Running the same using Docker containerizer instead of mesos
containerizer (with a Docker image) works fine.
It seems it fails to mount my local directory in the container. Any idea
of what is going wrong or how to debug this?
Thanks
--
Olivier Sallou
IRISA / University of Rennes 1
Campus de Beaulieu, 35000 RENNES - FRANCE
Tel: 02.99.84.71.95
gpg key id: 4096R/326D8438 (keyring.debian.org)
Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D 8438
Re: volume / mount point error with Unified Containerizer
Posted by Guangya Liu <gy...@gmail.com>.
Hi Zhitao,
This issue was already fixed in 0.28.x and upper version. I have a try with
both 0.28.x and master branch, it works well for Olivier's case.
Thanks,
Guangya
On Tue, May 24, 2016 at 11:58 PM, Zhitao Li <zh...@gmail.com> wrote:
> Guangya,
>
> Is there a JIRA work board or document to track what's being fixed, working
> on or planned in 0.29 and further releases?
>
> On Tue, May 24, 2016 at 1:51 AM, Guangya Liu <gy...@gmail.com> wrote:
>
> > Olivier,
> >
> > If it is not urgent, I think that you can use 0.29 in next week, the
> > unified container will have more features in 0.29 such as external
> storage
> > and network integration.
> >
> > Thanks,
> >
> > Guangya
> >
> > On Tue, May 24, 2016 at 2:17 AM, Olivier Sallou <olivier.sallou@irisa.fr
> >
> > wrote:
> >
> > >
> > >
> > > ----- Mail original -----
> > > > De: "Guangya Liu" <gy...@gmail.com>
> > > > À: "dev" <de...@mesos.apache.org>, "Jie Yu" <ji...@mesosphere.io>,
> > "Gilbert
> > > Song" <gi...@mesosphere.io>
> > > > Envoyé: Lundi 23 Mai 2016 17:34:41
> > > > Objet: Re: volume / mount point error with Unified Containerizer
> > > >
> > > > It is a bit strange to me, I also did some test and review code for
> > > > relative path, and found that relative path works well.
> > > >
> > > > In 0.28.1, if deploy a docker container with MesosContaineirizer,
> then
> > if
> > > > using absolute path as continer_path, the mesos agent will update the
> > > > container_path to a relative path by adding a prefix ./rootfs to the
> > > > container_path, e.g. /file/path = > ./rootfs/file/path.
> > > >
> > > > If deploy a docker container with MesosContaineirizer with relative
> > path
> > > as
> > > > container_path, then the mesos agent will not update the
> > container_path.
> > > >
> > > > So the final mount point for the container should be either
> > > >
> > > > 1) /tmp/mesos/slaves/agent_id/frameworks/framework_id/
> > > > executors/51/runs/container_id/.rootfs/file/path
> > > > 2)
> > /tmp/mesos/slaves/agent_id/frameworks/framework_id/executors/51/runs/
> > > > container_id/file/path
> > > >
> > > > The only difference is adding ./rootfs as a prefix or not, the test
> > > result
> > > > is that 1) does not work and 2) works well. And even the mount for 1)
> > > > failed, but I can see the mount point path does exist.
> > > >
> > >
> > > @Guangya
> > > I confirm that using relative path works fine, I get volumes in mesos
> > path
> > > (but it does not help for my implementation).
> > > If I use the Docker containerizer, absolute paths are fine, this is
> what
> > I
> > > use for the moment in my code, and am investigating to switch to
> unified
> > > container.
> > >
> > >
> > > > @Yu Jie and @Gilbert, any comments for this?
> > > >
> > > > @Oilivier,
> > > >
> > > > In order not to block your test, can you please use mesos after
> 0.28.1?
> > > You
> > > > can use either 0.28.2 or above version.
> > >
> > > Well, as this is not an urgent matter, I am waiting 0.29 to test
> against
> > > this release (with other features I am waiting for).
> > >
> > > >
> > > > Thanks,
> > > >
> > > > Guangya
> > > >
> > > >
> > > > On Mon, May 23, 2016 at 10:30 PM, Guangya Liu <gy...@gmail.com>
> > > wrote:
> > > >
> > > > > Thanks Olivier, I can reproduce this issue now and still checking
> > what
> > > is
> > > > > wrong.
> > > > >
> > > > > What I did is as following:
> > > > > 1) Check out code with tag of 0.28.1
> > > > > 2) update mesos-execute to add a host path volume
> > > > > diff --git a/src/cli/execute.cpp b/src/cli/execute.cpp
> > > > > index 81a0388..0ff913c 100644
> > > > > --- a/src/cli/execute.cpp
> > > > > +++ b/src/cli/execute.cpp
> > > > > @@ -72,6 +72,8 @@ using mesos::v1::TaskID;
> > > > > using mesos::v1::TaskInfo;
> > > > > using mesos::v1::TaskState;
> > > > > using mesos::v1::TaskStatus;
> > > > > +using mesos::v1::Volume;
> > > > > +using mesos::v1::Parameters;
> > > > >
> > > > > using mesos::v1::scheduler::Call;
> > > > > using mesos::v1::scheduler::Event;
> > > > > @@ -572,6 +574,12 @@ private:
> > > > > }
> > > > > }
> > > > >
> > > > > + Volume* volume1 = containerInfo.add_volumes();
> > > > > + volume1->set_container_path("/tmp/abcd");
> > > > > + volume1->set_mode(Volume::RW);
> > > > > + volume1->set_host_path("/root/convoy");
> > > > > + cout << "Add Voume 1" << endl;
> > > > > +
> > > > > return containerInfo;
> > > > > } else if (containerizer == "docker") {
> > > > > // 'docker' containerizer only supports 'docker' images.
> > > > > 3) launch a task with docker image, task failed.
> > > > >
> > > > > 4) Check sandbox:
> > > > > + /root/src/mesos/m1/mesos/build/src/mesos-containerizer mount
> > > > > --help=false --operation=make-rslave --path=/
> > > > > + grep -E /tmp/mesos/.+ /proc/self/mountinfo
> > > > > + grep -v 3239aafc-78d8-4f70-81e5-f32fb3733339
> > > > > + cut+ -d -f5
> > > > > xargs --no-run-if-empty umount -l
> > > > > + mount -n --rbind
> > > > >
> > >
> >
> /tmp/mesos/provisioner/containers/3239aafc-78d8-4f70-81e5-f32fb3733339/backends/copy/rootfses/5e8bf3fa-53b1-4bd5-bb3d-525ddc7900b6
> > > > >
> > >
> >
> /tmp/mesos/slaves/a4294ed5-10e8-47db-a3b9-a43a4f951374-S0/frameworks/a4294ed5-10e8-47db-a3b9-a43a4f951374-0000/executors/test/runs/3239aafc-78d8-4f70-81e5-f32fb3733339/.rootfs
> > > > > + mount -n --rbind /root/convoy
> > > > >
> > >
> >
> /tmp/mesos/slaves/a4294ed5-10e8-47db-a3b9-a43a4f951374-S0/frameworks/a4294ed5-10e8-47db-a3b9-a43a4f951374-0000/executors/test/runs/3239aafc-78d8-4f70-81e5-f32fb3733339/.rootfs/tmp/abcd
> > > > > mount: mount point
> > > > >
> > >
> >
> /tmp/mesos/slaves/a4294ed5-10e8-47db-a3b9-a43a4f951374-S0/frameworks/a4294ed5-10e8-47db-a3b9-a43a4f951374-0000/executors/test/runs/3239aafc-78d8-4f70-81e5-f32fb3733339/.rootfs/tmp/abcd
> > > > > does not exist
> > > > > Failed to execute a preparation shell command
> > > > >
> > > > > Will check more for this.
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Guangya
> > > > >
> > > > > On Mon, May 23, 2016 at 3:35 PM, Olivier Sallou <
> > > olivier.sallou@irisa.fr>
> > > > > wrote:
> > > > >
> > > > >>
> > > > >>
> > > > >> On 05/23/2016 09:33 AM, Olivier Sallou wrote:
> > > > >> >
> > > > >> > On 05/20/2016 03:26 PM, Guangya Liu wrote:
> > > > >> >> Since you are using docker image which means that your
> container
> > > will
> > > > >> have
> > > > >> >> rootfs, so it is not required to have the absolute path exist,
> > the
> > > > >> linux
> > > > >> >> file system isolator will help create the path automatically
> > > > >> >>
> > > > >>
> > >
> >
> https://github.com/apache/mesos/blob/0.28.x/src/slave/containerizer/mesos/isolators/filesystem/linux.cpp#L390-L402
> > > > >> >>
> > > > >> >> Can you please share your framework? How did you set the volume
> > > part in
> > > > >> >> your framework?
> > > > >> > @Guangya
> > > > >> >
> > > > >> > I use Python API.
> > > > >> >
> > > > >> > Here is related code:
> > > > >> >
> > > > >> > ....
> > > > >> > # Define container volumes
> > > > >> > for v in job['container']['volumes']:
> > > > >> > volume = container.volumes.add()
> > > > >> > volume.container_path = v['mount']
> > > > >> > volume.host_path = v['path']
> > > > >> > if v['acl'] == 'rw':
> > > > >> > volume.mode = 1 # mesos_pb2.Volume.Mode.RW
> > > > >> > else:
> > > > >> > volume.mode = 2 # mesos_pb2.Volume.Mode.RO
> > > > >> >
> > > > >> > => In my test case, I add 2 volumes from a host shared
> > > directory,
> > > > >> > mounted in container as /mnt/go-docker and /mnt/god-data.
> > > > >> >
> > > > >> > ...
> > > > >> > # Define docker image and network
> > > > >> > docker = mesos_pb2.ContainerInfo.MesosInfo()
> > > > >> > docker.image.type = 2 # Docker
> > > > >> > docker.image.docker.name ='centos:latest'
> > > > >> > # Request an IP from a network module
> > > > >> > network_info = container.network_infos.add()
> > > > >> > network_info_name = 'sampletest'
> > > > >> > # Get an IP V4 address
> > > > >> > ip_address = network_info.ip_addresses.add()
> > > > >> > ip_address.protocol = 1
> > > > >> > # The network group to join
> > > > >> > group =
> network_info.groups.append(network_info_name)
> > > > >> > port_list = [22]
> > > > >> > if port_list:
> > > > >> > for port in port_list:
> > > > >> >
> > job['container']['port_mapping'].append({'host':
> > > > >> > port, 'container': port})
> > > > >> > container.mesos.MergeFrom(docker)
> > > > >> >
> > > > >> > It results in error message:
> > > > >> >
> > > > >> > + mount -n --rbind
> > > > >> >
> > > > >>
> > >
> >
> /tmp/mesos/provisioner/containers/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/backends/copy/rootfses/f9f66bb2-308d-4555-ba77-49ec61cbeb4f
> > > > >> >
> > > > >>
> > >
> >
> /tmp/mesos/slaves/2a296daf-7419-4659-ade1-763c792cd522-S0/frameworks/aef1b0e3-ea2d-4770-baac-96d673ab88f9-0000/executors/51/runs/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/.rootfs
> > > > >> > + mount -n --rbind
> > > > >> >
> > > > >>
> > >
> >
> /home/osallou/Development/NOSAVE/go-docker/godshared/tasks/pairtree_root/us/er/_o/sa/ll/ou/task
> > > > >> >
> > > > >>
> > >
> >
> /tmp/mesos/slaves/2a296daf-7419-4659-ade1-763c792cd522-S0/frameworks/aef1b0e3-ea2d-4770-baac-96d673ab88f9-0000/executors/51/runs/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/.rootfs/mnt/god-data
> > > > >> > mount: mount point
> > > > >> >
> > > > >>
> > >
> >
> /tmp/mesos/slaves/2a296daf-7419-4659-ade1-763c792cd522-S0/frameworks/aef1b0e3-ea2d-4770-baac-96d673ab88f9-0000/executors/51/runs/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/.rootfs/mnt/god-data
> > > > >> > does not exist
> > > > >> > Failed to execute a preparation shell command
> > > > >> >
> > > > >> >
> > > > >> > We can see the .rootfs, but mnt/god-data under .rootfs fails.
> > Local
> > > > >> > directory exists, it does not pre-exists in container. What is
> > > strange
> > > > >> > is , if I look in mesos task dir, .rootfs/mnt/go-data, directory
> > is
> > > > >> present.
> > > > >> >
> > > > >> > Or, is the error message (.rootfs/mnt/god-data does not exist)
> > > simply a
> > > > >> > warning, and it creates it, and final error (Failed to execute a
> > > > >> > preparation shell command) not related (and unclear...)
> > > > >> Additional info: command to execute in container is located in one
> > of
> > > > >> mounted volume.
> > > > >> >
> > > > >> > Olivier
> > > > >> >> Thanks,
> > > > >> >>
> > > > >> >> Guangya
> > > > >> >>
> > > > >> >> On Fri, May 20, 2016 at 4:54 AM, Olivier Sallou <
> > > > >> olivier.sallou@irisa.fr>
> > > > >> >> wrote:
> > > > >> >>
> > > > >> >>> ----- Mail original -----
> > > > >> >>>> De: "Gilbert Song" <gi...@mesosphere.io>
> > > > >> >>>> À: "dev" <de...@mesos.apache.org>
> > > > >> >>>> Envoyé: Jeudi 19 Mai 2016 01:57:16
> > > > >> >>>> Objet: Re: volume / mount point error with Unified
> > Containerizer
> > > > >> >>>>
> > > > >> >>>> @Olivier,
> > > > >> >>>> In mesos 0.28.1, you are supposed to be able bind mount a
> > volume
> > > from
> > > > >> >>>> the host into the mesos container. Did you specify a docker
> > > image (we
> > > > >> >>>> determine
> > > > >> >>>> the mount point differently depending whether the container
> > has a
> > > > >> >>> rootfs)?
> > > > >> >>>
> > > > >> >>> Yes I specified an image, a Docker image URI.
> > > > >> >>>
> > > > >> >>>> How
> > > > >> >>>> do you specify your 'container_path' (the mount point in the
> > > > >> container)?
> > > > >> >>> If
> > > > >> >>>> it is an
> > > > >> >>>> absolute path, we require that dir to be pre-existed. If it
> is
> > a
> > > > >> relative
> > > > >> >>>> path, we will
> > > > >> >>>> mkdir for it.
> > > > >> >>> It is an absolute path, but it does not exists in image (this
> is
> > > the
> > > > >> >>> issue). Images are custom Docker images (images containing
> tools
> > > for
> > > > >> batch
> > > > >> >>> computing), and I want, for example, to mount some shared
> > > resources
> > > > >> (user
> > > > >> >>> home dir, common data, etc.) in the image. Of course those
> > > > >> directories do
> > > > >> >>> not pre-exists in container images as they are specific to the
> > > > >> environment.
> > > > >> >>> Requiring existence of the directory in the image is not issue
> > as
> > > it
> > > > >> >>> prevents using any existing image from a repo.
> > > > >> >>>
> > > > >> >>> When using Docker containerizer it works fine, I can mount any
> > > > >> external
> > > > >> >>> storage in the container.
> > > > >> >>>
> > > > >> >>> Olivie
> > > > >> >>>
> > > > >> >>>
> > > > >> >>>> @Joshua,
> > > > >> >>>> Thank for posting your workaround on mesos. As I mentioned
> > > above, in
> > > > >> >>> 0.28.1
> > > > >> >>>> or
> > > > >> >>>> older, we only mkdir for container_path which is relative
> path
> > > (not
> > > > >> >>>> starting with "/").
> > > > >> >>>> Because if no rootfs specified for a mesos container, the
> > > container
> > > > >> >>> shares
> > > > >> >>>> the host
> > > > >> >>>> root filesystem. Obviously we don't want any random files to
> be
> > > > >> created
> > > > >> >>>> implicitly
> > > > >> >>>> on your host fs.
> > > > >> >>>> From mesos 0.29 (release by the end of this month), we will
> > > mkdir the
> > > > >> >>> mount
> > > > >> >>>> point in the container except for the command task case that
> > > specify
> > > > >> an
> > > > >> >>>> absolute
> > > > >> >>>> container_path without a rootfs. Because we simplify the
> > mounting
> > > > >> logic,
> > > > >> >>> and
> > > > >> >>>> sandbox bind mount will only be done in container mount
> > namespace
> > > > >> >>> instead of
> > > > >> >>>> host mount namespace (what we did before). Please keep tuned.
> > > > >> >>>>
> > > > >> >>>> Cheers,
> > > > >> >>>> Gilbert
> > > > >> >>>>
> > > > >> >>>> On Wed, May 18, 2016 at 8:14 AM, Joshua Cohen <
> > jcohen@apache.org
> > > >
> > > > >> wrote:
> > > > >> >>>>
> > > > >> >>>>> Hi Olivier,
> > > > >> >>>>>
> > > > >> >>>>> I touched on this issue as part of
> > > > >> >>>>> https://issues.apache.org/jira/browse/MESOS-5229. It would
> be
> > > nice
> > > > >> if
> > > > >> >>>>> Mesos
> > > > >> >>>>> automatically created container mount points if they don't
> > > already
> > > > >> >>> exist.
> > > > >> >>>>> In the meantime, as a workaround for this, I've updated my
> > > > >> filesystem
> > > > >> >>>>> images to include the path (e.g. in Dockerfile, add `RUN
> mkdir
> > > -p
> > > > >> >>>>> /some/mount/point`). Not the best solution, but the only
> thing
> > > I've
> > > > >> >>> seen
> > > > >> >>>>> that works at the moment.
> > > > >> >>>>>
> > > > >> >>>>> Cheers,
> > > > >> >>>>>
> > > > >> >>>>> Joshua
> > > > >> >>>>>
> > > > >> >>>>> On Wed, May 18, 2016 at 7:36 AM, Guangya Liu <
> > > gyliu513@gmail.com>
> > > > >> >>> wrote:
> > > > >> >>>>>> It's pretty simple for you from scratch with source code
> > > > >> >>>>>>
> > > > >> >>>>>>
> > > > >> >>>
> > > > >>
> > >
> >
> https://github.com/apache/mesos/blob/master/docs/getting-started.md#building-mesos
> > > > >> >>>>>> ;-)
> > > > >> >>>>>>
> > > > >> >>>>>> Thanks,
> > > > >> >>>>>>
> > > > >> >>>>>> Guangya
> > > > >> >>>>>>
> > > > >> >>>>>> On Wed, May 18, 2016 at 8:30 PM, Olivier Sallou <
> > > > >> >>> olivier.sallou@irisa.fr
> > > > >> >>>>>> wrote:
> > > > >> >>>>>>
> > > > >> >>>>>>> On 05/18/2016 02:31 PM, Guangya Liu wrote:
> > > > >> >>>>>>>> Just saw that you are working with 0.28.1, the "docker
> > volume
> > > > >> >>> driver"
> > > > >> >>>>>>> code
> > > > >> >>>>>>>> was not in 0.28.1, can you please have a try with mesos
> > > master
> > > > >> >>> branch
> > > > >> >>>>>> if
> > > > >> >>>>>>>> you are only doing some test?
> > > > >> >>>>>>> this is indeed test only for the moment. But I will have
> to
> > > > >> >>>>>>> recompile/install mesos :-( (I used packages for
> install).
> > > > >> >>>>>>>
> > > > >> >>>>>>> I will try when possible, but thanks for the hint.
> > > > >> >>>>>>>> Thanks,
> > > > >> >>>>>>>>
> > > > >> >>>>>>>> Guangya
> > > > >> >>>>>>>>
> > > > >> >>>>>>>> On Wed, May 18, 2016 at 8:28 PM, Guangya Liu <
> > > gyliu513@gmail.com
> > > > >> >>>>>> wrote:
> > > > >> >>>>>>>>> Hi Olivier,
> > > > >> >>>>>>>>>
> > > > >> >>>>>>>>> I think that you need to enable "docker volume isolator"
> > if
> > > you
> > > > >> >>> want
> > > > >> >>>>>> use
> > > > >> >>>>>>>>> external storage with unified container I was writing a
> > > document
> > > > >> >>>>> here
> > > > >> >>>>>>>>> https://reviews.apache.org/r/47511/, perhaps you can
> have
> > > a try
> > > > >> >>>>>>> according
> > > > >> >>>>>>>>> to the document and post some comments there if you find
> > any
> > > > >> >>> issues.
> > > > >> >>>>>>>>> Also you can patch mesos-execute here
> > > > >> >>>>>>> https://reviews.apache.org/r/46762/ to
> > > > >> >>>>>>>>> have a try with mesos-execute.
> > > > >> >>>>>>>>>
> > > > >> >>>>>>>>> Thanks,
> > > > >> >>>>>>>>>
> > > > >> >>>>>>>>> Guangya
> > > > >> >>>>>>>>>
> > > > >> >>>>>>>>> On Wed, May 18, 2016 at 7:17 PM, Olivier Sallou <
> > > > >> >>>>>>> olivier.sallou@irisa.fr>
> > > > >> >>>>>>>>> wrote:
> > > > >> >>>>>>>>>
> > > > >> >>>>>>>>>> Answering (partially) to myself.
> > > > >> >>>>>>>>>>
> > > > >> >>>>>>>>>> I seems issue is container_path does not exists inside
> > > > >> >>> container.
> > > > >> >>>>> On
> > > > >> >>>>>>>>>> Docker, path is created and mounted. With pure mesos,
> > > > >> >>>>> container_path
> > > > >> >>>>>>>>>> must exists.
> > > > >> >>>>>>>>>>
> > > > >> >>>>>>>>>> mesos.proto says: "If the path is an absolute path,
> that
> > > path
> > > > >> >>> must
> > > > >> >>>>>>>>>> already exist."
> > > > >> >>>>>>>>>>
> > > > >> >>>>>>>>>> This is an issue however, using Docker images, the
> path I
> > > want
> > > > >> >>> to
> > > > >> >>>>>> mount
> > > > >> >>>>>>>>>> does not exists, and it cannot be modified "on the
> fly".
> > > > >> >>>>>>>>>>
> > > > >> >>>>>>>>>> Is there a workaround for this ?
> > > > >> >>>>>>>>>>
> > > > >> >>>>>>>>>>
> > > > >> >>>>>>>>>> On 05/18/2016 12:24 PM, Olivier Sallou wrote:
> > > > >> >>>>>>>>>>> Hi,
> > > > >> >>>>>>>>>>> I am trying unified containerizer on a single server
> > > > >> >>>>> (master/slave)
> > > > >> >>>>>> on
> > > > >> >>>>>>>>>>> mesos 0.28.1, to switch from docker containerizer to
> > > > >> >>> mesos+docker
> > > > >> >>>>>>> image
> > > > >> >>>>>>>>>>> container.
> > > > >> >>>>>>>>>>>
> > > > >> >>>>>>>>>>> I have setup slave config as suggested in
> documentation:
> > > > >> >>>>>>>>>>>
> > > > >> >>>>>>>>>>> containerizers=docker,mesos
> > > > >> >>>>>>>>>>> image_providers=docker \
> > > > >> >>>>>>>>>>> isolation=filesystem/linux,docker/runtime
> > > > >> >>>>>>>>>>>
> > > > >> >>>>>>>>>>> However, when I execute my task with a volume I have
> an
> > > error:
> > > > >> >>>>>>>>>>>
> > > > >> >>>>>>>>>>> ....
> > > > >> >>>>>>>>>>> + mount -n --rbind
> > > > >> >>>>>>>>>>>
> > > > >> >>>
> > > > >>
> > >
> >
> /tmp/mesos/provisioner/containers/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/backends/copy/rootfses/f9f66bb2-308d-4555-ba77-49ec61cbeb4f
> > > > >> >>>
> > > > >>
> > >
> >
> /tmp/mesos/slaves/2a296daf-7419-4659-ade1-763c792cd522-S0/frameworks/aef1b0e3-ea2d-4770-baac-96d673ab88f9-0000/executors/51/runs/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/.rootfs
> > > > >> >>>>>>>>>>> + mount -n --rbind
> > > > >> >>>>>>>>>>>
> > > > >> >>>
> > > > >>
> > >
> >
> /home/osallou/Development/NOSAVE/go-docker/godshared/tasks/pairtree_root/us/er/_o/sa/ll/ou/task
> > > > >> >>>
> > > > >>
> > >
> >
> /tmp/mesos/slaves/2a296daf-7419-4659-ade1-763c792cd522-S0/frameworks/aef1b0e3-ea2d-4770-baac-96d673ab88f9-0000/executors/51/runs/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/.rootfs/mnt/god-data
> > > > >> >>>>>>>>>>> mount: mount point
> > > > >> >>>>>>>>>>>
> > > > >> >>>
> > > > >>
> > >
> >
> /tmp/mesos/slaves/2a296daf-7419-4659-ade1-763c792cd522-S0/frameworks/aef1b0e3-ea2d-4770-baac-96d673ab88f9-0000/executors/51/runs/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/.rootfs/mnt/god-data
> > > > >> >>>>>>>>>>> does not exist
> > > > >> >>>>>>>>>>> Failed to execute a preparation shell command
> > > > >> >>>>>>>>>>>
> > > > >> >>>>>>>>>>> Then, my task switches to FAILED.
> > > > >> >>>>>>>>>>>
> > > > >> >>>>>>>>>>> I define a local volume to bind mount in my
> "container"
> > > > >> >>>>>>>>>>>
> > > > >> >>>
> > > > >>
> > >
> >
> /home/osallou/Development/NOSAVE/go-docker/godshared/tasks/pairtree_root/us/er/_o/sa/ll/ou/task
> > > > >> >>>>>>>>>>> => /mnt/god-data
> > > > >> >>>>>>>>>>> My directory exists on local server.
> > > > >> >>>>>>>>>>> In mesos UI, I can see the .rootfs directory along
> > stdout
> > > and
> > > > >> >>>>> stderr
> > > > >> >>>>>>>>>>> files, and inside .rootfs, I can see /mnt/god-data
> > > (empty).
> > > > >> >>>>>>>>>>>
> > > > >> >>>>>>>>>>> Running the same using Docker containerizer instead of
> > > mesos
> > > > >> >>>>>>>>>>> containerizer (with a Docker image) works fine.
> > > > >> >>>>>>>>>>>
> > > > >> >>>>>>>>>>> It seems it fails to mount my local directory in the
> > > > >> >>> container.
> > > > >> >>>>> Any
> > > > >> >>>>>>> idea
> > > > >> >>>>>>>>>>> of what is going wrong or how to debug this?
> > > > >> >>>>>>>>>>>
> > > > >> >>>>>>>>>>>
> > > > >> >>>>>>>>>>> Thanks
> > > > >> >>>>>>>>>>>
> > > > >> >>>>>>>>>> --
> > > > >> >>>>>>>>>> Olivier Sallou
> > > > >> >>>>>>>>>> IRISA / University of Rennes 1
> > > > >> >>>>>>>>>> Campus de Beaulieu, 35000 RENNES - FRANCE
> > > > >> >>>>>>>>>> Tel: 02.99.84.71.95
> > > > >> >>>>>>>>>>
> > > > >> >>>>>>>>>> gpg key id: 4096R/326D8438 (keyring.debian.org)
> > > > >> >>>>>>>>>> Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC
> > 68DB
> > > 326D
> > > > >> >>>>> 8438
> > > > >> >>>>>>> --
> > > > >> >>>>>>> Olivier Sallou
> > > > >> >>>>>>> IRISA / University of Rennes 1
> > > > >> >>>>>>> Campus de Beaulieu, 35000 RENNES - FRANCE
> > > > >> >>>>>>> Tel: 02.99.84.71.95
> > > > >> >>>>>>>
> > > > >> >>>>>>> gpg key id: 4096R/326D8438 (keyring.debian.org)
> > > > >> >>>>>>> Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB
> > > 326D
> > > > >> >>> 8438
> > > > >>
> > > > >> --
> > > > >> Olivier Sallou
> > > > >> IRISA / University of Rennes 1
> > > > >> Campus de Beaulieu, 35000 RENNES - FRANCE
> > > > >> Tel: 02.99.84.71.95
> > > > >>
> > > > >> gpg key id: 4096R/326D8438 (keyring.debian.org)
> > > > >> Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D
> 8438
> > > > >>
> > > > >>
> > > > >
> > > >
> > >
> >
>
>
>
> --
> Cheers,
>
> Zhitao Li
>
Re: volume / mount point error with Unified Containerizer
Posted by Zhitao Li <zh...@gmail.com>.
Guangya,
Is there a JIRA work board or document to track what's being fixed, working
on or planned in 0.29 and further releases?
On Tue, May 24, 2016 at 1:51 AM, Guangya Liu <gy...@gmail.com> wrote:
> Olivier,
>
> If it is not urgent, I think that you can use 0.29 in next week, the
> unified container will have more features in 0.29 such as external storage
> and network integration.
>
> Thanks,
>
> Guangya
>
> On Tue, May 24, 2016 at 2:17 AM, Olivier Sallou <ol...@irisa.fr>
> wrote:
>
> >
> >
> > ----- Mail original -----
> > > De: "Guangya Liu" <gy...@gmail.com>
> > > À: "dev" <de...@mesos.apache.org>, "Jie Yu" <ji...@mesosphere.io>,
> "Gilbert
> > Song" <gi...@mesosphere.io>
> > > Envoyé: Lundi 23 Mai 2016 17:34:41
> > > Objet: Re: volume / mount point error with Unified Containerizer
> > >
> > > It is a bit strange to me, I also did some test and review code for
> > > relative path, and found that relative path works well.
> > >
> > > In 0.28.1, if deploy a docker container with MesosContaineirizer, then
> if
> > > using absolute path as continer_path, the mesos agent will update the
> > > container_path to a relative path by adding a prefix ./rootfs to the
> > > container_path, e.g. /file/path = > ./rootfs/file/path.
> > >
> > > If deploy a docker container with MesosContaineirizer with relative
> path
> > as
> > > container_path, then the mesos agent will not update the
> container_path.
> > >
> > > So the final mount point for the container should be either
> > >
> > > 1) /tmp/mesos/slaves/agent_id/frameworks/framework_id/
> > > executors/51/runs/container_id/.rootfs/file/path
> > > 2)
> /tmp/mesos/slaves/agent_id/frameworks/framework_id/executors/51/runs/
> > > container_id/file/path
> > >
> > > The only difference is adding ./rootfs as a prefix or not, the test
> > result
> > > is that 1) does not work and 2) works well. And even the mount for 1)
> > > failed, but I can see the mount point path does exist.
> > >
> >
> > @Guangya
> > I confirm that using relative path works fine, I get volumes in mesos
> path
> > (but it does not help for my implementation).
> > If I use the Docker containerizer, absolute paths are fine, this is what
> I
> > use for the moment in my code, and am investigating to switch to unified
> > container.
> >
> >
> > > @Yu Jie and @Gilbert, any comments for this?
> > >
> > > @Oilivier,
> > >
> > > In order not to block your test, can you please use mesos after 0.28.1?
> > You
> > > can use either 0.28.2 or above version.
> >
> > Well, as this is not an urgent matter, I am waiting 0.29 to test against
> > this release (with other features I am waiting for).
> >
> > >
> > > Thanks,
> > >
> > > Guangya
> > >
> > >
> > > On Mon, May 23, 2016 at 10:30 PM, Guangya Liu <gy...@gmail.com>
> > wrote:
> > >
> > > > Thanks Olivier, I can reproduce this issue now and still checking
> what
> > is
> > > > wrong.
> > > >
> > > > What I did is as following:
> > > > 1) Check out code with tag of 0.28.1
> > > > 2) update mesos-execute to add a host path volume
> > > > diff --git a/src/cli/execute.cpp b/src/cli/execute.cpp
> > > > index 81a0388..0ff913c 100644
> > > > --- a/src/cli/execute.cpp
> > > > +++ b/src/cli/execute.cpp
> > > > @@ -72,6 +72,8 @@ using mesos::v1::TaskID;
> > > > using mesos::v1::TaskInfo;
> > > > using mesos::v1::TaskState;
> > > > using mesos::v1::TaskStatus;
> > > > +using mesos::v1::Volume;
> > > > +using mesos::v1::Parameters;
> > > >
> > > > using mesos::v1::scheduler::Call;
> > > > using mesos::v1::scheduler::Event;
> > > > @@ -572,6 +574,12 @@ private:
> > > > }
> > > > }
> > > >
> > > > + Volume* volume1 = containerInfo.add_volumes();
> > > > + volume1->set_container_path("/tmp/abcd");
> > > > + volume1->set_mode(Volume::RW);
> > > > + volume1->set_host_path("/root/convoy");
> > > > + cout << "Add Voume 1" << endl;
> > > > +
> > > > return containerInfo;
> > > > } else if (containerizer == "docker") {
> > > > // 'docker' containerizer only supports 'docker' images.
> > > > 3) launch a task with docker image, task failed.
> > > >
> > > > 4) Check sandbox:
> > > > + /root/src/mesos/m1/mesos/build/src/mesos-containerizer mount
> > > > --help=false --operation=make-rslave --path=/
> > > > + grep -E /tmp/mesos/.+ /proc/self/mountinfo
> > > > + grep -v 3239aafc-78d8-4f70-81e5-f32fb3733339
> > > > + cut+ -d -f5
> > > > xargs --no-run-if-empty umount -l
> > > > + mount -n --rbind
> > > >
> >
> /tmp/mesos/provisioner/containers/3239aafc-78d8-4f70-81e5-f32fb3733339/backends/copy/rootfses/5e8bf3fa-53b1-4bd5-bb3d-525ddc7900b6
> > > >
> >
> /tmp/mesos/slaves/a4294ed5-10e8-47db-a3b9-a43a4f951374-S0/frameworks/a4294ed5-10e8-47db-a3b9-a43a4f951374-0000/executors/test/runs/3239aafc-78d8-4f70-81e5-f32fb3733339/.rootfs
> > > > + mount -n --rbind /root/convoy
> > > >
> >
> /tmp/mesos/slaves/a4294ed5-10e8-47db-a3b9-a43a4f951374-S0/frameworks/a4294ed5-10e8-47db-a3b9-a43a4f951374-0000/executors/test/runs/3239aafc-78d8-4f70-81e5-f32fb3733339/.rootfs/tmp/abcd
> > > > mount: mount point
> > > >
> >
> /tmp/mesos/slaves/a4294ed5-10e8-47db-a3b9-a43a4f951374-S0/frameworks/a4294ed5-10e8-47db-a3b9-a43a4f951374-0000/executors/test/runs/3239aafc-78d8-4f70-81e5-f32fb3733339/.rootfs/tmp/abcd
> > > > does not exist
> > > > Failed to execute a preparation shell command
> > > >
> > > > Will check more for this.
> > > >
> > > > Thanks,
> > > >
> > > > Guangya
> > > >
> > > > On Mon, May 23, 2016 at 3:35 PM, Olivier Sallou <
> > olivier.sallou@irisa.fr>
> > > > wrote:
> > > >
> > > >>
> > > >>
> > > >> On 05/23/2016 09:33 AM, Olivier Sallou wrote:
> > > >> >
> > > >> > On 05/20/2016 03:26 PM, Guangya Liu wrote:
> > > >> >> Since you are using docker image which means that your container
> > will
> > > >> have
> > > >> >> rootfs, so it is not required to have the absolute path exist,
> the
> > > >> linux
> > > >> >> file system isolator will help create the path automatically
> > > >> >>
> > > >>
> >
> https://github.com/apache/mesos/blob/0.28.x/src/slave/containerizer/mesos/isolators/filesystem/linux.cpp#L390-L402
> > > >> >>
> > > >> >> Can you please share your framework? How did you set the volume
> > part in
> > > >> >> your framework?
> > > >> > @Guangya
> > > >> >
> > > >> > I use Python API.
> > > >> >
> > > >> > Here is related code:
> > > >> >
> > > >> > ....
> > > >> > # Define container volumes
> > > >> > for v in job['container']['volumes']:
> > > >> > volume = container.volumes.add()
> > > >> > volume.container_path = v['mount']
> > > >> > volume.host_path = v['path']
> > > >> > if v['acl'] == 'rw':
> > > >> > volume.mode = 1 # mesos_pb2.Volume.Mode.RW
> > > >> > else:
> > > >> > volume.mode = 2 # mesos_pb2.Volume.Mode.RO
> > > >> >
> > > >> > => In my test case, I add 2 volumes from a host shared
> > directory,
> > > >> > mounted in container as /mnt/go-docker and /mnt/god-data.
> > > >> >
> > > >> > ...
> > > >> > # Define docker image and network
> > > >> > docker = mesos_pb2.ContainerInfo.MesosInfo()
> > > >> > docker.image.type = 2 # Docker
> > > >> > docker.image.docker.name ='centos:latest'
> > > >> > # Request an IP from a network module
> > > >> > network_info = container.network_infos.add()
> > > >> > network_info_name = 'sampletest'
> > > >> > # Get an IP V4 address
> > > >> > ip_address = network_info.ip_addresses.add()
> > > >> > ip_address.protocol = 1
> > > >> > # The network group to join
> > > >> > group = network_info.groups.append(network_info_name)
> > > >> > port_list = [22]
> > > >> > if port_list:
> > > >> > for port in port_list:
> > > >> >
> job['container']['port_mapping'].append({'host':
> > > >> > port, 'container': port})
> > > >> > container.mesos.MergeFrom(docker)
> > > >> >
> > > >> > It results in error message:
> > > >> >
> > > >> > + mount -n --rbind
> > > >> >
> > > >>
> >
> /tmp/mesos/provisioner/containers/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/backends/copy/rootfses/f9f66bb2-308d-4555-ba77-49ec61cbeb4f
> > > >> >
> > > >>
> >
> /tmp/mesos/slaves/2a296daf-7419-4659-ade1-763c792cd522-S0/frameworks/aef1b0e3-ea2d-4770-baac-96d673ab88f9-0000/executors/51/runs/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/.rootfs
> > > >> > + mount -n --rbind
> > > >> >
> > > >>
> >
> /home/osallou/Development/NOSAVE/go-docker/godshared/tasks/pairtree_root/us/er/_o/sa/ll/ou/task
> > > >> >
> > > >>
> >
> /tmp/mesos/slaves/2a296daf-7419-4659-ade1-763c792cd522-S0/frameworks/aef1b0e3-ea2d-4770-baac-96d673ab88f9-0000/executors/51/runs/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/.rootfs/mnt/god-data
> > > >> > mount: mount point
> > > >> >
> > > >>
> >
> /tmp/mesos/slaves/2a296daf-7419-4659-ade1-763c792cd522-S0/frameworks/aef1b0e3-ea2d-4770-baac-96d673ab88f9-0000/executors/51/runs/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/.rootfs/mnt/god-data
> > > >> > does not exist
> > > >> > Failed to execute a preparation shell command
> > > >> >
> > > >> >
> > > >> > We can see the .rootfs, but mnt/god-data under .rootfs fails.
> Local
> > > >> > directory exists, it does not pre-exists in container. What is
> > strange
> > > >> > is , if I look in mesos task dir, .rootfs/mnt/go-data, directory
> is
> > > >> present.
> > > >> >
> > > >> > Or, is the error message (.rootfs/mnt/god-data does not exist)
> > simply a
> > > >> > warning, and it creates it, and final error (Failed to execute a
> > > >> > preparation shell command) not related (and unclear...)
> > > >> Additional info: command to execute in container is located in one
> of
> > > >> mounted volume.
> > > >> >
> > > >> > Olivier
> > > >> >> Thanks,
> > > >> >>
> > > >> >> Guangya
> > > >> >>
> > > >> >> On Fri, May 20, 2016 at 4:54 AM, Olivier Sallou <
> > > >> olivier.sallou@irisa.fr>
> > > >> >> wrote:
> > > >> >>
> > > >> >>> ----- Mail original -----
> > > >> >>>> De: "Gilbert Song" <gi...@mesosphere.io>
> > > >> >>>> À: "dev" <de...@mesos.apache.org>
> > > >> >>>> Envoyé: Jeudi 19 Mai 2016 01:57:16
> > > >> >>>> Objet: Re: volume / mount point error with Unified
> Containerizer
> > > >> >>>>
> > > >> >>>> @Olivier,
> > > >> >>>> In mesos 0.28.1, you are supposed to be able bind mount a
> volume
> > from
> > > >> >>>> the host into the mesos container. Did you specify a docker
> > image (we
> > > >> >>>> determine
> > > >> >>>> the mount point differently depending whether the container
> has a
> > > >> >>> rootfs)?
> > > >> >>>
> > > >> >>> Yes I specified an image, a Docker image URI.
> > > >> >>>
> > > >> >>>> How
> > > >> >>>> do you specify your 'container_path' (the mount point in the
> > > >> container)?
> > > >> >>> If
> > > >> >>>> it is an
> > > >> >>>> absolute path, we require that dir to be pre-existed. If it is
> a
> > > >> relative
> > > >> >>>> path, we will
> > > >> >>>> mkdir for it.
> > > >> >>> It is an absolute path, but it does not exists in image (this is
> > the
> > > >> >>> issue). Images are custom Docker images (images containing tools
> > for
> > > >> batch
> > > >> >>> computing), and I want, for example, to mount some shared
> > resources
> > > >> (user
> > > >> >>> home dir, common data, etc.) in the image. Of course those
> > > >> directories do
> > > >> >>> not pre-exists in container images as they are specific to the
> > > >> environment.
> > > >> >>> Requiring existence of the directory in the image is not issue
> as
> > it
> > > >> >>> prevents using any existing image from a repo.
> > > >> >>>
> > > >> >>> When using Docker containerizer it works fine, I can mount any
> > > >> external
> > > >> >>> storage in the container.
> > > >> >>>
> > > >> >>> Olivie
> > > >> >>>
> > > >> >>>
> > > >> >>>> @Joshua,
> > > >> >>>> Thank for posting your workaround on mesos. As I mentioned
> > above, in
> > > >> >>> 0.28.1
> > > >> >>>> or
> > > >> >>>> older, we only mkdir for container_path which is relative path
> > (not
> > > >> >>>> starting with "/").
> > > >> >>>> Because if no rootfs specified for a mesos container, the
> > container
> > > >> >>> shares
> > > >> >>>> the host
> > > >> >>>> root filesystem. Obviously we don't want any random files to be
> > > >> created
> > > >> >>>> implicitly
> > > >> >>>> on your host fs.
> > > >> >>>> From mesos 0.29 (release by the end of this month), we will
> > mkdir the
> > > >> >>> mount
> > > >> >>>> point in the container except for the command task case that
> > specify
> > > >> an
> > > >> >>>> absolute
> > > >> >>>> container_path without a rootfs. Because we simplify the
> mounting
> > > >> logic,
> > > >> >>> and
> > > >> >>>> sandbox bind mount will only be done in container mount
> namespace
> > > >> >>> instead of
> > > >> >>>> host mount namespace (what we did before). Please keep tuned.
> > > >> >>>>
> > > >> >>>> Cheers,
> > > >> >>>> Gilbert
> > > >> >>>>
> > > >> >>>> On Wed, May 18, 2016 at 8:14 AM, Joshua Cohen <
> jcohen@apache.org
> > >
> > > >> wrote:
> > > >> >>>>
> > > >> >>>>> Hi Olivier,
> > > >> >>>>>
> > > >> >>>>> I touched on this issue as part of
> > > >> >>>>> https://issues.apache.org/jira/browse/MESOS-5229. It would be
> > nice
> > > >> if
> > > >> >>>>> Mesos
> > > >> >>>>> automatically created container mount points if they don't
> > already
> > > >> >>> exist.
> > > >> >>>>> In the meantime, as a workaround for this, I've updated my
> > > >> filesystem
> > > >> >>>>> images to include the path (e.g. in Dockerfile, add `RUN mkdir
> > -p
> > > >> >>>>> /some/mount/point`). Not the best solution, but the only thing
> > I've
> > > >> >>> seen
> > > >> >>>>> that works at the moment.
> > > >> >>>>>
> > > >> >>>>> Cheers,
> > > >> >>>>>
> > > >> >>>>> Joshua
> > > >> >>>>>
> > > >> >>>>> On Wed, May 18, 2016 at 7:36 AM, Guangya Liu <
> > gyliu513@gmail.com>
> > > >> >>> wrote:
> > > >> >>>>>> It's pretty simple for you from scratch with source code
> > > >> >>>>>>
> > > >> >>>>>>
> > > >> >>>
> > > >>
> >
> https://github.com/apache/mesos/blob/master/docs/getting-started.md#building-mesos
> > > >> >>>>>> ;-)
> > > >> >>>>>>
> > > >> >>>>>> Thanks,
> > > >> >>>>>>
> > > >> >>>>>> Guangya
> > > >> >>>>>>
> > > >> >>>>>> On Wed, May 18, 2016 at 8:30 PM, Olivier Sallou <
> > > >> >>> olivier.sallou@irisa.fr
> > > >> >>>>>> wrote:
> > > >> >>>>>>
> > > >> >>>>>>> On 05/18/2016 02:31 PM, Guangya Liu wrote:
> > > >> >>>>>>>> Just saw that you are working with 0.28.1, the "docker
> volume
> > > >> >>> driver"
> > > >> >>>>>>> code
> > > >> >>>>>>>> was not in 0.28.1, can you please have a try with mesos
> > master
> > > >> >>> branch
> > > >> >>>>>> if
> > > >> >>>>>>>> you are only doing some test?
> > > >> >>>>>>> this is indeed test only for the moment. But I will have to
> > > >> >>>>>>> recompile/install mesos :-( (I used packages for install).
> > > >> >>>>>>>
> > > >> >>>>>>> I will try when possible, but thanks for the hint.
> > > >> >>>>>>>> Thanks,
> > > >> >>>>>>>>
> > > >> >>>>>>>> Guangya
> > > >> >>>>>>>>
> > > >> >>>>>>>> On Wed, May 18, 2016 at 8:28 PM, Guangya Liu <
> > gyliu513@gmail.com
> > > >> >>>>>> wrote:
> > > >> >>>>>>>>> Hi Olivier,
> > > >> >>>>>>>>>
> > > >> >>>>>>>>> I think that you need to enable "docker volume isolator"
> if
> > you
> > > >> >>> want
> > > >> >>>>>> use
> > > >> >>>>>>>>> external storage with unified container I was writing a
> > document
> > > >> >>>>> here
> > > >> >>>>>>>>> https://reviews.apache.org/r/47511/, perhaps you can have
> > a try
> > > >> >>>>>>> according
> > > >> >>>>>>>>> to the document and post some comments there if you find
> any
> > > >> >>> issues.
> > > >> >>>>>>>>> Also you can patch mesos-execute here
> > > >> >>>>>>> https://reviews.apache.org/r/46762/ to
> > > >> >>>>>>>>> have a try with mesos-execute.
> > > >> >>>>>>>>>
> > > >> >>>>>>>>> Thanks,
> > > >> >>>>>>>>>
> > > >> >>>>>>>>> Guangya
> > > >> >>>>>>>>>
> > > >> >>>>>>>>> On Wed, May 18, 2016 at 7:17 PM, Olivier Sallou <
> > > >> >>>>>>> olivier.sallou@irisa.fr>
> > > >> >>>>>>>>> wrote:
> > > >> >>>>>>>>>
> > > >> >>>>>>>>>> Answering (partially) to myself.
> > > >> >>>>>>>>>>
> > > >> >>>>>>>>>> I seems issue is container_path does not exists inside
> > > >> >>> container.
> > > >> >>>>> On
> > > >> >>>>>>>>>> Docker, path is created and mounted. With pure mesos,
> > > >> >>>>> container_path
> > > >> >>>>>>>>>> must exists.
> > > >> >>>>>>>>>>
> > > >> >>>>>>>>>> mesos.proto says: "If the path is an absolute path, that
> > path
> > > >> >>> must
> > > >> >>>>>>>>>> already exist."
> > > >> >>>>>>>>>>
> > > >> >>>>>>>>>> This is an issue however, using Docker images, the path I
> > want
> > > >> >>> to
> > > >> >>>>>> mount
> > > >> >>>>>>>>>> does not exists, and it cannot be modified "on the fly".
> > > >> >>>>>>>>>>
> > > >> >>>>>>>>>> Is there a workaround for this ?
> > > >> >>>>>>>>>>
> > > >> >>>>>>>>>>
> > > >> >>>>>>>>>> On 05/18/2016 12:24 PM, Olivier Sallou wrote:
> > > >> >>>>>>>>>>> Hi,
> > > >> >>>>>>>>>>> I am trying unified containerizer on a single server
> > > >> >>>>> (master/slave)
> > > >> >>>>>> on
> > > >> >>>>>>>>>>> mesos 0.28.1, to switch from docker containerizer to
> > > >> >>> mesos+docker
> > > >> >>>>>>> image
> > > >> >>>>>>>>>>> container.
> > > >> >>>>>>>>>>>
> > > >> >>>>>>>>>>> I have setup slave config as suggested in documentation:
> > > >> >>>>>>>>>>>
> > > >> >>>>>>>>>>> containerizers=docker,mesos
> > > >> >>>>>>>>>>> image_providers=docker \
> > > >> >>>>>>>>>>> isolation=filesystem/linux,docker/runtime
> > > >> >>>>>>>>>>>
> > > >> >>>>>>>>>>> However, when I execute my task with a volume I have an
> > error:
> > > >> >>>>>>>>>>>
> > > >> >>>>>>>>>>> ....
> > > >> >>>>>>>>>>> + mount -n --rbind
> > > >> >>>>>>>>>>>
> > > >> >>>
> > > >>
> >
> /tmp/mesos/provisioner/containers/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/backends/copy/rootfses/f9f66bb2-308d-4555-ba77-49ec61cbeb4f
> > > >> >>>
> > > >>
> >
> /tmp/mesos/slaves/2a296daf-7419-4659-ade1-763c792cd522-S0/frameworks/aef1b0e3-ea2d-4770-baac-96d673ab88f9-0000/executors/51/runs/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/.rootfs
> > > >> >>>>>>>>>>> + mount -n --rbind
> > > >> >>>>>>>>>>>
> > > >> >>>
> > > >>
> >
> /home/osallou/Development/NOSAVE/go-docker/godshared/tasks/pairtree_root/us/er/_o/sa/ll/ou/task
> > > >> >>>
> > > >>
> >
> /tmp/mesos/slaves/2a296daf-7419-4659-ade1-763c792cd522-S0/frameworks/aef1b0e3-ea2d-4770-baac-96d673ab88f9-0000/executors/51/runs/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/.rootfs/mnt/god-data
> > > >> >>>>>>>>>>> mount: mount point
> > > >> >>>>>>>>>>>
> > > >> >>>
> > > >>
> >
> /tmp/mesos/slaves/2a296daf-7419-4659-ade1-763c792cd522-S0/frameworks/aef1b0e3-ea2d-4770-baac-96d673ab88f9-0000/executors/51/runs/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/.rootfs/mnt/god-data
> > > >> >>>>>>>>>>> does not exist
> > > >> >>>>>>>>>>> Failed to execute a preparation shell command
> > > >> >>>>>>>>>>>
> > > >> >>>>>>>>>>> Then, my task switches to FAILED.
> > > >> >>>>>>>>>>>
> > > >> >>>>>>>>>>> I define a local volume to bind mount in my "container"
> > > >> >>>>>>>>>>>
> > > >> >>>
> > > >>
> >
> /home/osallou/Development/NOSAVE/go-docker/godshared/tasks/pairtree_root/us/er/_o/sa/ll/ou/task
> > > >> >>>>>>>>>>> => /mnt/god-data
> > > >> >>>>>>>>>>> My directory exists on local server.
> > > >> >>>>>>>>>>> In mesos UI, I can see the .rootfs directory along
> stdout
> > and
> > > >> >>>>> stderr
> > > >> >>>>>>>>>>> files, and inside .rootfs, I can see /mnt/god-data
> > (empty).
> > > >> >>>>>>>>>>>
> > > >> >>>>>>>>>>> Running the same using Docker containerizer instead of
> > mesos
> > > >> >>>>>>>>>>> containerizer (with a Docker image) works fine.
> > > >> >>>>>>>>>>>
> > > >> >>>>>>>>>>> It seems it fails to mount my local directory in the
> > > >> >>> container.
> > > >> >>>>> Any
> > > >> >>>>>>> idea
> > > >> >>>>>>>>>>> of what is going wrong or how to debug this?
> > > >> >>>>>>>>>>>
> > > >> >>>>>>>>>>>
> > > >> >>>>>>>>>>> Thanks
> > > >> >>>>>>>>>>>
> > > >> >>>>>>>>>> --
> > > >> >>>>>>>>>> Olivier Sallou
> > > >> >>>>>>>>>> IRISA / University of Rennes 1
> > > >> >>>>>>>>>> Campus de Beaulieu, 35000 RENNES - FRANCE
> > > >> >>>>>>>>>> Tel: 02.99.84.71.95
> > > >> >>>>>>>>>>
> > > >> >>>>>>>>>> gpg key id: 4096R/326D8438 (keyring.debian.org)
> > > >> >>>>>>>>>> Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC
> 68DB
> > 326D
> > > >> >>>>> 8438
> > > >> >>>>>>> --
> > > >> >>>>>>> Olivier Sallou
> > > >> >>>>>>> IRISA / University of Rennes 1
> > > >> >>>>>>> Campus de Beaulieu, 35000 RENNES - FRANCE
> > > >> >>>>>>> Tel: 02.99.84.71.95
> > > >> >>>>>>>
> > > >> >>>>>>> gpg key id: 4096R/326D8438 (keyring.debian.org)
> > > >> >>>>>>> Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB
> > 326D
> > > >> >>> 8438
> > > >>
> > > >> --
> > > >> Olivier Sallou
> > > >> IRISA / University of Rennes 1
> > > >> Campus de Beaulieu, 35000 RENNES - FRANCE
> > > >> Tel: 02.99.84.71.95
> > > >>
> > > >> gpg key id: 4096R/326D8438 (keyring.debian.org)
> > > >> Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D 8438
> > > >>
> > > >>
> > > >
> > >
> >
>
--
Cheers,
Zhitao Li
Re: volume / mount point error with Unified Containerizer
Posted by Guangya Liu <gy...@gmail.com>.
Olivier,
If it is not urgent, I think that you can use 0.29 in next week, the
unified container will have more features in 0.29 such as external storage
and network integration.
Thanks,
Guangya
On Tue, May 24, 2016 at 2:17 AM, Olivier Sallou <ol...@irisa.fr>
wrote:
>
>
> ----- Mail original -----
> > De: "Guangya Liu" <gy...@gmail.com>
> > À: "dev" <de...@mesos.apache.org>, "Jie Yu" <ji...@mesosphere.io>, "Gilbert
> Song" <gi...@mesosphere.io>
> > Envoyé: Lundi 23 Mai 2016 17:34:41
> > Objet: Re: volume / mount point error with Unified Containerizer
> >
> > It is a bit strange to me, I also did some test and review code for
> > relative path, and found that relative path works well.
> >
> > In 0.28.1, if deploy a docker container with MesosContaineirizer, then if
> > using absolute path as continer_path, the mesos agent will update the
> > container_path to a relative path by adding a prefix ./rootfs to the
> > container_path, e.g. /file/path = > ./rootfs/file/path.
> >
> > If deploy a docker container with MesosContaineirizer with relative path
> as
> > container_path, then the mesos agent will not update the container_path.
> >
> > So the final mount point for the container should be either
> >
> > 1) /tmp/mesos/slaves/agent_id/frameworks/framework_id/
> > executors/51/runs/container_id/.rootfs/file/path
> > 2) /tmp/mesos/slaves/agent_id/frameworks/framework_id/executors/51/runs/
> > container_id/file/path
> >
> > The only difference is adding ./rootfs as a prefix or not, the test
> result
> > is that 1) does not work and 2) works well. And even the mount for 1)
> > failed, but I can see the mount point path does exist.
> >
>
> @Guangya
> I confirm that using relative path works fine, I get volumes in mesos path
> (but it does not help for my implementation).
> If I use the Docker containerizer, absolute paths are fine, this is what I
> use for the moment in my code, and am investigating to switch to unified
> container.
>
>
> > @Yu Jie and @Gilbert, any comments for this?
> >
> > @Oilivier,
> >
> > In order not to block your test, can you please use mesos after 0.28.1?
> You
> > can use either 0.28.2 or above version.
>
> Well, as this is not an urgent matter, I am waiting 0.29 to test against
> this release (with other features I am waiting for).
>
> >
> > Thanks,
> >
> > Guangya
> >
> >
> > On Mon, May 23, 2016 at 10:30 PM, Guangya Liu <gy...@gmail.com>
> wrote:
> >
> > > Thanks Olivier, I can reproduce this issue now and still checking what
> is
> > > wrong.
> > >
> > > What I did is as following:
> > > 1) Check out code with tag of 0.28.1
> > > 2) update mesos-execute to add a host path volume
> > > diff --git a/src/cli/execute.cpp b/src/cli/execute.cpp
> > > index 81a0388..0ff913c 100644
> > > --- a/src/cli/execute.cpp
> > > +++ b/src/cli/execute.cpp
> > > @@ -72,6 +72,8 @@ using mesos::v1::TaskID;
> > > using mesos::v1::TaskInfo;
> > > using mesos::v1::TaskState;
> > > using mesos::v1::TaskStatus;
> > > +using mesos::v1::Volume;
> > > +using mesos::v1::Parameters;
> > >
> > > using mesos::v1::scheduler::Call;
> > > using mesos::v1::scheduler::Event;
> > > @@ -572,6 +574,12 @@ private:
> > > }
> > > }
> > >
> > > + Volume* volume1 = containerInfo.add_volumes();
> > > + volume1->set_container_path("/tmp/abcd");
> > > + volume1->set_mode(Volume::RW);
> > > + volume1->set_host_path("/root/convoy");
> > > + cout << "Add Voume 1" << endl;
> > > +
> > > return containerInfo;
> > > } else if (containerizer == "docker") {
> > > // 'docker' containerizer only supports 'docker' images.
> > > 3) launch a task with docker image, task failed.
> > >
> > > 4) Check sandbox:
> > > + /root/src/mesos/m1/mesos/build/src/mesos-containerizer mount
> > > --help=false --operation=make-rslave --path=/
> > > + grep -E /tmp/mesos/.+ /proc/self/mountinfo
> > > + grep -v 3239aafc-78d8-4f70-81e5-f32fb3733339
> > > + cut+ -d -f5
> > > xargs --no-run-if-empty umount -l
> > > + mount -n --rbind
> > >
> /tmp/mesos/provisioner/containers/3239aafc-78d8-4f70-81e5-f32fb3733339/backends/copy/rootfses/5e8bf3fa-53b1-4bd5-bb3d-525ddc7900b6
> > >
> /tmp/mesos/slaves/a4294ed5-10e8-47db-a3b9-a43a4f951374-S0/frameworks/a4294ed5-10e8-47db-a3b9-a43a4f951374-0000/executors/test/runs/3239aafc-78d8-4f70-81e5-f32fb3733339/.rootfs
> > > + mount -n --rbind /root/convoy
> > >
> /tmp/mesos/slaves/a4294ed5-10e8-47db-a3b9-a43a4f951374-S0/frameworks/a4294ed5-10e8-47db-a3b9-a43a4f951374-0000/executors/test/runs/3239aafc-78d8-4f70-81e5-f32fb3733339/.rootfs/tmp/abcd
> > > mount: mount point
> > >
> /tmp/mesos/slaves/a4294ed5-10e8-47db-a3b9-a43a4f951374-S0/frameworks/a4294ed5-10e8-47db-a3b9-a43a4f951374-0000/executors/test/runs/3239aafc-78d8-4f70-81e5-f32fb3733339/.rootfs/tmp/abcd
> > > does not exist
> > > Failed to execute a preparation shell command
> > >
> > > Will check more for this.
> > >
> > > Thanks,
> > >
> > > Guangya
> > >
> > > On Mon, May 23, 2016 at 3:35 PM, Olivier Sallou <
> olivier.sallou@irisa.fr>
> > > wrote:
> > >
> > >>
> > >>
> > >> On 05/23/2016 09:33 AM, Olivier Sallou wrote:
> > >> >
> > >> > On 05/20/2016 03:26 PM, Guangya Liu wrote:
> > >> >> Since you are using docker image which means that your container
> will
> > >> have
> > >> >> rootfs, so it is not required to have the absolute path exist, the
> > >> linux
> > >> >> file system isolator will help create the path automatically
> > >> >>
> > >>
> https://github.com/apache/mesos/blob/0.28.x/src/slave/containerizer/mesos/isolators/filesystem/linux.cpp#L390-L402
> > >> >>
> > >> >> Can you please share your framework? How did you set the volume
> part in
> > >> >> your framework?
> > >> > @Guangya
> > >> >
> > >> > I use Python API.
> > >> >
> > >> > Here is related code:
> > >> >
> > >> > ....
> > >> > # Define container volumes
> > >> > for v in job['container']['volumes']:
> > >> > volume = container.volumes.add()
> > >> > volume.container_path = v['mount']
> > >> > volume.host_path = v['path']
> > >> > if v['acl'] == 'rw':
> > >> > volume.mode = 1 # mesos_pb2.Volume.Mode.RW
> > >> > else:
> > >> > volume.mode = 2 # mesos_pb2.Volume.Mode.RO
> > >> >
> > >> > => In my test case, I add 2 volumes from a host shared
> directory,
> > >> > mounted in container as /mnt/go-docker and /mnt/god-data.
> > >> >
> > >> > ...
> > >> > # Define docker image and network
> > >> > docker = mesos_pb2.ContainerInfo.MesosInfo()
> > >> > docker.image.type = 2 # Docker
> > >> > docker.image.docker.name ='centos:latest'
> > >> > # Request an IP from a network module
> > >> > network_info = container.network_infos.add()
> > >> > network_info_name = 'sampletest'
> > >> > # Get an IP V4 address
> > >> > ip_address = network_info.ip_addresses.add()
> > >> > ip_address.protocol = 1
> > >> > # The network group to join
> > >> > group = network_info.groups.append(network_info_name)
> > >> > port_list = [22]
> > >> > if port_list:
> > >> > for port in port_list:
> > >> > job['container']['port_mapping'].append({'host':
> > >> > port, 'container': port})
> > >> > container.mesos.MergeFrom(docker)
> > >> >
> > >> > It results in error message:
> > >> >
> > >> > + mount -n --rbind
> > >> >
> > >>
> /tmp/mesos/provisioner/containers/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/backends/copy/rootfses/f9f66bb2-308d-4555-ba77-49ec61cbeb4f
> > >> >
> > >>
> /tmp/mesos/slaves/2a296daf-7419-4659-ade1-763c792cd522-S0/frameworks/aef1b0e3-ea2d-4770-baac-96d673ab88f9-0000/executors/51/runs/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/.rootfs
> > >> > + mount -n --rbind
> > >> >
> > >>
> /home/osallou/Development/NOSAVE/go-docker/godshared/tasks/pairtree_root/us/er/_o/sa/ll/ou/task
> > >> >
> > >>
> /tmp/mesos/slaves/2a296daf-7419-4659-ade1-763c792cd522-S0/frameworks/aef1b0e3-ea2d-4770-baac-96d673ab88f9-0000/executors/51/runs/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/.rootfs/mnt/god-data
> > >> > mount: mount point
> > >> >
> > >>
> /tmp/mesos/slaves/2a296daf-7419-4659-ade1-763c792cd522-S0/frameworks/aef1b0e3-ea2d-4770-baac-96d673ab88f9-0000/executors/51/runs/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/.rootfs/mnt/god-data
> > >> > does not exist
> > >> > Failed to execute a preparation shell command
> > >> >
> > >> >
> > >> > We can see the .rootfs, but mnt/god-data under .rootfs fails. Local
> > >> > directory exists, it does not pre-exists in container. What is
> strange
> > >> > is , if I look in mesos task dir, .rootfs/mnt/go-data, directory is
> > >> present.
> > >> >
> > >> > Or, is the error message (.rootfs/mnt/god-data does not exist)
> simply a
> > >> > warning, and it creates it, and final error (Failed to execute a
> > >> > preparation shell command) not related (and unclear...)
> > >> Additional info: command to execute in container is located in one of
> > >> mounted volume.
> > >> >
> > >> > Olivier
> > >> >> Thanks,
> > >> >>
> > >> >> Guangya
> > >> >>
> > >> >> On Fri, May 20, 2016 at 4:54 AM, Olivier Sallou <
> > >> olivier.sallou@irisa.fr>
> > >> >> wrote:
> > >> >>
> > >> >>> ----- Mail original -----
> > >> >>>> De: "Gilbert Song" <gi...@mesosphere.io>
> > >> >>>> À: "dev" <de...@mesos.apache.org>
> > >> >>>> Envoyé: Jeudi 19 Mai 2016 01:57:16
> > >> >>>> Objet: Re: volume / mount point error with Unified Containerizer
> > >> >>>>
> > >> >>>> @Olivier,
> > >> >>>> In mesos 0.28.1, you are supposed to be able bind mount a volume
> from
> > >> >>>> the host into the mesos container. Did you specify a docker
> image (we
> > >> >>>> determine
> > >> >>>> the mount point differently depending whether the container has a
> > >> >>> rootfs)?
> > >> >>>
> > >> >>> Yes I specified an image, a Docker image URI.
> > >> >>>
> > >> >>>> How
> > >> >>>> do you specify your 'container_path' (the mount point in the
> > >> container)?
> > >> >>> If
> > >> >>>> it is an
> > >> >>>> absolute path, we require that dir to be pre-existed. If it is a
> > >> relative
> > >> >>>> path, we will
> > >> >>>> mkdir for it.
> > >> >>> It is an absolute path, but it does not exists in image (this is
> the
> > >> >>> issue). Images are custom Docker images (images containing tools
> for
> > >> batch
> > >> >>> computing), and I want, for example, to mount some shared
> resources
> > >> (user
> > >> >>> home dir, common data, etc.) in the image. Of course those
> > >> directories do
> > >> >>> not pre-exists in container images as they are specific to the
> > >> environment.
> > >> >>> Requiring existence of the directory in the image is not issue as
> it
> > >> >>> prevents using any existing image from a repo.
> > >> >>>
> > >> >>> When using Docker containerizer it works fine, I can mount any
> > >> external
> > >> >>> storage in the container.
> > >> >>>
> > >> >>> Olivie
> > >> >>>
> > >> >>>
> > >> >>>> @Joshua,
> > >> >>>> Thank for posting your workaround on mesos. As I mentioned
> above, in
> > >> >>> 0.28.1
> > >> >>>> or
> > >> >>>> older, we only mkdir for container_path which is relative path
> (not
> > >> >>>> starting with "/").
> > >> >>>> Because if no rootfs specified for a mesos container, the
> container
> > >> >>> shares
> > >> >>>> the host
> > >> >>>> root filesystem. Obviously we don't want any random files to be
> > >> created
> > >> >>>> implicitly
> > >> >>>> on your host fs.
> > >> >>>> From mesos 0.29 (release by the end of this month), we will
> mkdir the
> > >> >>> mount
> > >> >>>> point in the container except for the command task case that
> specify
> > >> an
> > >> >>>> absolute
> > >> >>>> container_path without a rootfs. Because we simplify the mounting
> > >> logic,
> > >> >>> and
> > >> >>>> sandbox bind mount will only be done in container mount namespace
> > >> >>> instead of
> > >> >>>> host mount namespace (what we did before). Please keep tuned.
> > >> >>>>
> > >> >>>> Cheers,
> > >> >>>> Gilbert
> > >> >>>>
> > >> >>>> On Wed, May 18, 2016 at 8:14 AM, Joshua Cohen <jcohen@apache.org
> >
> > >> wrote:
> > >> >>>>
> > >> >>>>> Hi Olivier,
> > >> >>>>>
> > >> >>>>> I touched on this issue as part of
> > >> >>>>> https://issues.apache.org/jira/browse/MESOS-5229. It would be
> nice
> > >> if
> > >> >>>>> Mesos
> > >> >>>>> automatically created container mount points if they don't
> already
> > >> >>> exist.
> > >> >>>>> In the meantime, as a workaround for this, I've updated my
> > >> filesystem
> > >> >>>>> images to include the path (e.g. in Dockerfile, add `RUN mkdir
> -p
> > >> >>>>> /some/mount/point`). Not the best solution, but the only thing
> I've
> > >> >>> seen
> > >> >>>>> that works at the moment.
> > >> >>>>>
> > >> >>>>> Cheers,
> > >> >>>>>
> > >> >>>>> Joshua
> > >> >>>>>
> > >> >>>>> On Wed, May 18, 2016 at 7:36 AM, Guangya Liu <
> gyliu513@gmail.com>
> > >> >>> wrote:
> > >> >>>>>> It's pretty simple for you from scratch with source code
> > >> >>>>>>
> > >> >>>>>>
> > >> >>>
> > >>
> https://github.com/apache/mesos/blob/master/docs/getting-started.md#building-mesos
> > >> >>>>>> ;-)
> > >> >>>>>>
> > >> >>>>>> Thanks,
> > >> >>>>>>
> > >> >>>>>> Guangya
> > >> >>>>>>
> > >> >>>>>> On Wed, May 18, 2016 at 8:30 PM, Olivier Sallou <
> > >> >>> olivier.sallou@irisa.fr
> > >> >>>>>> wrote:
> > >> >>>>>>
> > >> >>>>>>> On 05/18/2016 02:31 PM, Guangya Liu wrote:
> > >> >>>>>>>> Just saw that you are working with 0.28.1, the "docker volume
> > >> >>> driver"
> > >> >>>>>>> code
> > >> >>>>>>>> was not in 0.28.1, can you please have a try with mesos
> master
> > >> >>> branch
> > >> >>>>>> if
> > >> >>>>>>>> you are only doing some test?
> > >> >>>>>>> this is indeed test only for the moment. But I will have to
> > >> >>>>>>> recompile/install mesos :-( (I used packages for install).
> > >> >>>>>>>
> > >> >>>>>>> I will try when possible, but thanks for the hint.
> > >> >>>>>>>> Thanks,
> > >> >>>>>>>>
> > >> >>>>>>>> Guangya
> > >> >>>>>>>>
> > >> >>>>>>>> On Wed, May 18, 2016 at 8:28 PM, Guangya Liu <
> gyliu513@gmail.com
> > >> >>>>>> wrote:
> > >> >>>>>>>>> Hi Olivier,
> > >> >>>>>>>>>
> > >> >>>>>>>>> I think that you need to enable "docker volume isolator" if
> you
> > >> >>> want
> > >> >>>>>> use
> > >> >>>>>>>>> external storage with unified container I was writing a
> document
> > >> >>>>> here
> > >> >>>>>>>>> https://reviews.apache.org/r/47511/, perhaps you can have
> a try
> > >> >>>>>>> according
> > >> >>>>>>>>> to the document and post some comments there if you find any
> > >> >>> issues.
> > >> >>>>>>>>> Also you can patch mesos-execute here
> > >> >>>>>>> https://reviews.apache.org/r/46762/ to
> > >> >>>>>>>>> have a try with mesos-execute.
> > >> >>>>>>>>>
> > >> >>>>>>>>> Thanks,
> > >> >>>>>>>>>
> > >> >>>>>>>>> Guangya
> > >> >>>>>>>>>
> > >> >>>>>>>>> On Wed, May 18, 2016 at 7:17 PM, Olivier Sallou <
> > >> >>>>>>> olivier.sallou@irisa.fr>
> > >> >>>>>>>>> wrote:
> > >> >>>>>>>>>
> > >> >>>>>>>>>> Answering (partially) to myself.
> > >> >>>>>>>>>>
> > >> >>>>>>>>>> I seems issue is container_path does not exists inside
> > >> >>> container.
> > >> >>>>> On
> > >> >>>>>>>>>> Docker, path is created and mounted. With pure mesos,
> > >> >>>>> container_path
> > >> >>>>>>>>>> must exists.
> > >> >>>>>>>>>>
> > >> >>>>>>>>>> mesos.proto says: "If the path is an absolute path, that
> path
> > >> >>> must
> > >> >>>>>>>>>> already exist."
> > >> >>>>>>>>>>
> > >> >>>>>>>>>> This is an issue however, using Docker images, the path I
> want
> > >> >>> to
> > >> >>>>>> mount
> > >> >>>>>>>>>> does not exists, and it cannot be modified "on the fly".
> > >> >>>>>>>>>>
> > >> >>>>>>>>>> Is there a workaround for this ?
> > >> >>>>>>>>>>
> > >> >>>>>>>>>>
> > >> >>>>>>>>>> On 05/18/2016 12:24 PM, Olivier Sallou wrote:
> > >> >>>>>>>>>>> Hi,
> > >> >>>>>>>>>>> I am trying unified containerizer on a single server
> > >> >>>>> (master/slave)
> > >> >>>>>> on
> > >> >>>>>>>>>>> mesos 0.28.1, to switch from docker containerizer to
> > >> >>> mesos+docker
> > >> >>>>>>> image
> > >> >>>>>>>>>>> container.
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>>> I have setup slave config as suggested in documentation:
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>>> containerizers=docker,mesos
> > >> >>>>>>>>>>> image_providers=docker \
> > >> >>>>>>>>>>> isolation=filesystem/linux,docker/runtime
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>>> However, when I execute my task with a volume I have an
> error:
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>>> ....
> > >> >>>>>>>>>>> + mount -n --rbind
> > >> >>>>>>>>>>>
> > >> >>>
> > >>
> /tmp/mesos/provisioner/containers/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/backends/copy/rootfses/f9f66bb2-308d-4555-ba77-49ec61cbeb4f
> > >> >>>
> > >>
> /tmp/mesos/slaves/2a296daf-7419-4659-ade1-763c792cd522-S0/frameworks/aef1b0e3-ea2d-4770-baac-96d673ab88f9-0000/executors/51/runs/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/.rootfs
> > >> >>>>>>>>>>> + mount -n --rbind
> > >> >>>>>>>>>>>
> > >> >>>
> > >>
> /home/osallou/Development/NOSAVE/go-docker/godshared/tasks/pairtree_root/us/er/_o/sa/ll/ou/task
> > >> >>>
> > >>
> /tmp/mesos/slaves/2a296daf-7419-4659-ade1-763c792cd522-S0/frameworks/aef1b0e3-ea2d-4770-baac-96d673ab88f9-0000/executors/51/runs/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/.rootfs/mnt/god-data
> > >> >>>>>>>>>>> mount: mount point
> > >> >>>>>>>>>>>
> > >> >>>
> > >>
> /tmp/mesos/slaves/2a296daf-7419-4659-ade1-763c792cd522-S0/frameworks/aef1b0e3-ea2d-4770-baac-96d673ab88f9-0000/executors/51/runs/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/.rootfs/mnt/god-data
> > >> >>>>>>>>>>> does not exist
> > >> >>>>>>>>>>> Failed to execute a preparation shell command
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>>> Then, my task switches to FAILED.
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>>> I define a local volume to bind mount in my "container"
> > >> >>>>>>>>>>>
> > >> >>>
> > >>
> /home/osallou/Development/NOSAVE/go-docker/godshared/tasks/pairtree_root/us/er/_o/sa/ll/ou/task
> > >> >>>>>>>>>>> => /mnt/god-data
> > >> >>>>>>>>>>> My directory exists on local server.
> > >> >>>>>>>>>>> In mesos UI, I can see the .rootfs directory along stdout
> and
> > >> >>>>> stderr
> > >> >>>>>>>>>>> files, and inside .rootfs, I can see /mnt/god-data
> (empty).
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>>> Running the same using Docker containerizer instead of
> mesos
> > >> >>>>>>>>>>> containerizer (with a Docker image) works fine.
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>>> It seems it fails to mount my local directory in the
> > >> >>> container.
> > >> >>>>> Any
> > >> >>>>>>> idea
> > >> >>>>>>>>>>> of what is going wrong or how to debug this?
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>>> Thanks
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>> --
> > >> >>>>>>>>>> Olivier Sallou
> > >> >>>>>>>>>> IRISA / University of Rennes 1
> > >> >>>>>>>>>> Campus de Beaulieu, 35000 RENNES - FRANCE
> > >> >>>>>>>>>> Tel: 02.99.84.71.95
> > >> >>>>>>>>>>
> > >> >>>>>>>>>> gpg key id: 4096R/326D8438 (keyring.debian.org)
> > >> >>>>>>>>>> Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB
> 326D
> > >> >>>>> 8438
> > >> >>>>>>> --
> > >> >>>>>>> Olivier Sallou
> > >> >>>>>>> IRISA / University of Rennes 1
> > >> >>>>>>> Campus de Beaulieu, 35000 RENNES - FRANCE
> > >> >>>>>>> Tel: 02.99.84.71.95
> > >> >>>>>>>
> > >> >>>>>>> gpg key id: 4096R/326D8438 (keyring.debian.org)
> > >> >>>>>>> Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB
> 326D
> > >> >>> 8438
> > >>
> > >> --
> > >> Olivier Sallou
> > >> IRISA / University of Rennes 1
> > >> Campus de Beaulieu, 35000 RENNES - FRANCE
> > >> Tel: 02.99.84.71.95
> > >>
> > >> gpg key id: 4096R/326D8438 (keyring.debian.org)
> > >> Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D 8438
> > >>
> > >>
> > >
> >
>
Re: volume / mount point error with Unified Containerizer
Posted by Olivier Sallou <ol...@irisa.fr>.
----- Mail original -----
> De: "Guangya Liu" <gy...@gmail.com>
> À: "dev" <de...@mesos.apache.org>, "Jie Yu" <ji...@mesosphere.io>, "Gilbert Song" <gi...@mesosphere.io>
> Envoyé: Lundi 23 Mai 2016 17:34:41
> Objet: Re: volume / mount point error with Unified Containerizer
>
> It is a bit strange to me, I also did some test and review code for
> relative path, and found that relative path works well.
>
> In 0.28.1, if deploy a docker container with MesosContaineirizer, then if
> using absolute path as continer_path, the mesos agent will update the
> container_path to a relative path by adding a prefix ./rootfs to the
> container_path, e.g. /file/path = > ./rootfs/file/path.
>
> If deploy a docker container with MesosContaineirizer with relative path as
> container_path, then the mesos agent will not update the container_path.
>
> So the final mount point for the container should be either
>
> 1) /tmp/mesos/slaves/agent_id/frameworks/framework_id/
> executors/51/runs/container_id/.rootfs/file/path
> 2) /tmp/mesos/slaves/agent_id/frameworks/framework_id/executors/51/runs/
> container_id/file/path
>
> The only difference is adding ./rootfs as a prefix or not, the test result
> is that 1) does not work and 2) works well. And even the mount for 1)
> failed, but I can see the mount point path does exist.
>
@Guangya
I confirm that using relative path works fine, I get volumes in mesos path (but it does not help for my implementation).
If I use the Docker containerizer, absolute paths are fine, this is what I use for the moment in my code, and am investigating to switch to unified container.
> @Yu Jie and @Gilbert, any comments for this?
>
> @Oilivier,
>
> In order not to block your test, can you please use mesos after 0.28.1? You
> can use either 0.28.2 or above version.
Well, as this is not an urgent matter, I am waiting 0.29 to test against this release (with other features I am waiting for).
>
> Thanks,
>
> Guangya
>
>
> On Mon, May 23, 2016 at 10:30 PM, Guangya Liu <gy...@gmail.com> wrote:
>
> > Thanks Olivier, I can reproduce this issue now and still checking what is
> > wrong.
> >
> > What I did is as following:
> > 1) Check out code with tag of 0.28.1
> > 2) update mesos-execute to add a host path volume
> > diff --git a/src/cli/execute.cpp b/src/cli/execute.cpp
> > index 81a0388..0ff913c 100644
> > --- a/src/cli/execute.cpp
> > +++ b/src/cli/execute.cpp
> > @@ -72,6 +72,8 @@ using mesos::v1::TaskID;
> > using mesos::v1::TaskInfo;
> > using mesos::v1::TaskState;
> > using mesos::v1::TaskStatus;
> > +using mesos::v1::Volume;
> > +using mesos::v1::Parameters;
> >
> > using mesos::v1::scheduler::Call;
> > using mesos::v1::scheduler::Event;
> > @@ -572,6 +574,12 @@ private:
> > }
> > }
> >
> > + Volume* volume1 = containerInfo.add_volumes();
> > + volume1->set_container_path("/tmp/abcd");
> > + volume1->set_mode(Volume::RW);
> > + volume1->set_host_path("/root/convoy");
> > + cout << "Add Voume 1" << endl;
> > +
> > return containerInfo;
> > } else if (containerizer == "docker") {
> > // 'docker' containerizer only supports 'docker' images.
> > 3) launch a task with docker image, task failed.
> >
> > 4) Check sandbox:
> > + /root/src/mesos/m1/mesos/build/src/mesos-containerizer mount
> > --help=false --operation=make-rslave --path=/
> > + grep -E /tmp/mesos/.+ /proc/self/mountinfo
> > + grep -v 3239aafc-78d8-4f70-81e5-f32fb3733339
> > + cut+ -d -f5
> > xargs --no-run-if-empty umount -l
> > + mount -n --rbind
> > /tmp/mesos/provisioner/containers/3239aafc-78d8-4f70-81e5-f32fb3733339/backends/copy/rootfses/5e8bf3fa-53b1-4bd5-bb3d-525ddc7900b6
> > /tmp/mesos/slaves/a4294ed5-10e8-47db-a3b9-a43a4f951374-S0/frameworks/a4294ed5-10e8-47db-a3b9-a43a4f951374-0000/executors/test/runs/3239aafc-78d8-4f70-81e5-f32fb3733339/.rootfs
> > + mount -n --rbind /root/convoy
> > /tmp/mesos/slaves/a4294ed5-10e8-47db-a3b9-a43a4f951374-S0/frameworks/a4294ed5-10e8-47db-a3b9-a43a4f951374-0000/executors/test/runs/3239aafc-78d8-4f70-81e5-f32fb3733339/.rootfs/tmp/abcd
> > mount: mount point
> > /tmp/mesos/slaves/a4294ed5-10e8-47db-a3b9-a43a4f951374-S0/frameworks/a4294ed5-10e8-47db-a3b9-a43a4f951374-0000/executors/test/runs/3239aafc-78d8-4f70-81e5-f32fb3733339/.rootfs/tmp/abcd
> > does not exist
> > Failed to execute a preparation shell command
> >
> > Will check more for this.
> >
> > Thanks,
> >
> > Guangya
> >
> > On Mon, May 23, 2016 at 3:35 PM, Olivier Sallou <ol...@irisa.fr>
> > wrote:
> >
> >>
> >>
> >> On 05/23/2016 09:33 AM, Olivier Sallou wrote:
> >> >
> >> > On 05/20/2016 03:26 PM, Guangya Liu wrote:
> >> >> Since you are using docker image which means that your container will
> >> have
> >> >> rootfs, so it is not required to have the absolute path exist, the
> >> linux
> >> >> file system isolator will help create the path automatically
> >> >>
> >> https://github.com/apache/mesos/blob/0.28.x/src/slave/containerizer/mesos/isolators/filesystem/linux.cpp#L390-L402
> >> >>
> >> >> Can you please share your framework? How did you set the volume part in
> >> >> your framework?
> >> > @Guangya
> >> >
> >> > I use Python API.
> >> >
> >> > Here is related code:
> >> >
> >> > ....
> >> > # Define container volumes
> >> > for v in job['container']['volumes']:
> >> > volume = container.volumes.add()
> >> > volume.container_path = v['mount']
> >> > volume.host_path = v['path']
> >> > if v['acl'] == 'rw':
> >> > volume.mode = 1 # mesos_pb2.Volume.Mode.RW
> >> > else:
> >> > volume.mode = 2 # mesos_pb2.Volume.Mode.RO
> >> >
> >> > => In my test case, I add 2 volumes from a host shared directory,
> >> > mounted in container as /mnt/go-docker and /mnt/god-data.
> >> >
> >> > ...
> >> > # Define docker image and network
> >> > docker = mesos_pb2.ContainerInfo.MesosInfo()
> >> > docker.image.type = 2 # Docker
> >> > docker.image.docker.name ='centos:latest'
> >> > # Request an IP from a network module
> >> > network_info = container.network_infos.add()
> >> > network_info_name = 'sampletest'
> >> > # Get an IP V4 address
> >> > ip_address = network_info.ip_addresses.add()
> >> > ip_address.protocol = 1
> >> > # The network group to join
> >> > group = network_info.groups.append(network_info_name)
> >> > port_list = [22]
> >> > if port_list:
> >> > for port in port_list:
> >> > job['container']['port_mapping'].append({'host':
> >> > port, 'container': port})
> >> > container.mesos.MergeFrom(docker)
> >> >
> >> > It results in error message:
> >> >
> >> > + mount -n --rbind
> >> >
> >> /tmp/mesos/provisioner/containers/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/backends/copy/rootfses/f9f66bb2-308d-4555-ba77-49ec61cbeb4f
> >> >
> >> /tmp/mesos/slaves/2a296daf-7419-4659-ade1-763c792cd522-S0/frameworks/aef1b0e3-ea2d-4770-baac-96d673ab88f9-0000/executors/51/runs/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/.rootfs
> >> > + mount -n --rbind
> >> >
> >> /home/osallou/Development/NOSAVE/go-docker/godshared/tasks/pairtree_root/us/er/_o/sa/ll/ou/task
> >> >
> >> /tmp/mesos/slaves/2a296daf-7419-4659-ade1-763c792cd522-S0/frameworks/aef1b0e3-ea2d-4770-baac-96d673ab88f9-0000/executors/51/runs/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/.rootfs/mnt/god-data
> >> > mount: mount point
> >> >
> >> /tmp/mesos/slaves/2a296daf-7419-4659-ade1-763c792cd522-S0/frameworks/aef1b0e3-ea2d-4770-baac-96d673ab88f9-0000/executors/51/runs/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/.rootfs/mnt/god-data
> >> > does not exist
> >> > Failed to execute a preparation shell command
> >> >
> >> >
> >> > We can see the .rootfs, but mnt/god-data under .rootfs fails. Local
> >> > directory exists, it does not pre-exists in container. What is strange
> >> > is , if I look in mesos task dir, .rootfs/mnt/go-data, directory is
> >> present.
> >> >
> >> > Or, is the error message (.rootfs/mnt/god-data does not exist) simply a
> >> > warning, and it creates it, and final error (Failed to execute a
> >> > preparation shell command) not related (and unclear...)
> >> Additional info: command to execute in container is located in one of
> >> mounted volume.
> >> >
> >> > Olivier
> >> >> Thanks,
> >> >>
> >> >> Guangya
> >> >>
> >> >> On Fri, May 20, 2016 at 4:54 AM, Olivier Sallou <
> >> olivier.sallou@irisa.fr>
> >> >> wrote:
> >> >>
> >> >>> ----- Mail original -----
> >> >>>> De: "Gilbert Song" <gi...@mesosphere.io>
> >> >>>> À: "dev" <de...@mesos.apache.org>
> >> >>>> Envoyé: Jeudi 19 Mai 2016 01:57:16
> >> >>>> Objet: Re: volume / mount point error with Unified Containerizer
> >> >>>>
> >> >>>> @Olivier,
> >> >>>> In mesos 0.28.1, you are supposed to be able bind mount a volume from
> >> >>>> the host into the mesos container. Did you specify a docker image (we
> >> >>>> determine
> >> >>>> the mount point differently depending whether the container has a
> >> >>> rootfs)?
> >> >>>
> >> >>> Yes I specified an image, a Docker image URI.
> >> >>>
> >> >>>> How
> >> >>>> do you specify your 'container_path' (the mount point in the
> >> container)?
> >> >>> If
> >> >>>> it is an
> >> >>>> absolute path, we require that dir to be pre-existed. If it is a
> >> relative
> >> >>>> path, we will
> >> >>>> mkdir for it.
> >> >>> It is an absolute path, but it does not exists in image (this is the
> >> >>> issue). Images are custom Docker images (images containing tools for
> >> batch
> >> >>> computing), and I want, for example, to mount some shared resources
> >> (user
> >> >>> home dir, common data, etc.) in the image. Of course those
> >> directories do
> >> >>> not pre-exists in container images as they are specific to the
> >> environment.
> >> >>> Requiring existence of the directory in the image is not issue as it
> >> >>> prevents using any existing image from a repo.
> >> >>>
> >> >>> When using Docker containerizer it works fine, I can mount any
> >> external
> >> >>> storage in the container.
> >> >>>
> >> >>> Olivie
> >> >>>
> >> >>>
> >> >>>> @Joshua,
> >> >>>> Thank for posting your workaround on mesos. As I mentioned above, in
> >> >>> 0.28.1
> >> >>>> or
> >> >>>> older, we only mkdir for container_path which is relative path (not
> >> >>>> starting with "/").
> >> >>>> Because if no rootfs specified for a mesos container, the container
> >> >>> shares
> >> >>>> the host
> >> >>>> root filesystem. Obviously we don't want any random files to be
> >> created
> >> >>>> implicitly
> >> >>>> on your host fs.
> >> >>>> From mesos 0.29 (release by the end of this month), we will mkdir the
> >> >>> mount
> >> >>>> point in the container except for the command task case that specify
> >> an
> >> >>>> absolute
> >> >>>> container_path without a rootfs. Because we simplify the mounting
> >> logic,
> >> >>> and
> >> >>>> sandbox bind mount will only be done in container mount namespace
> >> >>> instead of
> >> >>>> host mount namespace (what we did before). Please keep tuned.
> >> >>>>
> >> >>>> Cheers,
> >> >>>> Gilbert
> >> >>>>
> >> >>>> On Wed, May 18, 2016 at 8:14 AM, Joshua Cohen <jc...@apache.org>
> >> wrote:
> >> >>>>
> >> >>>>> Hi Olivier,
> >> >>>>>
> >> >>>>> I touched on this issue as part of
> >> >>>>> https://issues.apache.org/jira/browse/MESOS-5229. It would be nice
> >> if
> >> >>>>> Mesos
> >> >>>>> automatically created container mount points if they don't already
> >> >>> exist.
> >> >>>>> In the meantime, as a workaround for this, I've updated my
> >> filesystem
> >> >>>>> images to include the path (e.g. in Dockerfile, add `RUN mkdir -p
> >> >>>>> /some/mount/point`). Not the best solution, but the only thing I've
> >> >>> seen
> >> >>>>> that works at the moment.
> >> >>>>>
> >> >>>>> Cheers,
> >> >>>>>
> >> >>>>> Joshua
> >> >>>>>
> >> >>>>> On Wed, May 18, 2016 at 7:36 AM, Guangya Liu <gy...@gmail.com>
> >> >>> wrote:
> >> >>>>>> It's pretty simple for you from scratch with source code
> >> >>>>>>
> >> >>>>>>
> >> >>>
> >> https://github.com/apache/mesos/blob/master/docs/getting-started.md#building-mesos
> >> >>>>>> ;-)
> >> >>>>>>
> >> >>>>>> Thanks,
> >> >>>>>>
> >> >>>>>> Guangya
> >> >>>>>>
> >> >>>>>> On Wed, May 18, 2016 at 8:30 PM, Olivier Sallou <
> >> >>> olivier.sallou@irisa.fr
> >> >>>>>> wrote:
> >> >>>>>>
> >> >>>>>>> On 05/18/2016 02:31 PM, Guangya Liu wrote:
> >> >>>>>>>> Just saw that you are working with 0.28.1, the "docker volume
> >> >>> driver"
> >> >>>>>>> code
> >> >>>>>>>> was not in 0.28.1, can you please have a try with mesos master
> >> >>> branch
> >> >>>>>> if
> >> >>>>>>>> you are only doing some test?
> >> >>>>>>> this is indeed test only for the moment. But I will have to
> >> >>>>>>> recompile/install mesos :-( (I used packages for install).
> >> >>>>>>>
> >> >>>>>>> I will try when possible, but thanks for the hint.
> >> >>>>>>>> Thanks,
> >> >>>>>>>>
> >> >>>>>>>> Guangya
> >> >>>>>>>>
> >> >>>>>>>> On Wed, May 18, 2016 at 8:28 PM, Guangya Liu <gyliu513@gmail.com
> >> >>>>>> wrote:
> >> >>>>>>>>> Hi Olivier,
> >> >>>>>>>>>
> >> >>>>>>>>> I think that you need to enable "docker volume isolator" if you
> >> >>> want
> >> >>>>>> use
> >> >>>>>>>>> external storage with unified container I was writing a document
> >> >>>>> here
> >> >>>>>>>>> https://reviews.apache.org/r/47511/, perhaps you can have a try
> >> >>>>>>> according
> >> >>>>>>>>> to the document and post some comments there if you find any
> >> >>> issues.
> >> >>>>>>>>> Also you can patch mesos-execute here
> >> >>>>>>> https://reviews.apache.org/r/46762/ to
> >> >>>>>>>>> have a try with mesos-execute.
> >> >>>>>>>>>
> >> >>>>>>>>> Thanks,
> >> >>>>>>>>>
> >> >>>>>>>>> Guangya
> >> >>>>>>>>>
> >> >>>>>>>>> On Wed, May 18, 2016 at 7:17 PM, Olivier Sallou <
> >> >>>>>>> olivier.sallou@irisa.fr>
> >> >>>>>>>>> wrote:
> >> >>>>>>>>>
> >> >>>>>>>>>> Answering (partially) to myself.
> >> >>>>>>>>>>
> >> >>>>>>>>>> I seems issue is container_path does not exists inside
> >> >>> container.
> >> >>>>> On
> >> >>>>>>>>>> Docker, path is created and mounted. With pure mesos,
> >> >>>>> container_path
> >> >>>>>>>>>> must exists.
> >> >>>>>>>>>>
> >> >>>>>>>>>> mesos.proto says: "If the path is an absolute path, that path
> >> >>> must
> >> >>>>>>>>>> already exist."
> >> >>>>>>>>>>
> >> >>>>>>>>>> This is an issue however, using Docker images, the path I want
> >> >>> to
> >> >>>>>> mount
> >> >>>>>>>>>> does not exists, and it cannot be modified "on the fly".
> >> >>>>>>>>>>
> >> >>>>>>>>>> Is there a workaround for this ?
> >> >>>>>>>>>>
> >> >>>>>>>>>>
> >> >>>>>>>>>> On 05/18/2016 12:24 PM, Olivier Sallou wrote:
> >> >>>>>>>>>>> Hi,
> >> >>>>>>>>>>> I am trying unified containerizer on a single server
> >> >>>>> (master/slave)
> >> >>>>>> on
> >> >>>>>>>>>>> mesos 0.28.1, to switch from docker containerizer to
> >> >>> mesos+docker
> >> >>>>>>> image
> >> >>>>>>>>>>> container.
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> I have setup slave config as suggested in documentation:
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> containerizers=docker,mesos
> >> >>>>>>>>>>> image_providers=docker \
> >> >>>>>>>>>>> isolation=filesystem/linux,docker/runtime
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> However, when I execute my task with a volume I have an error:
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> ....
> >> >>>>>>>>>>> + mount -n --rbind
> >> >>>>>>>>>>>
> >> >>>
> >> /tmp/mesos/provisioner/containers/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/backends/copy/rootfses/f9f66bb2-308d-4555-ba77-49ec61cbeb4f
> >> >>>
> >> /tmp/mesos/slaves/2a296daf-7419-4659-ade1-763c792cd522-S0/frameworks/aef1b0e3-ea2d-4770-baac-96d673ab88f9-0000/executors/51/runs/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/.rootfs
> >> >>>>>>>>>>> + mount -n --rbind
> >> >>>>>>>>>>>
> >> >>>
> >> /home/osallou/Development/NOSAVE/go-docker/godshared/tasks/pairtree_root/us/er/_o/sa/ll/ou/task
> >> >>>
> >> /tmp/mesos/slaves/2a296daf-7419-4659-ade1-763c792cd522-S0/frameworks/aef1b0e3-ea2d-4770-baac-96d673ab88f9-0000/executors/51/runs/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/.rootfs/mnt/god-data
> >> >>>>>>>>>>> mount: mount point
> >> >>>>>>>>>>>
> >> >>>
> >> /tmp/mesos/slaves/2a296daf-7419-4659-ade1-763c792cd522-S0/frameworks/aef1b0e3-ea2d-4770-baac-96d673ab88f9-0000/executors/51/runs/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/.rootfs/mnt/god-data
> >> >>>>>>>>>>> does not exist
> >> >>>>>>>>>>> Failed to execute a preparation shell command
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> Then, my task switches to FAILED.
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> I define a local volume to bind mount in my "container"
> >> >>>>>>>>>>>
> >> >>>
> >> /home/osallou/Development/NOSAVE/go-docker/godshared/tasks/pairtree_root/us/er/_o/sa/ll/ou/task
> >> >>>>>>>>>>> => /mnt/god-data
> >> >>>>>>>>>>> My directory exists on local server.
> >> >>>>>>>>>>> In mesos UI, I can see the .rootfs directory along stdout and
> >> >>>>> stderr
> >> >>>>>>>>>>> files, and inside .rootfs, I can see /mnt/god-data (empty).
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> Running the same using Docker containerizer instead of mesos
> >> >>>>>>>>>>> containerizer (with a Docker image) works fine.
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> It seems it fails to mount my local directory in the
> >> >>> container.
> >> >>>>> Any
> >> >>>>>>> idea
> >> >>>>>>>>>>> of what is going wrong or how to debug this?
> >> >>>>>>>>>>>
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> Thanks
> >> >>>>>>>>>>>
> >> >>>>>>>>>> --
> >> >>>>>>>>>> Olivier Sallou
> >> >>>>>>>>>> IRISA / University of Rennes 1
> >> >>>>>>>>>> Campus de Beaulieu, 35000 RENNES - FRANCE
> >> >>>>>>>>>> Tel: 02.99.84.71.95
> >> >>>>>>>>>>
> >> >>>>>>>>>> gpg key id: 4096R/326D8438 (keyring.debian.org)
> >> >>>>>>>>>> Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D
> >> >>>>> 8438
> >> >>>>>>> --
> >> >>>>>>> Olivier Sallou
> >> >>>>>>> IRISA / University of Rennes 1
> >> >>>>>>> Campus de Beaulieu, 35000 RENNES - FRANCE
> >> >>>>>>> Tel: 02.99.84.71.95
> >> >>>>>>>
> >> >>>>>>> gpg key id: 4096R/326D8438 (keyring.debian.org)
> >> >>>>>>> Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D
> >> >>> 8438
> >>
> >> --
> >> Olivier Sallou
> >> IRISA / University of Rennes 1
> >> Campus de Beaulieu, 35000 RENNES - FRANCE
> >> Tel: 02.99.84.71.95
> >>
> >> gpg key id: 4096R/326D8438 (keyring.debian.org)
> >> Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D 8438
> >>
> >>
> >
>
Re: volume / mount point error with Unified Containerizer
Posted by Guangya Liu <gy...@gmail.com>.
It is a bit strange to me, I also did some test and review code for
relative path, and found that relative path works well.
In 0.28.1, if deploy a docker container with MesosContaineirizer, then if
using absolute path as continer_path, the mesos agent will update the
container_path to a relative path by adding a prefix ./rootfs to the
container_path, e.g. /file/path = > ./rootfs/file/path.
If deploy a docker container with MesosContaineirizer with relative path as
container_path, then the mesos agent will not update the container_path.
So the final mount point for the container should be either
1) /tmp/mesos/slaves/agent_id/frameworks/framework_id/
executors/51/runs/container_id/.rootfs/file/path
2) /tmp/mesos/slaves/agent_id/frameworks/framework_id/executors/51/runs/
container_id/file/path
The only difference is adding ./rootfs as a prefix or not, the test result
is that 1) does not work and 2) works well. And even the mount for 1)
failed, but I can see the mount point path does exist.
@Yu Jie and @Gilbert, any comments for this?
@Oilivier,
In order not to block your test, can you please use mesos after 0.28.1? You
can use either 0.28.2 or above version.
Thanks,
Guangya
On Mon, May 23, 2016 at 10:30 PM, Guangya Liu <gy...@gmail.com> wrote:
> Thanks Olivier, I can reproduce this issue now and still checking what is
> wrong.
>
> What I did is as following:
> 1) Check out code with tag of 0.28.1
> 2) update mesos-execute to add a host path volume
> diff --git a/src/cli/execute.cpp b/src/cli/execute.cpp
> index 81a0388..0ff913c 100644
> --- a/src/cli/execute.cpp
> +++ b/src/cli/execute.cpp
> @@ -72,6 +72,8 @@ using mesos::v1::TaskID;
> using mesos::v1::TaskInfo;
> using mesos::v1::TaskState;
> using mesos::v1::TaskStatus;
> +using mesos::v1::Volume;
> +using mesos::v1::Parameters;
>
> using mesos::v1::scheduler::Call;
> using mesos::v1::scheduler::Event;
> @@ -572,6 +574,12 @@ private:
> }
> }
>
> + Volume* volume1 = containerInfo.add_volumes();
> + volume1->set_container_path("/tmp/abcd");
> + volume1->set_mode(Volume::RW);
> + volume1->set_host_path("/root/convoy");
> + cout << "Add Voume 1" << endl;
> +
> return containerInfo;
> } else if (containerizer == "docker") {
> // 'docker' containerizer only supports 'docker' images.
> 3) launch a task with docker image, task failed.
>
> 4) Check sandbox:
> + /root/src/mesos/m1/mesos/build/src/mesos-containerizer mount
> --help=false --operation=make-rslave --path=/
> + grep -E /tmp/mesos/.+ /proc/self/mountinfo
> + grep -v 3239aafc-78d8-4f70-81e5-f32fb3733339
> + cut+ -d -f5
> xargs --no-run-if-empty umount -l
> + mount -n --rbind
> /tmp/mesos/provisioner/containers/3239aafc-78d8-4f70-81e5-f32fb3733339/backends/copy/rootfses/5e8bf3fa-53b1-4bd5-bb3d-525ddc7900b6
> /tmp/mesos/slaves/a4294ed5-10e8-47db-a3b9-a43a4f951374-S0/frameworks/a4294ed5-10e8-47db-a3b9-a43a4f951374-0000/executors/test/runs/3239aafc-78d8-4f70-81e5-f32fb3733339/.rootfs
> + mount -n --rbind /root/convoy
> /tmp/mesos/slaves/a4294ed5-10e8-47db-a3b9-a43a4f951374-S0/frameworks/a4294ed5-10e8-47db-a3b9-a43a4f951374-0000/executors/test/runs/3239aafc-78d8-4f70-81e5-f32fb3733339/.rootfs/tmp/abcd
> mount: mount point
> /tmp/mesos/slaves/a4294ed5-10e8-47db-a3b9-a43a4f951374-S0/frameworks/a4294ed5-10e8-47db-a3b9-a43a4f951374-0000/executors/test/runs/3239aafc-78d8-4f70-81e5-f32fb3733339/.rootfs/tmp/abcd
> does not exist
> Failed to execute a preparation shell command
>
> Will check more for this.
>
> Thanks,
>
> Guangya
>
> On Mon, May 23, 2016 at 3:35 PM, Olivier Sallou <ol...@irisa.fr>
> wrote:
>
>>
>>
>> On 05/23/2016 09:33 AM, Olivier Sallou wrote:
>> >
>> > On 05/20/2016 03:26 PM, Guangya Liu wrote:
>> >> Since you are using docker image which means that your container will
>> have
>> >> rootfs, so it is not required to have the absolute path exist, the
>> linux
>> >> file system isolator will help create the path automatically
>> >>
>> https://github.com/apache/mesos/blob/0.28.x/src/slave/containerizer/mesos/isolators/filesystem/linux.cpp#L390-L402
>> >>
>> >> Can you please share your framework? How did you set the volume part in
>> >> your framework?
>> > @Guangya
>> >
>> > I use Python API.
>> >
>> > Here is related code:
>> >
>> > ....
>> > # Define container volumes
>> > for v in job['container']['volumes']:
>> > volume = container.volumes.add()
>> > volume.container_path = v['mount']
>> > volume.host_path = v['path']
>> > if v['acl'] == 'rw':
>> > volume.mode = 1 # mesos_pb2.Volume.Mode.RW
>> > else:
>> > volume.mode = 2 # mesos_pb2.Volume.Mode.RO
>> >
>> > => In my test case, I add 2 volumes from a host shared directory,
>> > mounted in container as /mnt/go-docker and /mnt/god-data.
>> >
>> > ...
>> > # Define docker image and network
>> > docker = mesos_pb2.ContainerInfo.MesosInfo()
>> > docker.image.type = 2 # Docker
>> > docker.image.docker.name ='centos:latest'
>> > # Request an IP from a network module
>> > network_info = container.network_infos.add()
>> > network_info_name = 'sampletest'
>> > # Get an IP V4 address
>> > ip_address = network_info.ip_addresses.add()
>> > ip_address.protocol = 1
>> > # The network group to join
>> > group = network_info.groups.append(network_info_name)
>> > port_list = [22]
>> > if port_list:
>> > for port in port_list:
>> > job['container']['port_mapping'].append({'host':
>> > port, 'container': port})
>> > container.mesos.MergeFrom(docker)
>> >
>> > It results in error message:
>> >
>> > + mount -n --rbind
>> >
>> /tmp/mesos/provisioner/containers/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/backends/copy/rootfses/f9f66bb2-308d-4555-ba77-49ec61cbeb4f
>> >
>> /tmp/mesos/slaves/2a296daf-7419-4659-ade1-763c792cd522-S0/frameworks/aef1b0e3-ea2d-4770-baac-96d673ab88f9-0000/executors/51/runs/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/.rootfs
>> > + mount -n --rbind
>> >
>> /home/osallou/Development/NOSAVE/go-docker/godshared/tasks/pairtree_root/us/er/_o/sa/ll/ou/task
>> >
>> /tmp/mesos/slaves/2a296daf-7419-4659-ade1-763c792cd522-S0/frameworks/aef1b0e3-ea2d-4770-baac-96d673ab88f9-0000/executors/51/runs/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/.rootfs/mnt/god-data
>> > mount: mount point
>> >
>> /tmp/mesos/slaves/2a296daf-7419-4659-ade1-763c792cd522-S0/frameworks/aef1b0e3-ea2d-4770-baac-96d673ab88f9-0000/executors/51/runs/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/.rootfs/mnt/god-data
>> > does not exist
>> > Failed to execute a preparation shell command
>> >
>> >
>> > We can see the .rootfs, but mnt/god-data under .rootfs fails. Local
>> > directory exists, it does not pre-exists in container. What is strange
>> > is , if I look in mesos task dir, .rootfs/mnt/go-data, directory is
>> present.
>> >
>> > Or, is the error message (.rootfs/mnt/god-data does not exist) simply a
>> > warning, and it creates it, and final error (Failed to execute a
>> > preparation shell command) not related (and unclear...)
>> Additional info: command to execute in container is located in one of
>> mounted volume.
>> >
>> > Olivier
>> >> Thanks,
>> >>
>> >> Guangya
>> >>
>> >> On Fri, May 20, 2016 at 4:54 AM, Olivier Sallou <
>> olivier.sallou@irisa.fr>
>> >> wrote:
>> >>
>> >>> ----- Mail original -----
>> >>>> De: "Gilbert Song" <gi...@mesosphere.io>
>> >>>> À: "dev" <de...@mesos.apache.org>
>> >>>> Envoyé: Jeudi 19 Mai 2016 01:57:16
>> >>>> Objet: Re: volume / mount point error with Unified Containerizer
>> >>>>
>> >>>> @Olivier,
>> >>>> In mesos 0.28.1, you are supposed to be able bind mount a volume from
>> >>>> the host into the mesos container. Did you specify a docker image (we
>> >>>> determine
>> >>>> the mount point differently depending whether the container has a
>> >>> rootfs)?
>> >>>
>> >>> Yes I specified an image, a Docker image URI.
>> >>>
>> >>>> How
>> >>>> do you specify your 'container_path' (the mount point in the
>> container)?
>> >>> If
>> >>>> it is an
>> >>>> absolute path, we require that dir to be pre-existed. If it is a
>> relative
>> >>>> path, we will
>> >>>> mkdir for it.
>> >>> It is an absolute path, but it does not exists in image (this is the
>> >>> issue). Images are custom Docker images (images containing tools for
>> batch
>> >>> computing), and I want, for example, to mount some shared resources
>> (user
>> >>> home dir, common data, etc.) in the image. Of course those
>> directories do
>> >>> not pre-exists in container images as they are specific to the
>> environment.
>> >>> Requiring existence of the directory in the image is not issue as it
>> >>> prevents using any existing image from a repo.
>> >>>
>> >>> When using Docker containerizer it works fine, I can mount any
>> external
>> >>> storage in the container.
>> >>>
>> >>> Olivie
>> >>>
>> >>>
>> >>>> @Joshua,
>> >>>> Thank for posting your workaround on mesos. As I mentioned above, in
>> >>> 0.28.1
>> >>>> or
>> >>>> older, we only mkdir for container_path which is relative path (not
>> >>>> starting with "/").
>> >>>> Because if no rootfs specified for a mesos container, the container
>> >>> shares
>> >>>> the host
>> >>>> root filesystem. Obviously we don't want any random files to be
>> created
>> >>>> implicitly
>> >>>> on your host fs.
>> >>>> From mesos 0.29 (release by the end of this month), we will mkdir the
>> >>> mount
>> >>>> point in the container except for the command task case that specify
>> an
>> >>>> absolute
>> >>>> container_path without a rootfs. Because we simplify the mounting
>> logic,
>> >>> and
>> >>>> sandbox bind mount will only be done in container mount namespace
>> >>> instead of
>> >>>> host mount namespace (what we did before). Please keep tuned.
>> >>>>
>> >>>> Cheers,
>> >>>> Gilbert
>> >>>>
>> >>>> On Wed, May 18, 2016 at 8:14 AM, Joshua Cohen <jc...@apache.org>
>> wrote:
>> >>>>
>> >>>>> Hi Olivier,
>> >>>>>
>> >>>>> I touched on this issue as part of
>> >>>>> https://issues.apache.org/jira/browse/MESOS-5229. It would be nice
>> if
>> >>>>> Mesos
>> >>>>> automatically created container mount points if they don't already
>> >>> exist.
>> >>>>> In the meantime, as a workaround for this, I've updated my
>> filesystem
>> >>>>> images to include the path (e.g. in Dockerfile, add `RUN mkdir -p
>> >>>>> /some/mount/point`). Not the best solution, but the only thing I've
>> >>> seen
>> >>>>> that works at the moment.
>> >>>>>
>> >>>>> Cheers,
>> >>>>>
>> >>>>> Joshua
>> >>>>>
>> >>>>> On Wed, May 18, 2016 at 7:36 AM, Guangya Liu <gy...@gmail.com>
>> >>> wrote:
>> >>>>>> It's pretty simple for you from scratch with source code
>> >>>>>>
>> >>>>>>
>> >>>
>> https://github.com/apache/mesos/blob/master/docs/getting-started.md#building-mesos
>> >>>>>> ;-)
>> >>>>>>
>> >>>>>> Thanks,
>> >>>>>>
>> >>>>>> Guangya
>> >>>>>>
>> >>>>>> On Wed, May 18, 2016 at 8:30 PM, Olivier Sallou <
>> >>> olivier.sallou@irisa.fr
>> >>>>>> wrote:
>> >>>>>>
>> >>>>>>> On 05/18/2016 02:31 PM, Guangya Liu wrote:
>> >>>>>>>> Just saw that you are working with 0.28.1, the "docker volume
>> >>> driver"
>> >>>>>>> code
>> >>>>>>>> was not in 0.28.1, can you please have a try with mesos master
>> >>> branch
>> >>>>>> if
>> >>>>>>>> you are only doing some test?
>> >>>>>>> this is indeed test only for the moment. But I will have to
>> >>>>>>> recompile/install mesos :-( (I used packages for install).
>> >>>>>>>
>> >>>>>>> I will try when possible, but thanks for the hint.
>> >>>>>>>> Thanks,
>> >>>>>>>>
>> >>>>>>>> Guangya
>> >>>>>>>>
>> >>>>>>>> On Wed, May 18, 2016 at 8:28 PM, Guangya Liu <gyliu513@gmail.com
>> >>>>>> wrote:
>> >>>>>>>>> Hi Olivier,
>> >>>>>>>>>
>> >>>>>>>>> I think that you need to enable "docker volume isolator" if you
>> >>> want
>> >>>>>> use
>> >>>>>>>>> external storage with unified container I was writing a document
>> >>>>> here
>> >>>>>>>>> https://reviews.apache.org/r/47511/, perhaps you can have a try
>> >>>>>>> according
>> >>>>>>>>> to the document and post some comments there if you find any
>> >>> issues.
>> >>>>>>>>> Also you can patch mesos-execute here
>> >>>>>>> https://reviews.apache.org/r/46762/ to
>> >>>>>>>>> have a try with mesos-execute.
>> >>>>>>>>>
>> >>>>>>>>> Thanks,
>> >>>>>>>>>
>> >>>>>>>>> Guangya
>> >>>>>>>>>
>> >>>>>>>>> On Wed, May 18, 2016 at 7:17 PM, Olivier Sallou <
>> >>>>>>> olivier.sallou@irisa.fr>
>> >>>>>>>>> wrote:
>> >>>>>>>>>
>> >>>>>>>>>> Answering (partially) to myself.
>> >>>>>>>>>>
>> >>>>>>>>>> I seems issue is container_path does not exists inside
>> >>> container.
>> >>>>> On
>> >>>>>>>>>> Docker, path is created and mounted. With pure mesos,
>> >>>>> container_path
>> >>>>>>>>>> must exists.
>> >>>>>>>>>>
>> >>>>>>>>>> mesos.proto says: "If the path is an absolute path, that path
>> >>> must
>> >>>>>>>>>> already exist."
>> >>>>>>>>>>
>> >>>>>>>>>> This is an issue however, using Docker images, the path I want
>> >>> to
>> >>>>>> mount
>> >>>>>>>>>> does not exists, and it cannot be modified "on the fly".
>> >>>>>>>>>>
>> >>>>>>>>>> Is there a workaround for this ?
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>> On 05/18/2016 12:24 PM, Olivier Sallou wrote:
>> >>>>>>>>>>> Hi,
>> >>>>>>>>>>> I am trying unified containerizer on a single server
>> >>>>> (master/slave)
>> >>>>>> on
>> >>>>>>>>>>> mesos 0.28.1, to switch from docker containerizer to
>> >>> mesos+docker
>> >>>>>>> image
>> >>>>>>>>>>> container.
>> >>>>>>>>>>>
>> >>>>>>>>>>> I have setup slave config as suggested in documentation:
>> >>>>>>>>>>>
>> >>>>>>>>>>> containerizers=docker,mesos
>> >>>>>>>>>>> image_providers=docker \
>> >>>>>>>>>>> isolation=filesystem/linux,docker/runtime
>> >>>>>>>>>>>
>> >>>>>>>>>>> However, when I execute my task with a volume I have an error:
>> >>>>>>>>>>>
>> >>>>>>>>>>> ....
>> >>>>>>>>>>> + mount -n --rbind
>> >>>>>>>>>>>
>> >>>
>> /tmp/mesos/provisioner/containers/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/backends/copy/rootfses/f9f66bb2-308d-4555-ba77-49ec61cbeb4f
>> >>>
>> /tmp/mesos/slaves/2a296daf-7419-4659-ade1-763c792cd522-S0/frameworks/aef1b0e3-ea2d-4770-baac-96d673ab88f9-0000/executors/51/runs/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/.rootfs
>> >>>>>>>>>>> + mount -n --rbind
>> >>>>>>>>>>>
>> >>>
>> /home/osallou/Development/NOSAVE/go-docker/godshared/tasks/pairtree_root/us/er/_o/sa/ll/ou/task
>> >>>
>> /tmp/mesos/slaves/2a296daf-7419-4659-ade1-763c792cd522-S0/frameworks/aef1b0e3-ea2d-4770-baac-96d673ab88f9-0000/executors/51/runs/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/.rootfs/mnt/god-data
>> >>>>>>>>>>> mount: mount point
>> >>>>>>>>>>>
>> >>>
>> /tmp/mesos/slaves/2a296daf-7419-4659-ade1-763c792cd522-S0/frameworks/aef1b0e3-ea2d-4770-baac-96d673ab88f9-0000/executors/51/runs/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/.rootfs/mnt/god-data
>> >>>>>>>>>>> does not exist
>> >>>>>>>>>>> Failed to execute a preparation shell command
>> >>>>>>>>>>>
>> >>>>>>>>>>> Then, my task switches to FAILED.
>> >>>>>>>>>>>
>> >>>>>>>>>>> I define a local volume to bind mount in my "container"
>> >>>>>>>>>>>
>> >>>
>> /home/osallou/Development/NOSAVE/go-docker/godshared/tasks/pairtree_root/us/er/_o/sa/ll/ou/task
>> >>>>>>>>>>> => /mnt/god-data
>> >>>>>>>>>>> My directory exists on local server.
>> >>>>>>>>>>> In mesos UI, I can see the .rootfs directory along stdout and
>> >>>>> stderr
>> >>>>>>>>>>> files, and inside .rootfs, I can see /mnt/god-data (empty).
>> >>>>>>>>>>>
>> >>>>>>>>>>> Running the same using Docker containerizer instead of mesos
>> >>>>>>>>>>> containerizer (with a Docker image) works fine.
>> >>>>>>>>>>>
>> >>>>>>>>>>> It seems it fails to mount my local directory in the
>> >>> container.
>> >>>>> Any
>> >>>>>>> idea
>> >>>>>>>>>>> of what is going wrong or how to debug this?
>> >>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>> Thanks
>> >>>>>>>>>>>
>> >>>>>>>>>> --
>> >>>>>>>>>> Olivier Sallou
>> >>>>>>>>>> IRISA / University of Rennes 1
>> >>>>>>>>>> Campus de Beaulieu, 35000 RENNES - FRANCE
>> >>>>>>>>>> Tel: 02.99.84.71.95
>> >>>>>>>>>>
>> >>>>>>>>>> gpg key id: 4096R/326D8438 (keyring.debian.org)
>> >>>>>>>>>> Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D
>> >>>>> 8438
>> >>>>>>> --
>> >>>>>>> Olivier Sallou
>> >>>>>>> IRISA / University of Rennes 1
>> >>>>>>> Campus de Beaulieu, 35000 RENNES - FRANCE
>> >>>>>>> Tel: 02.99.84.71.95
>> >>>>>>>
>> >>>>>>> gpg key id: 4096R/326D8438 (keyring.debian.org)
>> >>>>>>> Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D
>> >>> 8438
>>
>> --
>> Olivier Sallou
>> IRISA / University of Rennes 1
>> Campus de Beaulieu, 35000 RENNES - FRANCE
>> Tel: 02.99.84.71.95
>>
>> gpg key id: 4096R/326D8438 (keyring.debian.org)
>> Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D 8438
>>
>>
>
Re: volume / mount point error with Unified Containerizer
Posted by Guangya Liu <gy...@gmail.com>.
Thanks Olivier, I can reproduce this issue now and still checking what is
wrong.
What I did is as following:
1) Check out code with tag of 0.28.1
2) update mesos-execute to add a host path volume
diff --git a/src/cli/execute.cpp b/src/cli/execute.cpp
index 81a0388..0ff913c 100644
--- a/src/cli/execute.cpp
+++ b/src/cli/execute.cpp
@@ -72,6 +72,8 @@ using mesos::v1::TaskID;
using mesos::v1::TaskInfo;
using mesos::v1::TaskState;
using mesos::v1::TaskStatus;
+using mesos::v1::Volume;
+using mesos::v1::Parameters;
using mesos::v1::scheduler::Call;
using mesos::v1::scheduler::Event;
@@ -572,6 +574,12 @@ private:
}
}
+ Volume* volume1 = containerInfo.add_volumes();
+ volume1->set_container_path("/tmp/abcd");
+ volume1->set_mode(Volume::RW);
+ volume1->set_host_path("/root/convoy");
+ cout << "Add Voume 1" << endl;
+
return containerInfo;
} else if (containerizer == "docker") {
// 'docker' containerizer only supports 'docker' images.
3) launch a task with docker image, task failed.
4) Check sandbox:
+ /root/src/mesos/m1/mesos/build/src/mesos-containerizer mount --help=false
--operation=make-rslave --path=/
+ grep -E /tmp/mesos/.+ /proc/self/mountinfo
+ grep -v 3239aafc-78d8-4f70-81e5-f32fb3733339
+ cut+ -d -f5
xargs --no-run-if-empty umount -l
+ mount -n --rbind
/tmp/mesos/provisioner/containers/3239aafc-78d8-4f70-81e5-f32fb3733339/backends/copy/rootfses/5e8bf3fa-53b1-4bd5-bb3d-525ddc7900b6
/tmp/mesos/slaves/a4294ed5-10e8-47db-a3b9-a43a4f951374-S0/frameworks/a4294ed5-10e8-47db-a3b9-a43a4f951374-0000/executors/test/runs/3239aafc-78d8-4f70-81e5-f32fb3733339/.rootfs
+ mount -n --rbind /root/convoy
/tmp/mesos/slaves/a4294ed5-10e8-47db-a3b9-a43a4f951374-S0/frameworks/a4294ed5-10e8-47db-a3b9-a43a4f951374-0000/executors/test/runs/3239aafc-78d8-4f70-81e5-f32fb3733339/.rootfs/tmp/abcd
mount: mount point
/tmp/mesos/slaves/a4294ed5-10e8-47db-a3b9-a43a4f951374-S0/frameworks/a4294ed5-10e8-47db-a3b9-a43a4f951374-0000/executors/test/runs/3239aafc-78d8-4f70-81e5-f32fb3733339/.rootfs/tmp/abcd
does not exist
Failed to execute a preparation shell command
Will check more for this.
Thanks,
Guangya
On Mon, May 23, 2016 at 3:35 PM, Olivier Sallou <ol...@irisa.fr>
wrote:
>
>
> On 05/23/2016 09:33 AM, Olivier Sallou wrote:
> >
> > On 05/20/2016 03:26 PM, Guangya Liu wrote:
> >> Since you are using docker image which means that your container will
> have
> >> rootfs, so it is not required to have the absolute path exist, the linux
> >> file system isolator will help create the path automatically
> >>
> https://github.com/apache/mesos/blob/0.28.x/src/slave/containerizer/mesos/isolators/filesystem/linux.cpp#L390-L402
> >>
> >> Can you please share your framework? How did you set the volume part in
> >> your framework?
> > @Guangya
> >
> > I use Python API.
> >
> > Here is related code:
> >
> > ....
> > # Define container volumes
> > for v in job['container']['volumes']:
> > volume = container.volumes.add()
> > volume.container_path = v['mount']
> > volume.host_path = v['path']
> > if v['acl'] == 'rw':
> > volume.mode = 1 # mesos_pb2.Volume.Mode.RW
> > else:
> > volume.mode = 2 # mesos_pb2.Volume.Mode.RO
> >
> > => In my test case, I add 2 volumes from a host shared directory,
> > mounted in container as /mnt/go-docker and /mnt/god-data.
> >
> > ...
> > # Define docker image and network
> > docker = mesos_pb2.ContainerInfo.MesosInfo()
> > docker.image.type = 2 # Docker
> > docker.image.docker.name ='centos:latest'
> > # Request an IP from a network module
> > network_info = container.network_infos.add()
> > network_info_name = 'sampletest'
> > # Get an IP V4 address
> > ip_address = network_info.ip_addresses.add()
> > ip_address.protocol = 1
> > # The network group to join
> > group = network_info.groups.append(network_info_name)
> > port_list = [22]
> > if port_list:
> > for port in port_list:
> > job['container']['port_mapping'].append({'host':
> > port, 'container': port})
> > container.mesos.MergeFrom(docker)
> >
> > It results in error message:
> >
> > + mount -n --rbind
> >
> /tmp/mesos/provisioner/containers/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/backends/copy/rootfses/f9f66bb2-308d-4555-ba77-49ec61cbeb4f
> >
> /tmp/mesos/slaves/2a296daf-7419-4659-ade1-763c792cd522-S0/frameworks/aef1b0e3-ea2d-4770-baac-96d673ab88f9-0000/executors/51/runs/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/.rootfs
> > + mount -n --rbind
> >
> /home/osallou/Development/NOSAVE/go-docker/godshared/tasks/pairtree_root/us/er/_o/sa/ll/ou/task
> >
> /tmp/mesos/slaves/2a296daf-7419-4659-ade1-763c792cd522-S0/frameworks/aef1b0e3-ea2d-4770-baac-96d673ab88f9-0000/executors/51/runs/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/.rootfs/mnt/god-data
> > mount: mount point
> >
> /tmp/mesos/slaves/2a296daf-7419-4659-ade1-763c792cd522-S0/frameworks/aef1b0e3-ea2d-4770-baac-96d673ab88f9-0000/executors/51/runs/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/.rootfs/mnt/god-data
> > does not exist
> > Failed to execute a preparation shell command
> >
> >
> > We can see the .rootfs, but mnt/god-data under .rootfs fails. Local
> > directory exists, it does not pre-exists in container. What is strange
> > is , if I look in mesos task dir, .rootfs/mnt/go-data, directory is
> present.
> >
> > Or, is the error message (.rootfs/mnt/god-data does not exist) simply a
> > warning, and it creates it, and final error (Failed to execute a
> > preparation shell command) not related (and unclear...)
> Additional info: command to execute in container is located in one of
> mounted volume.
> >
> > Olivier
> >> Thanks,
> >>
> >> Guangya
> >>
> >> On Fri, May 20, 2016 at 4:54 AM, Olivier Sallou <
> olivier.sallou@irisa.fr>
> >> wrote:
> >>
> >>> ----- Mail original -----
> >>>> De: "Gilbert Song" <gi...@mesosphere.io>
> >>>> À: "dev" <de...@mesos.apache.org>
> >>>> Envoyé: Jeudi 19 Mai 2016 01:57:16
> >>>> Objet: Re: volume / mount point error with Unified Containerizer
> >>>>
> >>>> @Olivier,
> >>>> In mesos 0.28.1, you are supposed to be able bind mount a volume from
> >>>> the host into the mesos container. Did you specify a docker image (we
> >>>> determine
> >>>> the mount point differently depending whether the container has a
> >>> rootfs)?
> >>>
> >>> Yes I specified an image, a Docker image URI.
> >>>
> >>>> How
> >>>> do you specify your 'container_path' (the mount point in the
> container)?
> >>> If
> >>>> it is an
> >>>> absolute path, we require that dir to be pre-existed. If it is a
> relative
> >>>> path, we will
> >>>> mkdir for it.
> >>> It is an absolute path, but it does not exists in image (this is the
> >>> issue). Images are custom Docker images (images containing tools for
> batch
> >>> computing), and I want, for example, to mount some shared resources
> (user
> >>> home dir, common data, etc.) in the image. Of course those directories
> do
> >>> not pre-exists in container images as they are specific to the
> environment.
> >>> Requiring existence of the directory in the image is not issue as it
> >>> prevents using any existing image from a repo.
> >>>
> >>> When using Docker containerizer it works fine, I can mount any external
> >>> storage in the container.
> >>>
> >>> Olivie
> >>>
> >>>
> >>>> @Joshua,
> >>>> Thank for posting your workaround on mesos. As I mentioned above, in
> >>> 0.28.1
> >>>> or
> >>>> older, we only mkdir for container_path which is relative path (not
> >>>> starting with "/").
> >>>> Because if no rootfs specified for a mesos container, the container
> >>> shares
> >>>> the host
> >>>> root filesystem. Obviously we don't want any random files to be
> created
> >>>> implicitly
> >>>> on your host fs.
> >>>> From mesos 0.29 (release by the end of this month), we will mkdir the
> >>> mount
> >>>> point in the container except for the command task case that specify
> an
> >>>> absolute
> >>>> container_path without a rootfs. Because we simplify the mounting
> logic,
> >>> and
> >>>> sandbox bind mount will only be done in container mount namespace
> >>> instead of
> >>>> host mount namespace (what we did before). Please keep tuned.
> >>>>
> >>>> Cheers,
> >>>> Gilbert
> >>>>
> >>>> On Wed, May 18, 2016 at 8:14 AM, Joshua Cohen <jc...@apache.org>
> wrote:
> >>>>
> >>>>> Hi Olivier,
> >>>>>
> >>>>> I touched on this issue as part of
> >>>>> https://issues.apache.org/jira/browse/MESOS-5229. It would be nice
> if
> >>>>> Mesos
> >>>>> automatically created container mount points if they don't already
> >>> exist.
> >>>>> In the meantime, as a workaround for this, I've updated my filesystem
> >>>>> images to include the path (e.g. in Dockerfile, add `RUN mkdir -p
> >>>>> /some/mount/point`). Not the best solution, but the only thing I've
> >>> seen
> >>>>> that works at the moment.
> >>>>>
> >>>>> Cheers,
> >>>>>
> >>>>> Joshua
> >>>>>
> >>>>> On Wed, May 18, 2016 at 7:36 AM, Guangya Liu <gy...@gmail.com>
> >>> wrote:
> >>>>>> It's pretty simple for you from scratch with source code
> >>>>>>
> >>>>>>
> >>>
> https://github.com/apache/mesos/blob/master/docs/getting-started.md#building-mesos
> >>>>>> ;-)
> >>>>>>
> >>>>>> Thanks,
> >>>>>>
> >>>>>> Guangya
> >>>>>>
> >>>>>> On Wed, May 18, 2016 at 8:30 PM, Olivier Sallou <
> >>> olivier.sallou@irisa.fr
> >>>>>> wrote:
> >>>>>>
> >>>>>>> On 05/18/2016 02:31 PM, Guangya Liu wrote:
> >>>>>>>> Just saw that you are working with 0.28.1, the "docker volume
> >>> driver"
> >>>>>>> code
> >>>>>>>> was not in 0.28.1, can you please have a try with mesos master
> >>> branch
> >>>>>> if
> >>>>>>>> you are only doing some test?
> >>>>>>> this is indeed test only for the moment. But I will have to
> >>>>>>> recompile/install mesos :-( (I used packages for install).
> >>>>>>>
> >>>>>>> I will try when possible, but thanks for the hint.
> >>>>>>>> Thanks,
> >>>>>>>>
> >>>>>>>> Guangya
> >>>>>>>>
> >>>>>>>> On Wed, May 18, 2016 at 8:28 PM, Guangya Liu <gyliu513@gmail.com
> >>>>>> wrote:
> >>>>>>>>> Hi Olivier,
> >>>>>>>>>
> >>>>>>>>> I think that you need to enable "docker volume isolator" if you
> >>> want
> >>>>>> use
> >>>>>>>>> external storage with unified container I was writing a document
> >>>>> here
> >>>>>>>>> https://reviews.apache.org/r/47511/, perhaps you can have a try
> >>>>>>> according
> >>>>>>>>> to the document and post some comments there if you find any
> >>> issues.
> >>>>>>>>> Also you can patch mesos-execute here
> >>>>>>> https://reviews.apache.org/r/46762/ to
> >>>>>>>>> have a try with mesos-execute.
> >>>>>>>>>
> >>>>>>>>> Thanks,
> >>>>>>>>>
> >>>>>>>>> Guangya
> >>>>>>>>>
> >>>>>>>>> On Wed, May 18, 2016 at 7:17 PM, Olivier Sallou <
> >>>>>>> olivier.sallou@irisa.fr>
> >>>>>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>>> Answering (partially) to myself.
> >>>>>>>>>>
> >>>>>>>>>> I seems issue is container_path does not exists inside
> >>> container.
> >>>>> On
> >>>>>>>>>> Docker, path is created and mounted. With pure mesos,
> >>>>> container_path
> >>>>>>>>>> must exists.
> >>>>>>>>>>
> >>>>>>>>>> mesos.proto says: "If the path is an absolute path, that path
> >>> must
> >>>>>>>>>> already exist."
> >>>>>>>>>>
> >>>>>>>>>> This is an issue however, using Docker images, the path I want
> >>> to
> >>>>>> mount
> >>>>>>>>>> does not exists, and it cannot be modified "on the fly".
> >>>>>>>>>>
> >>>>>>>>>> Is there a workaround for this ?
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> On 05/18/2016 12:24 PM, Olivier Sallou wrote:
> >>>>>>>>>>> Hi,
> >>>>>>>>>>> I am trying unified containerizer on a single server
> >>>>> (master/slave)
> >>>>>> on
> >>>>>>>>>>> mesos 0.28.1, to switch from docker containerizer to
> >>> mesos+docker
> >>>>>>> image
> >>>>>>>>>>> container.
> >>>>>>>>>>>
> >>>>>>>>>>> I have setup slave config as suggested in documentation:
> >>>>>>>>>>>
> >>>>>>>>>>> containerizers=docker,mesos
> >>>>>>>>>>> image_providers=docker \
> >>>>>>>>>>> isolation=filesystem/linux,docker/runtime
> >>>>>>>>>>>
> >>>>>>>>>>> However, when I execute my task with a volume I have an error:
> >>>>>>>>>>>
> >>>>>>>>>>> ....
> >>>>>>>>>>> + mount -n --rbind
> >>>>>>>>>>>
> >>>
> /tmp/mesos/provisioner/containers/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/backends/copy/rootfses/f9f66bb2-308d-4555-ba77-49ec61cbeb4f
> >>>
> /tmp/mesos/slaves/2a296daf-7419-4659-ade1-763c792cd522-S0/frameworks/aef1b0e3-ea2d-4770-baac-96d673ab88f9-0000/executors/51/runs/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/.rootfs
> >>>>>>>>>>> + mount -n --rbind
> >>>>>>>>>>>
> >>>
> /home/osallou/Development/NOSAVE/go-docker/godshared/tasks/pairtree_root/us/er/_o/sa/ll/ou/task
> >>>
> /tmp/mesos/slaves/2a296daf-7419-4659-ade1-763c792cd522-S0/frameworks/aef1b0e3-ea2d-4770-baac-96d673ab88f9-0000/executors/51/runs/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/.rootfs/mnt/god-data
> >>>>>>>>>>> mount: mount point
> >>>>>>>>>>>
> >>>
> /tmp/mesos/slaves/2a296daf-7419-4659-ade1-763c792cd522-S0/frameworks/aef1b0e3-ea2d-4770-baac-96d673ab88f9-0000/executors/51/runs/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/.rootfs/mnt/god-data
> >>>>>>>>>>> does not exist
> >>>>>>>>>>> Failed to execute a preparation shell command
> >>>>>>>>>>>
> >>>>>>>>>>> Then, my task switches to FAILED.
> >>>>>>>>>>>
> >>>>>>>>>>> I define a local volume to bind mount in my "container"
> >>>>>>>>>>>
> >>>
> /home/osallou/Development/NOSAVE/go-docker/godshared/tasks/pairtree_root/us/er/_o/sa/ll/ou/task
> >>>>>>>>>>> => /mnt/god-data
> >>>>>>>>>>> My directory exists on local server.
> >>>>>>>>>>> In mesos UI, I can see the .rootfs directory along stdout and
> >>>>> stderr
> >>>>>>>>>>> files, and inside .rootfs, I can see /mnt/god-data (empty).
> >>>>>>>>>>>
> >>>>>>>>>>> Running the same using Docker containerizer instead of mesos
> >>>>>>>>>>> containerizer (with a Docker image) works fine.
> >>>>>>>>>>>
> >>>>>>>>>>> It seems it fails to mount my local directory in the
> >>> container.
> >>>>> Any
> >>>>>>> idea
> >>>>>>>>>>> of what is going wrong or how to debug this?
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> Thanks
> >>>>>>>>>>>
> >>>>>>>>>> --
> >>>>>>>>>> Olivier Sallou
> >>>>>>>>>> IRISA / University of Rennes 1
> >>>>>>>>>> Campus de Beaulieu, 35000 RENNES - FRANCE
> >>>>>>>>>> Tel: 02.99.84.71.95
> >>>>>>>>>>
> >>>>>>>>>> gpg key id: 4096R/326D8438 (keyring.debian.org)
> >>>>>>>>>> Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D
> >>>>> 8438
> >>>>>>> --
> >>>>>>> Olivier Sallou
> >>>>>>> IRISA / University of Rennes 1
> >>>>>>> Campus de Beaulieu, 35000 RENNES - FRANCE
> >>>>>>> Tel: 02.99.84.71.95
> >>>>>>>
> >>>>>>> gpg key id: 4096R/326D8438 (keyring.debian.org)
> >>>>>>> Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D
> >>> 8438
>
> --
> Olivier Sallou
> IRISA / University of Rennes 1
> Campus de Beaulieu, 35000 RENNES - FRANCE
> Tel: 02.99.84.71.95
>
> gpg key id: 4096R/326D8438 (keyring.debian.org)
> Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D 8438
>
>
Re: volume / mount point error with Unified Containerizer
Posted by Olivier Sallou <ol...@irisa.fr>.
On 05/23/2016 09:33 AM, Olivier Sallou wrote:
>
> On 05/20/2016 03:26 PM, Guangya Liu wrote:
>> Since you are using docker image which means that your container will have
>> rootfs, so it is not required to have the absolute path exist, the linux
>> file system isolator will help create the path automatically
>> https://github.com/apache/mesos/blob/0.28.x/src/slave/containerizer/mesos/isolators/filesystem/linux.cpp#L390-L402
>>
>> Can you please share your framework? How did you set the volume part in
>> your framework?
> @Guangya
>
> I use Python API.
>
> Here is related code:
>
> ....
> # Define container volumes
> for v in job['container']['volumes']:
> volume = container.volumes.add()
> volume.container_path = v['mount']
> volume.host_path = v['path']
> if v['acl'] == 'rw':
> volume.mode = 1 # mesos_pb2.Volume.Mode.RW
> else:
> volume.mode = 2 # mesos_pb2.Volume.Mode.RO
>
> => In my test case, I add 2 volumes from a host shared directory,
> mounted in container as /mnt/go-docker and /mnt/god-data.
>
> ...
> # Define docker image and network
> docker = mesos_pb2.ContainerInfo.MesosInfo()
> docker.image.type = 2 # Docker
> docker.image.docker.name ='centos:latest'
> # Request an IP from a network module
> network_info = container.network_infos.add()
> network_info_name = 'sampletest'
> # Get an IP V4 address
> ip_address = network_info.ip_addresses.add()
> ip_address.protocol = 1
> # The network group to join
> group = network_info.groups.append(network_info_name)
> port_list = [22]
> if port_list:
> for port in port_list:
> job['container']['port_mapping'].append({'host':
> port, 'container': port})
> container.mesos.MergeFrom(docker)
>
> It results in error message:
>
> + mount -n --rbind
> /tmp/mesos/provisioner/containers/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/backends/copy/rootfses/f9f66bb2-308d-4555-ba77-49ec61cbeb4f
> /tmp/mesos/slaves/2a296daf-7419-4659-ade1-763c792cd522-S0/frameworks/aef1b0e3-ea2d-4770-baac-96d673ab88f9-0000/executors/51/runs/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/.rootfs
> + mount -n --rbind
> /home/osallou/Development/NOSAVE/go-docker/godshared/tasks/pairtree_root/us/er/_o/sa/ll/ou/task
> /tmp/mesos/slaves/2a296daf-7419-4659-ade1-763c792cd522-S0/frameworks/aef1b0e3-ea2d-4770-baac-96d673ab88f9-0000/executors/51/runs/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/.rootfs/mnt/god-data
> mount: mount point
> /tmp/mesos/slaves/2a296daf-7419-4659-ade1-763c792cd522-S0/frameworks/aef1b0e3-ea2d-4770-baac-96d673ab88f9-0000/executors/51/runs/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/.rootfs/mnt/god-data
> does not exist
> Failed to execute a preparation shell command
>
>
> We can see the .rootfs, but mnt/god-data under .rootfs fails. Local
> directory exists, it does not pre-exists in container. What is strange
> is , if I look in mesos task dir, .rootfs/mnt/go-data, directory is present.
>
> Or, is the error message (.rootfs/mnt/god-data does not exist) simply a
> warning, and it creates it, and final error (Failed to execute a
> preparation shell command) not related (and unclear...)
Additional info: command to execute in container is located in one of
mounted volume.
>
> Olivier
>> Thanks,
>>
>> Guangya
>>
>> On Fri, May 20, 2016 at 4:54 AM, Olivier Sallou <ol...@irisa.fr>
>> wrote:
>>
>>> ----- Mail original -----
>>>> De: "Gilbert Song" <gi...@mesosphere.io>
>>>> �: "dev" <de...@mesos.apache.org>
>>>> Envoy�: Jeudi 19 Mai 2016 01:57:16
>>>> Objet: Re: volume / mount point error with Unified Containerizer
>>>>
>>>> @Olivier,
>>>> In mesos 0.28.1, you are supposed to be able bind mount a volume from
>>>> the host into the mesos container. Did you specify a docker image (we
>>>> determine
>>>> the mount point differently depending whether the container has a
>>> rootfs)?
>>>
>>> Yes I specified an image, a Docker image URI.
>>>
>>>> How
>>>> do you specify your 'container_path' (the mount point in the container)?
>>> If
>>>> it is an
>>>> absolute path, we require that dir to be pre-existed. If it is a relative
>>>> path, we will
>>>> mkdir for it.
>>> It is an absolute path, but it does not exists in image (this is the
>>> issue). Images are custom Docker images (images containing tools for batch
>>> computing), and I want, for example, to mount some shared resources (user
>>> home dir, common data, etc.) in the image. Of course those directories do
>>> not pre-exists in container images as they are specific to the environment.
>>> Requiring existence of the directory in the image is not issue as it
>>> prevents using any existing image from a repo.
>>>
>>> When using Docker containerizer it works fine, I can mount any external
>>> storage in the container.
>>>
>>> Olivie
>>>
>>>
>>>> @Joshua,
>>>> Thank for posting your workaround on mesos. As I mentioned above, in
>>> 0.28.1
>>>> or
>>>> older, we only mkdir for container_path which is relative path (not
>>>> starting with "/").
>>>> Because if no rootfs specified for a mesos container, the container
>>> shares
>>>> the host
>>>> root filesystem. Obviously we don't want any random files to be created
>>>> implicitly
>>>> on your host fs.
>>>> From mesos 0.29 (release by the end of this month), we will mkdir the
>>> mount
>>>> point in the container except for the command task case that specify an
>>>> absolute
>>>> container_path without a rootfs. Because we simplify the mounting logic,
>>> and
>>>> sandbox bind mount will only be done in container mount namespace
>>> instead of
>>>> host mount namespace (what we did before). Please keep tuned.
>>>>
>>>> Cheers,
>>>> Gilbert
>>>>
>>>> On Wed, May 18, 2016 at 8:14 AM, Joshua Cohen <jc...@apache.org> wrote:
>>>>
>>>>> Hi Olivier,
>>>>>
>>>>> I touched on this issue as part of
>>>>> https://issues.apache.org/jira/browse/MESOS-5229. It would be nice if
>>>>> Mesos
>>>>> automatically created container mount points if they don't already
>>> exist.
>>>>> In the meantime, as a workaround for this, I've updated my filesystem
>>>>> images to include the path (e.g. in Dockerfile, add `RUN mkdir -p
>>>>> /some/mount/point`). Not the best solution, but the only thing I've
>>> seen
>>>>> that works at the moment.
>>>>>
>>>>> Cheers,
>>>>>
>>>>> Joshua
>>>>>
>>>>> On Wed, May 18, 2016 at 7:36 AM, Guangya Liu <gy...@gmail.com>
>>> wrote:
>>>>>> It's pretty simple for you from scratch with source code
>>>>>>
>>>>>>
>>> https://github.com/apache/mesos/blob/master/docs/getting-started.md#building-mesos
>>>>>> ;-)
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> Guangya
>>>>>>
>>>>>> On Wed, May 18, 2016 at 8:30 PM, Olivier Sallou <
>>> olivier.sallou@irisa.fr
>>>>>> wrote:
>>>>>>
>>>>>>> On 05/18/2016 02:31 PM, Guangya Liu wrote:
>>>>>>>> Just saw that you are working with 0.28.1, the "docker volume
>>> driver"
>>>>>>> code
>>>>>>>> was not in 0.28.1, can you please have a try with mesos master
>>> branch
>>>>>> if
>>>>>>>> you are only doing some test?
>>>>>>> this is indeed test only for the moment. But I will have to
>>>>>>> recompile/install mesos :-( (I used packages for install).
>>>>>>>
>>>>>>> I will try when possible, but thanks for the hint.
>>>>>>>> Thanks,
>>>>>>>>
>>>>>>>> Guangya
>>>>>>>>
>>>>>>>> On Wed, May 18, 2016 at 8:28 PM, Guangya Liu <gyliu513@gmail.com
>>>>>> wrote:
>>>>>>>>> Hi Olivier,
>>>>>>>>>
>>>>>>>>> I think that you need to enable "docker volume isolator" if you
>>> want
>>>>>> use
>>>>>>>>> external storage with unified container I was writing a document
>>>>> here
>>>>>>>>> https://reviews.apache.org/r/47511/, perhaps you can have a try
>>>>>>> according
>>>>>>>>> to the document and post some comments there if you find any
>>> issues.
>>>>>>>>> Also you can patch mesos-execute here
>>>>>>> https://reviews.apache.org/r/46762/ to
>>>>>>>>> have a try with mesos-execute.
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>>
>>>>>>>>> Guangya
>>>>>>>>>
>>>>>>>>> On Wed, May 18, 2016 at 7:17 PM, Olivier Sallou <
>>>>>>> olivier.sallou@irisa.fr>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Answering (partially) to myself.
>>>>>>>>>>
>>>>>>>>>> I seems issue is container_path does not exists inside
>>> container.
>>>>> On
>>>>>>>>>> Docker, path is created and mounted. With pure mesos,
>>>>> container_path
>>>>>>>>>> must exists.
>>>>>>>>>>
>>>>>>>>>> mesos.proto says: "If the path is an absolute path, that path
>>> must
>>>>>>>>>> already exist."
>>>>>>>>>>
>>>>>>>>>> This is an issue however, using Docker images, the path I want
>>> to
>>>>>> mount
>>>>>>>>>> does not exists, and it cannot be modified "on the fly".
>>>>>>>>>>
>>>>>>>>>> Is there a workaround for this ?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On 05/18/2016 12:24 PM, Olivier Sallou wrote:
>>>>>>>>>>> Hi,
>>>>>>>>>>> I am trying unified containerizer on a single server
>>>>> (master/slave)
>>>>>> on
>>>>>>>>>>> mesos 0.28.1, to switch from docker containerizer to
>>> mesos+docker
>>>>>>> image
>>>>>>>>>>> container.
>>>>>>>>>>>
>>>>>>>>>>> I have setup slave config as suggested in documentation:
>>>>>>>>>>>
>>>>>>>>>>> containerizers=docker,mesos
>>>>>>>>>>> image_providers=docker \
>>>>>>>>>>> isolation=filesystem/linux,docker/runtime
>>>>>>>>>>>
>>>>>>>>>>> However, when I execute my task with a volume I have an error:
>>>>>>>>>>>
>>>>>>>>>>> ....
>>>>>>>>>>> + mount -n --rbind
>>>>>>>>>>>
>>> /tmp/mesos/provisioner/containers/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/backends/copy/rootfses/f9f66bb2-308d-4555-ba77-49ec61cbeb4f
>>> /tmp/mesos/slaves/2a296daf-7419-4659-ade1-763c792cd522-S0/frameworks/aef1b0e3-ea2d-4770-baac-96d673ab88f9-0000/executors/51/runs/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/.rootfs
>>>>>>>>>>> + mount -n --rbind
>>>>>>>>>>>
>>> /home/osallou/Development/NOSAVE/go-docker/godshared/tasks/pairtree_root/us/er/_o/sa/ll/ou/task
>>> /tmp/mesos/slaves/2a296daf-7419-4659-ade1-763c792cd522-S0/frameworks/aef1b0e3-ea2d-4770-baac-96d673ab88f9-0000/executors/51/runs/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/.rootfs/mnt/god-data
>>>>>>>>>>> mount: mount point
>>>>>>>>>>>
>>> /tmp/mesos/slaves/2a296daf-7419-4659-ade1-763c792cd522-S0/frameworks/aef1b0e3-ea2d-4770-baac-96d673ab88f9-0000/executors/51/runs/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/.rootfs/mnt/god-data
>>>>>>>>>>> does not exist
>>>>>>>>>>> Failed to execute a preparation shell command
>>>>>>>>>>>
>>>>>>>>>>> Then, my task switches to FAILED.
>>>>>>>>>>>
>>>>>>>>>>> I define a local volume to bind mount in my "container"
>>>>>>>>>>>
>>> /home/osallou/Development/NOSAVE/go-docker/godshared/tasks/pairtree_root/us/er/_o/sa/ll/ou/task
>>>>>>>>>>> => /mnt/god-data
>>>>>>>>>>> My directory exists on local server.
>>>>>>>>>>> In mesos UI, I can see the .rootfs directory along stdout and
>>>>> stderr
>>>>>>>>>>> files, and inside .rootfs, I can see /mnt/god-data (empty).
>>>>>>>>>>>
>>>>>>>>>>> Running the same using Docker containerizer instead of mesos
>>>>>>>>>>> containerizer (with a Docker image) works fine.
>>>>>>>>>>>
>>>>>>>>>>> It seems it fails to mount my local directory in the
>>> container.
>>>>> Any
>>>>>>> idea
>>>>>>>>>>> of what is going wrong or how to debug this?
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Thanks
>>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Olivier Sallou
>>>>>>>>>> IRISA / University of Rennes 1
>>>>>>>>>> Campus de Beaulieu, 35000 RENNES - FRANCE
>>>>>>>>>> Tel: 02.99.84.71.95
>>>>>>>>>>
>>>>>>>>>> gpg key id: 4096R/326D8438 (keyring.debian.org)
>>>>>>>>>> Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D
>>>>> 8438
>>>>>>> --
>>>>>>> Olivier Sallou
>>>>>>> IRISA / University of Rennes 1
>>>>>>> Campus de Beaulieu, 35000 RENNES - FRANCE
>>>>>>> Tel: 02.99.84.71.95
>>>>>>>
>>>>>>> gpg key id: 4096R/326D8438 (keyring.debian.org)
>>>>>>> Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D
>>> 8438
--
Olivier Sallou
IRISA / University of Rennes 1
Campus de Beaulieu, 35000 RENNES - FRANCE
Tel: 02.99.84.71.95
gpg key id: 4096R/326D8438 (keyring.debian.org)
Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D 8438
Re: volume / mount point error with Unified Containerizer
Posted by Olivier Sallou <ol...@irisa.fr>.
On 05/20/2016 03:26 PM, Guangya Liu wrote:
> Since you are using docker image which means that your container will have
> rootfs, so it is not required to have the absolute path exist, the linux
> file system isolator will help create the path automatically
> https://github.com/apache/mesos/blob/0.28.x/src/slave/containerizer/mesos/isolators/filesystem/linux.cpp#L390-L402
>
> Can you please share your framework? How did you set the volume part in
> your framework?
@Guangya
I use Python API.
Here is related code:
....
# Define container volumes
for v in job['container']['volumes']:
volume = container.volumes.add()
volume.container_path = v['mount']
volume.host_path = v['path']
if v['acl'] == 'rw':
volume.mode = 1 # mesos_pb2.Volume.Mode.RW
else:
volume.mode = 2 # mesos_pb2.Volume.Mode.RO
=> In my test case, I add 2 volumes from a host shared directory,
mounted in container as /mnt/go-docker and /mnt/god-data.
...
# Define docker image and network
docker = mesos_pb2.ContainerInfo.MesosInfo()
docker.image.type = 2 # Docker
docker.image.docker.name ='centos:latest'
# Request an IP from a network module
network_info = container.network_infos.add()
network_info_name = 'sampletest'
# Get an IP V4 address
ip_address = network_info.ip_addresses.add()
ip_address.protocol = 1
# The network group to join
group = network_info.groups.append(network_info_name)
port_list = [22]
if port_list:
for port in port_list:
job['container']['port_mapping'].append({'host':
port, 'container': port})
container.mesos.MergeFrom(docker)
It results in error message:
+ mount -n --rbind
/tmp/mesos/provisioner/containers/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/backends/copy/rootfses/f9f66bb2-308d-4555-ba77-49ec61cbeb4f
/tmp/mesos/slaves/2a296daf-7419-4659-ade1-763c792cd522-S0/frameworks/aef1b0e3-ea2d-4770-baac-96d673ab88f9-0000/executors/51/runs/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/.rootfs
+ mount -n --rbind
/home/osallou/Development/NOSAVE/go-docker/godshared/tasks/pairtree_root/us/er/_o/sa/ll/ou/task
/tmp/mesos/slaves/2a296daf-7419-4659-ade1-763c792cd522-S0/frameworks/aef1b0e3-ea2d-4770-baac-96d673ab88f9-0000/executors/51/runs/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/.rootfs/mnt/god-data
mount: mount point
/tmp/mesos/slaves/2a296daf-7419-4659-ade1-763c792cd522-S0/frameworks/aef1b0e3-ea2d-4770-baac-96d673ab88f9-0000/executors/51/runs/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/.rootfs/mnt/god-data
does not exist
Failed to execute a preparation shell command
We can see the .rootfs, but mnt/god-data under .rootfs fails. Local
directory exists, it does not pre-exists in container. What is strange
is , if I look in mesos task dir, .rootfs/mnt/go-data, directory is present.
Or, is the error message (.rootfs/mnt/god-data does not exist) simply a
warning, and it creates it, and final error (Failed to execute a
preparation shell command) not related (and unclear...)
Olivier
>
> Thanks,
>
> Guangya
>
> On Fri, May 20, 2016 at 4:54 AM, Olivier Sallou <ol...@irisa.fr>
> wrote:
>
>>
>> ----- Mail original -----
>>> De: "Gilbert Song" <gi...@mesosphere.io>
>>> À: "dev" <de...@mesos.apache.org>
>>> Envoyé: Jeudi 19 Mai 2016 01:57:16
>>> Objet: Re: volume / mount point error with Unified Containerizer
>>>
>>> @Olivier,
>>> In mesos 0.28.1, you are supposed to be able bind mount a volume from
>>> the host into the mesos container. Did you specify a docker image (we
>>> determine
>>> the mount point differently depending whether the container has a
>> rootfs)?
>>
>> Yes I specified an image, a Docker image URI.
>>
>>> How
>>> do you specify your 'container_path' (the mount point in the container)?
>> If
>>> it is an
>>> absolute path, we require that dir to be pre-existed. If it is a relative
>>> path, we will
>>> mkdir for it.
>> It is an absolute path, but it does not exists in image (this is the
>> issue). Images are custom Docker images (images containing tools for batch
>> computing), and I want, for example, to mount some shared resources (user
>> home dir, common data, etc.) in the image. Of course those directories do
>> not pre-exists in container images as they are specific to the environment.
>> Requiring existence of the directory in the image is not issue as it
>> prevents using any existing image from a repo.
>>
>> When using Docker containerizer it works fine, I can mount any external
>> storage in the container.
>>
>> Olivie
>>
>>
>>> @Joshua,
>>> Thank for posting your workaround on mesos. As I mentioned above, in
>> 0.28.1
>>> or
>>> older, we only mkdir for container_path which is relative path (not
>>> starting with "/").
>>> Because if no rootfs specified for a mesos container, the container
>> shares
>>> the host
>>> root filesystem. Obviously we don't want any random files to be created
>>> implicitly
>>> on your host fs.
>>> From mesos 0.29 (release by the end of this month), we will mkdir the
>> mount
>>> point in the container except for the command task case that specify an
>>> absolute
>>> container_path without a rootfs. Because we simplify the mounting logic,
>> and
>>> sandbox bind mount will only be done in container mount namespace
>> instead of
>>> host mount namespace (what we did before). Please keep tuned.
>>>
>>> Cheers,
>>> Gilbert
>>>
>>> On Wed, May 18, 2016 at 8:14 AM, Joshua Cohen <jc...@apache.org> wrote:
>>>
>>>> Hi Olivier,
>>>>
>>>> I touched on this issue as part of
>>>> https://issues.apache.org/jira/browse/MESOS-5229. It would be nice if
>>>> Mesos
>>>> automatically created container mount points if they don't already
>> exist.
>>>> In the meantime, as a workaround for this, I've updated my filesystem
>>>> images to include the path (e.g. in Dockerfile, add `RUN mkdir -p
>>>> /some/mount/point`). Not the best solution, but the only thing I've
>> seen
>>>> that works at the moment.
>>>>
>>>> Cheers,
>>>>
>>>> Joshua
>>>>
>>>> On Wed, May 18, 2016 at 7:36 AM, Guangya Liu <gy...@gmail.com>
>> wrote:
>>>>> It's pretty simple for you from scratch with source code
>>>>>
>>>>>
>> https://github.com/apache/mesos/blob/master/docs/getting-started.md#building-mesos
>>>>> ;-)
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Guangya
>>>>>
>>>>> On Wed, May 18, 2016 at 8:30 PM, Olivier Sallou <
>> olivier.sallou@irisa.fr
>>>>> wrote:
>>>>>
>>>>>>
>>>>>> On 05/18/2016 02:31 PM, Guangya Liu wrote:
>>>>>>> Just saw that you are working with 0.28.1, the "docker volume
>> driver"
>>>>>> code
>>>>>>> was not in 0.28.1, can you please have a try with mesos master
>> branch
>>>>> if
>>>>>>> you are only doing some test?
>>>>>> this is indeed test only for the moment. But I will have to
>>>>>> recompile/install mesos :-( (I used packages for install).
>>>>>>
>>>>>> I will try when possible, but thanks for the hint.
>>>>>>> Thanks,
>>>>>>>
>>>>>>> Guangya
>>>>>>>
>>>>>>> On Wed, May 18, 2016 at 8:28 PM, Guangya Liu <gyliu513@gmail.com
>>>>> wrote:
>>>>>>>> Hi Olivier,
>>>>>>>>
>>>>>>>> I think that you need to enable "docker volume isolator" if you
>> want
>>>>> use
>>>>>>>> external storage with unified container I was writing a document
>>>> here
>>>>>>>> https://reviews.apache.org/r/47511/, perhaps you can have a try
>>>>>> according
>>>>>>>> to the document and post some comments there if you find any
>> issues.
>>>>>>>> Also you can patch mesos-execute here
>>>>>> https://reviews.apache.org/r/46762/ to
>>>>>>>> have a try with mesos-execute.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>>
>>>>>>>> Guangya
>>>>>>>>
>>>>>>>> On Wed, May 18, 2016 at 7:17 PM, Olivier Sallou <
>>>>>> olivier.sallou@irisa.fr>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Answering (partially) to myself.
>>>>>>>>>
>>>>>>>>> I seems issue is container_path does not exists inside
>> container.
>>>> On
>>>>>>>>> Docker, path is created and mounted. With pure mesos,
>>>> container_path
>>>>>>>>> must exists.
>>>>>>>>>
>>>>>>>>> mesos.proto says: "If the path is an absolute path, that path
>> must
>>>>>>>>> already exist."
>>>>>>>>>
>>>>>>>>> This is an issue however, using Docker images, the path I want
>> to
>>>>> mount
>>>>>>>>> does not exists, and it cannot be modified "on the fly".
>>>>>>>>>
>>>>>>>>> Is there a workaround for this ?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 05/18/2016 12:24 PM, Olivier Sallou wrote:
>>>>>>>>>> Hi,
>>>>>>>>>> I am trying unified containerizer on a single server
>>>> (master/slave)
>>>>> on
>>>>>>>>>> mesos 0.28.1, to switch from docker containerizer to
>> mesos+docker
>>>>>> image
>>>>>>>>>> container.
>>>>>>>>>>
>>>>>>>>>> I have setup slave config as suggested in documentation:
>>>>>>>>>>
>>>>>>>>>> containerizers=docker,mesos
>>>>>>>>>> image_providers=docker \
>>>>>>>>>> isolation=filesystem/linux,docker/runtime
>>>>>>>>>>
>>>>>>>>>> However, when I execute my task with a volume I have an error:
>>>>>>>>>>
>>>>>>>>>> ....
>>>>>>>>>> + mount -n --rbind
>>>>>>>>>>
>> /tmp/mesos/provisioner/containers/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/backends/copy/rootfses/f9f66bb2-308d-4555-ba77-49ec61cbeb4f
>> /tmp/mesos/slaves/2a296daf-7419-4659-ade1-763c792cd522-S0/frameworks/aef1b0e3-ea2d-4770-baac-96d673ab88f9-0000/executors/51/runs/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/.rootfs
>>>>>>>>>> + mount -n --rbind
>>>>>>>>>>
>> /home/osallou/Development/NOSAVE/go-docker/godshared/tasks/pairtree_root/us/er/_o/sa/ll/ou/task
>> /tmp/mesos/slaves/2a296daf-7419-4659-ade1-763c792cd522-S0/frameworks/aef1b0e3-ea2d-4770-baac-96d673ab88f9-0000/executors/51/runs/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/.rootfs/mnt/god-data
>>>>>>>>>> mount: mount point
>>>>>>>>>>
>> /tmp/mesos/slaves/2a296daf-7419-4659-ade1-763c792cd522-S0/frameworks/aef1b0e3-ea2d-4770-baac-96d673ab88f9-0000/executors/51/runs/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/.rootfs/mnt/god-data
>>>>>>>>>> does not exist
>>>>>>>>>> Failed to execute a preparation shell command
>>>>>>>>>>
>>>>>>>>>> Then, my task switches to FAILED.
>>>>>>>>>>
>>>>>>>>>> I define a local volume to bind mount in my "container"
>>>>>>>>>>
>> /home/osallou/Development/NOSAVE/go-docker/godshared/tasks/pairtree_root/us/er/_o/sa/ll/ou/task
>>>>>>>>>> => /mnt/god-data
>>>>>>>>>> My directory exists on local server.
>>>>>>>>>> In mesos UI, I can see the .rootfs directory along stdout and
>>>> stderr
>>>>>>>>>> files, and inside .rootfs, I can see /mnt/god-data (empty).
>>>>>>>>>>
>>>>>>>>>> Running the same using Docker containerizer instead of mesos
>>>>>>>>>> containerizer (with a Docker image) works fine.
>>>>>>>>>>
>>>>>>>>>> It seems it fails to mount my local directory in the
>> container.
>>>> Any
>>>>>> idea
>>>>>>>>>> of what is going wrong or how to debug this?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Thanks
>>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Olivier Sallou
>>>>>>>>> IRISA / University of Rennes 1
>>>>>>>>> Campus de Beaulieu, 35000 RENNES - FRANCE
>>>>>>>>> Tel: 02.99.84.71.95
>>>>>>>>>
>>>>>>>>> gpg key id: 4096R/326D8438 (keyring.debian.org)
>>>>>>>>> Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D
>>>> 8438
>>>>>>>>>
>>>>>> --
>>>>>> Olivier Sallou
>>>>>> IRISA / University of Rennes 1
>>>>>> Campus de Beaulieu, 35000 RENNES - FRANCE
>>>>>> Tel: 02.99.84.71.95
>>>>>>
>>>>>> gpg key id: 4096R/326D8438 (keyring.debian.org)
>>>>>> Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D
>> 8438
>>>>>>
--
Olivier Sallou
IRISA / University of Rennes 1
Campus de Beaulieu, 35000 RENNES - FRANCE
Tel: 02.99.84.71.95
gpg key id: 4096R/326D8438 (keyring.debian.org)
Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D 8438
Re: volume / mount point error with Unified Containerizer
Posted by Guangya Liu <gy...@gmail.com>.
Since you are using docker image which means that your container will have
rootfs, so it is not required to have the absolute path exist, the linux
file system isolator will help create the path automatically
https://github.com/apache/mesos/blob/0.28.x/src/slave/containerizer/mesos/isolators/filesystem/linux.cpp#L390-L402
Can you please share your framework? How did you set the volume part in
your framework?
Thanks,
Guangya
On Fri, May 20, 2016 at 4:54 AM, Olivier Sallou <ol...@irisa.fr>
wrote:
>
>
> ----- Mail original -----
> > De: "Gilbert Song" <gi...@mesosphere.io>
> > À: "dev" <de...@mesos.apache.org>
> > Envoyé: Jeudi 19 Mai 2016 01:57:16
> > Objet: Re: volume / mount point error with Unified Containerizer
> >
> > @Olivier,
> > In mesos 0.28.1, you are supposed to be able bind mount a volume from
> > the host into the mesos container. Did you specify a docker image (we
> > determine
> > the mount point differently depending whether the container has a
> rootfs)?
>
> Yes I specified an image, a Docker image URI.
>
> > How
> > do you specify your 'container_path' (the mount point in the container)?
> If
> > it is an
> > absolute path, we require that dir to be pre-existed. If it is a relative
> > path, we will
> > mkdir for it.
>
> It is an absolute path, but it does not exists in image (this is the
> issue). Images are custom Docker images (images containing tools for batch
> computing), and I want, for example, to mount some shared resources (user
> home dir, common data, etc.) in the image. Of course those directories do
> not pre-exists in container images as they are specific to the environment.
> Requiring existence of the directory in the image is not issue as it
> prevents using any existing image from a repo.
>
> When using Docker containerizer it works fine, I can mount any external
> storage in the container.
>
> Olivie
>
>
> >
> > @Joshua,
> > Thank for posting your workaround on mesos. As I mentioned above, in
> 0.28.1
> > or
> > older, we only mkdir for container_path which is relative path (not
> > starting with "/").
> > Because if no rootfs specified for a mesos container, the container
> shares
> > the host
> > root filesystem. Obviously we don't want any random files to be created
> > implicitly
> > on your host fs.
> > From mesos 0.29 (release by the end of this month), we will mkdir the
> mount
> > point in the container except for the command task case that specify an
> > absolute
> > container_path without a rootfs. Because we simplify the mounting logic,
> and
> > sandbox bind mount will only be done in container mount namespace
> instead of
> > host mount namespace (what we did before). Please keep tuned.
> >
> > Cheers,
> > Gilbert
> >
> > On Wed, May 18, 2016 at 8:14 AM, Joshua Cohen <jc...@apache.org> wrote:
> >
> > > Hi Olivier,
> > >
> > > I touched on this issue as part of
> > > https://issues.apache.org/jira/browse/MESOS-5229. It would be nice if
> > > Mesos
> > > automatically created container mount points if they don't already
> exist.
> > > In the meantime, as a workaround for this, I've updated my filesystem
> > > images to include the path (e.g. in Dockerfile, add `RUN mkdir -p
> > > /some/mount/point`). Not the best solution, but the only thing I've
> seen
> > > that works at the moment.
> > >
> > > Cheers,
> > >
> > > Joshua
> > >
> > > On Wed, May 18, 2016 at 7:36 AM, Guangya Liu <gy...@gmail.com>
> wrote:
> > >
> > > > It's pretty simple for you from scratch with source code
> > > >
> > > >
> > >
> https://github.com/apache/mesos/blob/master/docs/getting-started.md#building-mesos
> > > > ;-)
> > > >
> > > > Thanks,
> > > >
> > > > Guangya
> > > >
> > > > On Wed, May 18, 2016 at 8:30 PM, Olivier Sallou <
> olivier.sallou@irisa.fr
> > > >
> > > > wrote:
> > > >
> > > > >
> > > > >
> > > > > On 05/18/2016 02:31 PM, Guangya Liu wrote:
> > > > > > Just saw that you are working with 0.28.1, the "docker volume
> driver"
> > > > > code
> > > > > > was not in 0.28.1, can you please have a try with mesos master
> branch
> > > > if
> > > > > > you are only doing some test?
> > > > > this is indeed test only for the moment. But I will have to
> > > > > recompile/install mesos :-( (I used packages for install).
> > > > >
> > > > > I will try when possible, but thanks for the hint.
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Guangya
> > > > > >
> > > > > > On Wed, May 18, 2016 at 8:28 PM, Guangya Liu <gyliu513@gmail.com
> >
> > > > wrote:
> > > > > >
> > > > > >> Hi Olivier,
> > > > > >>
> > > > > >> I think that you need to enable "docker volume isolator" if you
> want
> > > > use
> > > > > >> external storage with unified container I was writing a document
> > > here
> > > > > >> https://reviews.apache.org/r/47511/, perhaps you can have a try
> > > > > according
> > > > > >> to the document and post some comments there if you find any
> issues.
> > > > > >>
> > > > > >> Also you can patch mesos-execute here
> > > > > https://reviews.apache.org/r/46762/ to
> > > > > >> have a try with mesos-execute.
> > > > > >>
> > > > > >> Thanks,
> > > > > >>
> > > > > >> Guangya
> > > > > >>
> > > > > >> On Wed, May 18, 2016 at 7:17 PM, Olivier Sallou <
> > > > > olivier.sallou@irisa.fr>
> > > > > >> wrote:
> > > > > >>
> > > > > >>> Answering (partially) to myself.
> > > > > >>>
> > > > > >>> I seems issue is container_path does not exists inside
> container.
> > > On
> > > > > >>> Docker, path is created and mounted. With pure mesos,
> > > container_path
> > > > > >>> must exists.
> > > > > >>>
> > > > > >>> mesos.proto says: "If the path is an absolute path, that path
> must
> > > > > >>> already exist."
> > > > > >>>
> > > > > >>> This is an issue however, using Docker images, the path I want
> to
> > > > mount
> > > > > >>> does not exists, and it cannot be modified "on the fly".
> > > > > >>>
> > > > > >>> Is there a workaround for this ?
> > > > > >>>
> > > > > >>>
> > > > > >>> On 05/18/2016 12:24 PM, Olivier Sallou wrote:
> > > > > >>>> Hi,
> > > > > >>>> I am trying unified containerizer on a single server
> > > (master/slave)
> > > > on
> > > > > >>>> mesos 0.28.1, to switch from docker containerizer to
> mesos+docker
> > > > > image
> > > > > >>>> container.
> > > > > >>>>
> > > > > >>>> I have setup slave config as suggested in documentation:
> > > > > >>>>
> > > > > >>>> containerizers=docker,mesos
> > > > > >>>> image_providers=docker \
> > > > > >>>> isolation=filesystem/linux,docker/runtime
> > > > > >>>>
> > > > > >>>> However, when I execute my task with a volume I have an error:
> > > > > >>>>
> > > > > >>>> ....
> > > > > >>>> + mount -n --rbind
> > > > > >>>>
> > > > > >>>
> > > > >
> > > >
> > >
> /tmp/mesos/provisioner/containers/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/backends/copy/rootfses/f9f66bb2-308d-4555-ba77-49ec61cbeb4f
> > > > > >>>
> > > > >
> > > >
> > >
> /tmp/mesos/slaves/2a296daf-7419-4659-ade1-763c792cd522-S0/frameworks/aef1b0e3-ea2d-4770-baac-96d673ab88f9-0000/executors/51/runs/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/.rootfs
> > > > > >>>> + mount -n --rbind
> > > > > >>>>
> > > > > >>>
> > > > >
> > > >
> > >
> /home/osallou/Development/NOSAVE/go-docker/godshared/tasks/pairtree_root/us/er/_o/sa/ll/ou/task
> > > > > >>>
> > > > >
> > > >
> > >
> /tmp/mesos/slaves/2a296daf-7419-4659-ade1-763c792cd522-S0/frameworks/aef1b0e3-ea2d-4770-baac-96d673ab88f9-0000/executors/51/runs/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/.rootfs/mnt/god-data
> > > > > >>>> mount: mount point
> > > > > >>>>
> > > > > >>>
> > > > >
> > > >
> > >
> /tmp/mesos/slaves/2a296daf-7419-4659-ade1-763c792cd522-S0/frameworks/aef1b0e3-ea2d-4770-baac-96d673ab88f9-0000/executors/51/runs/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/.rootfs/mnt/god-data
> > > > > >>>> does not exist
> > > > > >>>> Failed to execute a preparation shell command
> > > > > >>>>
> > > > > >>>> Then, my task switches to FAILED.
> > > > > >>>>
> > > > > >>>> I define a local volume to bind mount in my "container"
> > > > > >>>>
> > > > > >>>
> > > > >
> > > >
> > >
> /home/osallou/Development/NOSAVE/go-docker/godshared/tasks/pairtree_root/us/er/_o/sa/ll/ou/task
> > > > > >>>> => /mnt/god-data
> > > > > >>>> My directory exists on local server.
> > > > > >>>> In mesos UI, I can see the .rootfs directory along stdout and
> > > stderr
> > > > > >>>> files, and inside .rootfs, I can see /mnt/god-data (empty).
> > > > > >>>>
> > > > > >>>> Running the same using Docker containerizer instead of mesos
> > > > > >>>> containerizer (with a Docker image) works fine.
> > > > > >>>>
> > > > > >>>> It seems it fails to mount my local directory in the
> container.
> > > Any
> > > > > idea
> > > > > >>>> of what is going wrong or how to debug this?
> > > > > >>>>
> > > > > >>>>
> > > > > >>>> Thanks
> > > > > >>>>
> > > > > >>> --
> > > > > >>> Olivier Sallou
> > > > > >>> IRISA / University of Rennes 1
> > > > > >>> Campus de Beaulieu, 35000 RENNES - FRANCE
> > > > > >>> Tel: 02.99.84.71.95
> > > > > >>>
> > > > > >>> gpg key id: 4096R/326D8438 (keyring.debian.org)
> > > > > >>> Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D
> > > 8438
> > > > > >>>
> > > > > >>>
> > > > >
> > > > > --
> > > > > Olivier Sallou
> > > > > IRISA / University of Rennes 1
> > > > > Campus de Beaulieu, 35000 RENNES - FRANCE
> > > > > Tel: 02.99.84.71.95
> > > > >
> > > > > gpg key id: 4096R/326D8438 (keyring.debian.org)
> > > > > Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D
> 8438
> > > > >
> > > > >
> > > >
> > >
> >
>
Re: volume / mount point error with Unified Containerizer
Posted by Olivier Sallou <ol...@irisa.fr>.
----- Mail original -----
> De: "Gilbert Song" <gi...@mesosphere.io>
> À: "dev" <de...@mesos.apache.org>
> Envoyé: Jeudi 19 Mai 2016 01:57:16
> Objet: Re: volume / mount point error with Unified Containerizer
>
> @Olivier,
> In mesos 0.28.1, you are supposed to be able bind mount a volume from
> the host into the mesos container. Did you specify a docker image (we
> determine
> the mount point differently depending whether the container has a rootfs)?
Yes I specified an image, a Docker image URI.
> How
> do you specify your 'container_path' (the mount point in the container)? If
> it is an
> absolute path, we require that dir to be pre-existed. If it is a relative
> path, we will
> mkdir for it.
It is an absolute path, but it does not exists in image (this is the issue). Images are custom Docker images (images containing tools for batch computing), and I want, for example, to mount some shared resources (user home dir, common data, etc.) in the image. Of course those directories do not pre-exists in container images as they are specific to the environment. Requiring existence of the directory in the image is not issue as it prevents using any existing image from a repo.
When using Docker containerizer it works fine, I can mount any external storage in the container.
Olivie
>
> @Joshua,
> Thank for posting your workaround on mesos. As I mentioned above, in 0.28.1
> or
> older, we only mkdir for container_path which is relative path (not
> starting with "/").
> Because if no rootfs specified for a mesos container, the container shares
> the host
> root filesystem. Obviously we don't want any random files to be created
> implicitly
> on your host fs.
> From mesos 0.29 (release by the end of this month), we will mkdir the mount
> point in the container except for the command task case that specify an
> absolute
> container_path without a rootfs. Because we simplify the mounting logic, and
> sandbox bind mount will only be done in container mount namespace instead of
> host mount namespace (what we did before). Please keep tuned.
>
> Cheers,
> Gilbert
>
> On Wed, May 18, 2016 at 8:14 AM, Joshua Cohen <jc...@apache.org> wrote:
>
> > Hi Olivier,
> >
> > I touched on this issue as part of
> > https://issues.apache.org/jira/browse/MESOS-5229. It would be nice if
> > Mesos
> > automatically created container mount points if they don't already exist.
> > In the meantime, as a workaround for this, I've updated my filesystem
> > images to include the path (e.g. in Dockerfile, add `RUN mkdir -p
> > /some/mount/point`). Not the best solution, but the only thing I've seen
> > that works at the moment.
> >
> > Cheers,
> >
> > Joshua
> >
> > On Wed, May 18, 2016 at 7:36 AM, Guangya Liu <gy...@gmail.com> wrote:
> >
> > > It's pretty simple for you from scratch with source code
> > >
> > >
> > https://github.com/apache/mesos/blob/master/docs/getting-started.md#building-mesos
> > > ;-)
> > >
> > > Thanks,
> > >
> > > Guangya
> > >
> > > On Wed, May 18, 2016 at 8:30 PM, Olivier Sallou <olivier.sallou@irisa.fr
> > >
> > > wrote:
> > >
> > > >
> > > >
> > > > On 05/18/2016 02:31 PM, Guangya Liu wrote:
> > > > > Just saw that you are working with 0.28.1, the "docker volume driver"
> > > > code
> > > > > was not in 0.28.1, can you please have a try with mesos master branch
> > > if
> > > > > you are only doing some test?
> > > > this is indeed test only for the moment. But I will have to
> > > > recompile/install mesos :-( (I used packages for install).
> > > >
> > > > I will try when possible, but thanks for the hint.
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Guangya
> > > > >
> > > > > On Wed, May 18, 2016 at 8:28 PM, Guangya Liu <gy...@gmail.com>
> > > wrote:
> > > > >
> > > > >> Hi Olivier,
> > > > >>
> > > > >> I think that you need to enable "docker volume isolator" if you want
> > > use
> > > > >> external storage with unified container I was writing a document
> > here
> > > > >> https://reviews.apache.org/r/47511/, perhaps you can have a try
> > > > according
> > > > >> to the document and post some comments there if you find any issues.
> > > > >>
> > > > >> Also you can patch mesos-execute here
> > > > https://reviews.apache.org/r/46762/ to
> > > > >> have a try with mesos-execute.
> > > > >>
> > > > >> Thanks,
> > > > >>
> > > > >> Guangya
> > > > >>
> > > > >> On Wed, May 18, 2016 at 7:17 PM, Olivier Sallou <
> > > > olivier.sallou@irisa.fr>
> > > > >> wrote:
> > > > >>
> > > > >>> Answering (partially) to myself.
> > > > >>>
> > > > >>> I seems issue is container_path does not exists inside container.
> > On
> > > > >>> Docker, path is created and mounted. With pure mesos,
> > container_path
> > > > >>> must exists.
> > > > >>>
> > > > >>> mesos.proto says: "If the path is an absolute path, that path must
> > > > >>> already exist."
> > > > >>>
> > > > >>> This is an issue however, using Docker images, the path I want to
> > > mount
> > > > >>> does not exists, and it cannot be modified "on the fly".
> > > > >>>
> > > > >>> Is there a workaround for this ?
> > > > >>>
> > > > >>>
> > > > >>> On 05/18/2016 12:24 PM, Olivier Sallou wrote:
> > > > >>>> Hi,
> > > > >>>> I am trying unified containerizer on a single server
> > (master/slave)
> > > on
> > > > >>>> mesos 0.28.1, to switch from docker containerizer to mesos+docker
> > > > image
> > > > >>>> container.
> > > > >>>>
> > > > >>>> I have setup slave config as suggested in documentation:
> > > > >>>>
> > > > >>>> containerizers=docker,mesos
> > > > >>>> image_providers=docker \
> > > > >>>> isolation=filesystem/linux,docker/runtime
> > > > >>>>
> > > > >>>> However, when I execute my task with a volume I have an error:
> > > > >>>>
> > > > >>>> ....
> > > > >>>> + mount -n --rbind
> > > > >>>>
> > > > >>>
> > > >
> > >
> > /tmp/mesos/provisioner/containers/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/backends/copy/rootfses/f9f66bb2-308d-4555-ba77-49ec61cbeb4f
> > > > >>>
> > > >
> > >
> > /tmp/mesos/slaves/2a296daf-7419-4659-ade1-763c792cd522-S0/frameworks/aef1b0e3-ea2d-4770-baac-96d673ab88f9-0000/executors/51/runs/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/.rootfs
> > > > >>>> + mount -n --rbind
> > > > >>>>
> > > > >>>
> > > >
> > >
> > /home/osallou/Development/NOSAVE/go-docker/godshared/tasks/pairtree_root/us/er/_o/sa/ll/ou/task
> > > > >>>
> > > >
> > >
> > /tmp/mesos/slaves/2a296daf-7419-4659-ade1-763c792cd522-S0/frameworks/aef1b0e3-ea2d-4770-baac-96d673ab88f9-0000/executors/51/runs/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/.rootfs/mnt/god-data
> > > > >>>> mount: mount point
> > > > >>>>
> > > > >>>
> > > >
> > >
> > /tmp/mesos/slaves/2a296daf-7419-4659-ade1-763c792cd522-S0/frameworks/aef1b0e3-ea2d-4770-baac-96d673ab88f9-0000/executors/51/runs/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/.rootfs/mnt/god-data
> > > > >>>> does not exist
> > > > >>>> Failed to execute a preparation shell command
> > > > >>>>
> > > > >>>> Then, my task switches to FAILED.
> > > > >>>>
> > > > >>>> I define a local volume to bind mount in my "container"
> > > > >>>>
> > > > >>>
> > > >
> > >
> > /home/osallou/Development/NOSAVE/go-docker/godshared/tasks/pairtree_root/us/er/_o/sa/ll/ou/task
> > > > >>>> => /mnt/god-data
> > > > >>>> My directory exists on local server.
> > > > >>>> In mesos UI, I can see the .rootfs directory along stdout and
> > stderr
> > > > >>>> files, and inside .rootfs, I can see /mnt/god-data (empty).
> > > > >>>>
> > > > >>>> Running the same using Docker containerizer instead of mesos
> > > > >>>> containerizer (with a Docker image) works fine.
> > > > >>>>
> > > > >>>> It seems it fails to mount my local directory in the container.
> > Any
> > > > idea
> > > > >>>> of what is going wrong or how to debug this?
> > > > >>>>
> > > > >>>>
> > > > >>>> Thanks
> > > > >>>>
> > > > >>> --
> > > > >>> Olivier Sallou
> > > > >>> IRISA / University of Rennes 1
> > > > >>> Campus de Beaulieu, 35000 RENNES - FRANCE
> > > > >>> Tel: 02.99.84.71.95
> > > > >>>
> > > > >>> gpg key id: 4096R/326D8438 (keyring.debian.org)
> > > > >>> Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D
> > 8438
> > > > >>>
> > > > >>>
> > > >
> > > > --
> > > > Olivier Sallou
> > > > IRISA / University of Rennes 1
> > > > Campus de Beaulieu, 35000 RENNES - FRANCE
> > > > Tel: 02.99.84.71.95
> > > >
> > > > gpg key id: 4096R/326D8438 (keyring.debian.org)
> > > > Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D 8438
> > > >
> > > >
> > >
> >
>
Re: volume / mount point error with Unified Containerizer
Posted by Gilbert Song <gi...@mesosphere.io>.
@Olivier,
In mesos 0.28.1, you are supposed to be able bind mount a volume from
the host into the mesos container. Did you specify a docker image (we
determine
the mount point differently depending whether the container has a rootfs)?
How
do you specify your 'container_path' (the mount point in the container)? If
it is an
absolute path, we require that dir to be pre-existed. If it is a relative
path, we will
mkdir for it.
@Joshua,
Thank for posting your workaround on mesos. As I mentioned above, in 0.28.1
or
older, we only mkdir for container_path which is relative path (not
starting with "/").
Because if no rootfs specified for a mesos container, the container shares
the host
root filesystem. Obviously we don't want any random files to be created
implicitly
on your host fs.
From mesos 0.29 (release by the end of this month), we will mkdir the mount
point in the container except for the command task case that specify an
absolute
container_path without a rootfs. Because we simplify the mounting logic, and
sandbox bind mount will only be done in container mount namespace instead of
host mount namespace (what we did before). Please keep tuned.
Cheers,
Gilbert
On Wed, May 18, 2016 at 8:14 AM, Joshua Cohen <jc...@apache.org> wrote:
> Hi Olivier,
>
> I touched on this issue as part of
> https://issues.apache.org/jira/browse/MESOS-5229. It would be nice if
> Mesos
> automatically created container mount points if they don't already exist.
> In the meantime, as a workaround for this, I've updated my filesystem
> images to include the path (e.g. in Dockerfile, add `RUN mkdir -p
> /some/mount/point`). Not the best solution, but the only thing I've seen
> that works at the moment.
>
> Cheers,
>
> Joshua
>
> On Wed, May 18, 2016 at 7:36 AM, Guangya Liu <gy...@gmail.com> wrote:
>
> > It's pretty simple for you from scratch with source code
> >
> >
> https://github.com/apache/mesos/blob/master/docs/getting-started.md#building-mesos
> > ;-)
> >
> > Thanks,
> >
> > Guangya
> >
> > On Wed, May 18, 2016 at 8:30 PM, Olivier Sallou <olivier.sallou@irisa.fr
> >
> > wrote:
> >
> > >
> > >
> > > On 05/18/2016 02:31 PM, Guangya Liu wrote:
> > > > Just saw that you are working with 0.28.1, the "docker volume driver"
> > > code
> > > > was not in 0.28.1, can you please have a try with mesos master branch
> > if
> > > > you are only doing some test?
> > > this is indeed test only for the moment. But I will have to
> > > recompile/install mesos :-( (I used packages for install).
> > >
> > > I will try when possible, but thanks for the hint.
> > > >
> > > > Thanks,
> > > >
> > > > Guangya
> > > >
> > > > On Wed, May 18, 2016 at 8:28 PM, Guangya Liu <gy...@gmail.com>
> > wrote:
> > > >
> > > >> Hi Olivier,
> > > >>
> > > >> I think that you need to enable "docker volume isolator" if you want
> > use
> > > >> external storage with unified container I was writing a document
> here
> > > >> https://reviews.apache.org/r/47511/, perhaps you can have a try
> > > according
> > > >> to the document and post some comments there if you find any issues.
> > > >>
> > > >> Also you can patch mesos-execute here
> > > https://reviews.apache.org/r/46762/ to
> > > >> have a try with mesos-execute.
> > > >>
> > > >> Thanks,
> > > >>
> > > >> Guangya
> > > >>
> > > >> On Wed, May 18, 2016 at 7:17 PM, Olivier Sallou <
> > > olivier.sallou@irisa.fr>
> > > >> wrote:
> > > >>
> > > >>> Answering (partially) to myself.
> > > >>>
> > > >>> I seems issue is container_path does not exists inside container.
> On
> > > >>> Docker, path is created and mounted. With pure mesos,
> container_path
> > > >>> must exists.
> > > >>>
> > > >>> mesos.proto says: "If the path is an absolute path, that path must
> > > >>> already exist."
> > > >>>
> > > >>> This is an issue however, using Docker images, the path I want to
> > mount
> > > >>> does not exists, and it cannot be modified "on the fly".
> > > >>>
> > > >>> Is there a workaround for this ?
> > > >>>
> > > >>>
> > > >>> On 05/18/2016 12:24 PM, Olivier Sallou wrote:
> > > >>>> Hi,
> > > >>>> I am trying unified containerizer on a single server
> (master/slave)
> > on
> > > >>>> mesos 0.28.1, to switch from docker containerizer to mesos+docker
> > > image
> > > >>>> container.
> > > >>>>
> > > >>>> I have setup slave config as suggested in documentation:
> > > >>>>
> > > >>>> containerizers=docker,mesos
> > > >>>> image_providers=docker \
> > > >>>> isolation=filesystem/linux,docker/runtime
> > > >>>>
> > > >>>> However, when I execute my task with a volume I have an error:
> > > >>>>
> > > >>>> ....
> > > >>>> + mount -n --rbind
> > > >>>>
> > > >>>
> > >
> >
> /tmp/mesos/provisioner/containers/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/backends/copy/rootfses/f9f66bb2-308d-4555-ba77-49ec61cbeb4f
> > > >>>
> > >
> >
> /tmp/mesos/slaves/2a296daf-7419-4659-ade1-763c792cd522-S0/frameworks/aef1b0e3-ea2d-4770-baac-96d673ab88f9-0000/executors/51/runs/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/.rootfs
> > > >>>> + mount -n --rbind
> > > >>>>
> > > >>>
> > >
> >
> /home/osallou/Development/NOSAVE/go-docker/godshared/tasks/pairtree_root/us/er/_o/sa/ll/ou/task
> > > >>>
> > >
> >
> /tmp/mesos/slaves/2a296daf-7419-4659-ade1-763c792cd522-S0/frameworks/aef1b0e3-ea2d-4770-baac-96d673ab88f9-0000/executors/51/runs/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/.rootfs/mnt/god-data
> > > >>>> mount: mount point
> > > >>>>
> > > >>>
> > >
> >
> /tmp/mesos/slaves/2a296daf-7419-4659-ade1-763c792cd522-S0/frameworks/aef1b0e3-ea2d-4770-baac-96d673ab88f9-0000/executors/51/runs/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/.rootfs/mnt/god-data
> > > >>>> does not exist
> > > >>>> Failed to execute a preparation shell command
> > > >>>>
> > > >>>> Then, my task switches to FAILED.
> > > >>>>
> > > >>>> I define a local volume to bind mount in my "container"
> > > >>>>
> > > >>>
> > >
> >
> /home/osallou/Development/NOSAVE/go-docker/godshared/tasks/pairtree_root/us/er/_o/sa/ll/ou/task
> > > >>>> => /mnt/god-data
> > > >>>> My directory exists on local server.
> > > >>>> In mesos UI, I can see the .rootfs directory along stdout and
> stderr
> > > >>>> files, and inside .rootfs, I can see /mnt/god-data (empty).
> > > >>>>
> > > >>>> Running the same using Docker containerizer instead of mesos
> > > >>>> containerizer (with a Docker image) works fine.
> > > >>>>
> > > >>>> It seems it fails to mount my local directory in the container.
> Any
> > > idea
> > > >>>> of what is going wrong or how to debug this?
> > > >>>>
> > > >>>>
> > > >>>> Thanks
> > > >>>>
> > > >>> --
> > > >>> Olivier Sallou
> > > >>> IRISA / University of Rennes 1
> > > >>> Campus de Beaulieu, 35000 RENNES - FRANCE
> > > >>> Tel: 02.99.84.71.95
> > > >>>
> > > >>> gpg key id: 4096R/326D8438 (keyring.debian.org)
> > > >>> Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D
> 8438
> > > >>>
> > > >>>
> > >
> > > --
> > > Olivier Sallou
> > > IRISA / University of Rennes 1
> > > Campus de Beaulieu, 35000 RENNES - FRANCE
> > > Tel: 02.99.84.71.95
> > >
> > > gpg key id: 4096R/326D8438 (keyring.debian.org)
> > > Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D 8438
> > >
> > >
> >
>
Re: volume / mount point error with Unified Containerizer
Posted by Joshua Cohen <jc...@apache.org>.
Hi Olivier,
I touched on this issue as part of
https://issues.apache.org/jira/browse/MESOS-5229. It would be nice if Mesos
automatically created container mount points if they don't already exist.
In the meantime, as a workaround for this, I've updated my filesystem
images to include the path (e.g. in Dockerfile, add `RUN mkdir -p
/some/mount/point`). Not the best solution, but the only thing I've seen
that works at the moment.
Cheers,
Joshua
On Wed, May 18, 2016 at 7:36 AM, Guangya Liu <gy...@gmail.com> wrote:
> It's pretty simple for you from scratch with source code
>
> https://github.com/apache/mesos/blob/master/docs/getting-started.md#building-mesos
> ;-)
>
> Thanks,
>
> Guangya
>
> On Wed, May 18, 2016 at 8:30 PM, Olivier Sallou <ol...@irisa.fr>
> wrote:
>
> >
> >
> > On 05/18/2016 02:31 PM, Guangya Liu wrote:
> > > Just saw that you are working with 0.28.1, the "docker volume driver"
> > code
> > > was not in 0.28.1, can you please have a try with mesos master branch
> if
> > > you are only doing some test?
> > this is indeed test only for the moment. But I will have to
> > recompile/install mesos :-( (I used packages for install).
> >
> > I will try when possible, but thanks for the hint.
> > >
> > > Thanks,
> > >
> > > Guangya
> > >
> > > On Wed, May 18, 2016 at 8:28 PM, Guangya Liu <gy...@gmail.com>
> wrote:
> > >
> > >> Hi Olivier,
> > >>
> > >> I think that you need to enable "docker volume isolator" if you want
> use
> > >> external storage with unified container I was writing a document here
> > >> https://reviews.apache.org/r/47511/, perhaps you can have a try
> > according
> > >> to the document and post some comments there if you find any issues.
> > >>
> > >> Also you can patch mesos-execute here
> > https://reviews.apache.org/r/46762/ to
> > >> have a try with mesos-execute.
> > >>
> > >> Thanks,
> > >>
> > >> Guangya
> > >>
> > >> On Wed, May 18, 2016 at 7:17 PM, Olivier Sallou <
> > olivier.sallou@irisa.fr>
> > >> wrote:
> > >>
> > >>> Answering (partially) to myself.
> > >>>
> > >>> I seems issue is container_path does not exists inside container. On
> > >>> Docker, path is created and mounted. With pure mesos, container_path
> > >>> must exists.
> > >>>
> > >>> mesos.proto says: "If the path is an absolute path, that path must
> > >>> already exist."
> > >>>
> > >>> This is an issue however, using Docker images, the path I want to
> mount
> > >>> does not exists, and it cannot be modified "on the fly".
> > >>>
> > >>> Is there a workaround for this ?
> > >>>
> > >>>
> > >>> On 05/18/2016 12:24 PM, Olivier Sallou wrote:
> > >>>> Hi,
> > >>>> I am trying unified containerizer on a single server (master/slave)
> on
> > >>>> mesos 0.28.1, to switch from docker containerizer to mesos+docker
> > image
> > >>>> container.
> > >>>>
> > >>>> I have setup slave config as suggested in documentation:
> > >>>>
> > >>>> containerizers=docker,mesos
> > >>>> image_providers=docker \
> > >>>> isolation=filesystem/linux,docker/runtime
> > >>>>
> > >>>> However, when I execute my task with a volume I have an error:
> > >>>>
> > >>>> ....
> > >>>> + mount -n --rbind
> > >>>>
> > >>>
> >
> /tmp/mesos/provisioner/containers/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/backends/copy/rootfses/f9f66bb2-308d-4555-ba77-49ec61cbeb4f
> > >>>
> >
> /tmp/mesos/slaves/2a296daf-7419-4659-ade1-763c792cd522-S0/frameworks/aef1b0e3-ea2d-4770-baac-96d673ab88f9-0000/executors/51/runs/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/.rootfs
> > >>>> + mount -n --rbind
> > >>>>
> > >>>
> >
> /home/osallou/Development/NOSAVE/go-docker/godshared/tasks/pairtree_root/us/er/_o/sa/ll/ou/task
> > >>>
> >
> /tmp/mesos/slaves/2a296daf-7419-4659-ade1-763c792cd522-S0/frameworks/aef1b0e3-ea2d-4770-baac-96d673ab88f9-0000/executors/51/runs/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/.rootfs/mnt/god-data
> > >>>> mount: mount point
> > >>>>
> > >>>
> >
> /tmp/mesos/slaves/2a296daf-7419-4659-ade1-763c792cd522-S0/frameworks/aef1b0e3-ea2d-4770-baac-96d673ab88f9-0000/executors/51/runs/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/.rootfs/mnt/god-data
> > >>>> does not exist
> > >>>> Failed to execute a preparation shell command
> > >>>>
> > >>>> Then, my task switches to FAILED.
> > >>>>
> > >>>> I define a local volume to bind mount in my "container"
> > >>>>
> > >>>
> >
> /home/osallou/Development/NOSAVE/go-docker/godshared/tasks/pairtree_root/us/er/_o/sa/ll/ou/task
> > >>>> => /mnt/god-data
> > >>>> My directory exists on local server.
> > >>>> In mesos UI, I can see the .rootfs directory along stdout and stderr
> > >>>> files, and inside .rootfs, I can see /mnt/god-data (empty).
> > >>>>
> > >>>> Running the same using Docker containerizer instead of mesos
> > >>>> containerizer (with a Docker image) works fine.
> > >>>>
> > >>>> It seems it fails to mount my local directory in the container. Any
> > idea
> > >>>> of what is going wrong or how to debug this?
> > >>>>
> > >>>>
> > >>>> Thanks
> > >>>>
> > >>> --
> > >>> Olivier Sallou
> > >>> IRISA / University of Rennes 1
> > >>> Campus de Beaulieu, 35000 RENNES - FRANCE
> > >>> Tel: 02.99.84.71.95
> > >>>
> > >>> gpg key id: 4096R/326D8438 (keyring.debian.org)
> > >>> Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D 8438
> > >>>
> > >>>
> >
> > --
> > Olivier Sallou
> > IRISA / University of Rennes 1
> > Campus de Beaulieu, 35000 RENNES - FRANCE
> > Tel: 02.99.84.71.95
> >
> > gpg key id: 4096R/326D8438 (keyring.debian.org)
> > Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D 8438
> >
> >
>
Re: volume / mount point error with Unified Containerizer
Posted by Guangya Liu <gy...@gmail.com>.
It's pretty simple for you from scratch with source code
https://github.com/apache/mesos/blob/master/docs/getting-started.md#building-mesos
;-)
Thanks,
Guangya
On Wed, May 18, 2016 at 8:30 PM, Olivier Sallou <ol...@irisa.fr>
wrote:
>
>
> On 05/18/2016 02:31 PM, Guangya Liu wrote:
> > Just saw that you are working with 0.28.1, the "docker volume driver"
> code
> > was not in 0.28.1, can you please have a try with mesos master branch if
> > you are only doing some test?
> this is indeed test only for the moment. But I will have to
> recompile/install mesos :-( (I used packages for install).
>
> I will try when possible, but thanks for the hint.
> >
> > Thanks,
> >
> > Guangya
> >
> > On Wed, May 18, 2016 at 8:28 PM, Guangya Liu <gy...@gmail.com> wrote:
> >
> >> Hi Olivier,
> >>
> >> I think that you need to enable "docker volume isolator" if you want use
> >> external storage with unified container I was writing a document here
> >> https://reviews.apache.org/r/47511/, perhaps you can have a try
> according
> >> to the document and post some comments there if you find any issues.
> >>
> >> Also you can patch mesos-execute here
> https://reviews.apache.org/r/46762/ to
> >> have a try with mesos-execute.
> >>
> >> Thanks,
> >>
> >> Guangya
> >>
> >> On Wed, May 18, 2016 at 7:17 PM, Olivier Sallou <
> olivier.sallou@irisa.fr>
> >> wrote:
> >>
> >>> Answering (partially) to myself.
> >>>
> >>> I seems issue is container_path does not exists inside container. On
> >>> Docker, path is created and mounted. With pure mesos, container_path
> >>> must exists.
> >>>
> >>> mesos.proto says: "If the path is an absolute path, that path must
> >>> already exist."
> >>>
> >>> This is an issue however, using Docker images, the path I want to mount
> >>> does not exists, and it cannot be modified "on the fly".
> >>>
> >>> Is there a workaround for this ?
> >>>
> >>>
> >>> On 05/18/2016 12:24 PM, Olivier Sallou wrote:
> >>>> Hi,
> >>>> I am trying unified containerizer on a single server (master/slave) on
> >>>> mesos 0.28.1, to switch from docker containerizer to mesos+docker
> image
> >>>> container.
> >>>>
> >>>> I have setup slave config as suggested in documentation:
> >>>>
> >>>> containerizers=docker,mesos
> >>>> image_providers=docker \
> >>>> isolation=filesystem/linux,docker/runtime
> >>>>
> >>>> However, when I execute my task with a volume I have an error:
> >>>>
> >>>> ....
> >>>> + mount -n --rbind
> >>>>
> >>>
> /tmp/mesos/provisioner/containers/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/backends/copy/rootfses/f9f66bb2-308d-4555-ba77-49ec61cbeb4f
> >>>
> /tmp/mesos/slaves/2a296daf-7419-4659-ade1-763c792cd522-S0/frameworks/aef1b0e3-ea2d-4770-baac-96d673ab88f9-0000/executors/51/runs/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/.rootfs
> >>>> + mount -n --rbind
> >>>>
> >>>
> /home/osallou/Development/NOSAVE/go-docker/godshared/tasks/pairtree_root/us/er/_o/sa/ll/ou/task
> >>>
> /tmp/mesos/slaves/2a296daf-7419-4659-ade1-763c792cd522-S0/frameworks/aef1b0e3-ea2d-4770-baac-96d673ab88f9-0000/executors/51/runs/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/.rootfs/mnt/god-data
> >>>> mount: mount point
> >>>>
> >>>
> /tmp/mesos/slaves/2a296daf-7419-4659-ade1-763c792cd522-S0/frameworks/aef1b0e3-ea2d-4770-baac-96d673ab88f9-0000/executors/51/runs/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/.rootfs/mnt/god-data
> >>>> does not exist
> >>>> Failed to execute a preparation shell command
> >>>>
> >>>> Then, my task switches to FAILED.
> >>>>
> >>>> I define a local volume to bind mount in my "container"
> >>>>
> >>>
> /home/osallou/Development/NOSAVE/go-docker/godshared/tasks/pairtree_root/us/er/_o/sa/ll/ou/task
> >>>> => /mnt/god-data
> >>>> My directory exists on local server.
> >>>> In mesos UI, I can see the .rootfs directory along stdout and stderr
> >>>> files, and inside .rootfs, I can see /mnt/god-data (empty).
> >>>>
> >>>> Running the same using Docker containerizer instead of mesos
> >>>> containerizer (with a Docker image) works fine.
> >>>>
> >>>> It seems it fails to mount my local directory in the container. Any
> idea
> >>>> of what is going wrong or how to debug this?
> >>>>
> >>>>
> >>>> Thanks
> >>>>
> >>> --
> >>> Olivier Sallou
> >>> IRISA / University of Rennes 1
> >>> Campus de Beaulieu, 35000 RENNES - FRANCE
> >>> Tel: 02.99.84.71.95
> >>>
> >>> gpg key id: 4096R/326D8438 (keyring.debian.org)
> >>> Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D 8438
> >>>
> >>>
>
> --
> Olivier Sallou
> IRISA / University of Rennes 1
> Campus de Beaulieu, 35000 RENNES - FRANCE
> Tel: 02.99.84.71.95
>
> gpg key id: 4096R/326D8438 (keyring.debian.org)
> Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D 8438
>
>
Re: volume / mount point error with Unified Containerizer
Posted by Olivier Sallou <ol...@irisa.fr>.
On 05/18/2016 02:31 PM, Guangya Liu wrote:
> Just saw that you are working with 0.28.1, the "docker volume driver" code
> was not in 0.28.1, can you please have a try with mesos master branch if
> you are only doing some test?
this is indeed test only for the moment. But I will have to
recompile/install mesos :-( (I used packages for install).
I will try when possible, but thanks for the hint.
>
> Thanks,
>
> Guangya
>
> On Wed, May 18, 2016 at 8:28 PM, Guangya Liu <gy...@gmail.com> wrote:
>
>> Hi Olivier,
>>
>> I think that you need to enable "docker volume isolator" if you want use
>> external storage with unified container I was writing a document here
>> https://reviews.apache.org/r/47511/, perhaps you can have a try according
>> to the document and post some comments there if you find any issues.
>>
>> Also you can patch mesos-execute here https://reviews.apache.org/r/46762/ to
>> have a try with mesos-execute.
>>
>> Thanks,
>>
>> Guangya
>>
>> On Wed, May 18, 2016 at 7:17 PM, Olivier Sallou <ol...@irisa.fr>
>> wrote:
>>
>>> Answering (partially) to myself.
>>>
>>> I seems issue is container_path does not exists inside container. On
>>> Docker, path is created and mounted. With pure mesos, container_path
>>> must exists.
>>>
>>> mesos.proto says: "If the path is an absolute path, that path must
>>> already exist."
>>>
>>> This is an issue however, using Docker images, the path I want to mount
>>> does not exists, and it cannot be modified "on the fly".
>>>
>>> Is there a workaround for this ?
>>>
>>>
>>> On 05/18/2016 12:24 PM, Olivier Sallou wrote:
>>>> Hi,
>>>> I am trying unified containerizer on a single server (master/slave) on
>>>> mesos 0.28.1, to switch from docker containerizer to mesos+docker image
>>>> container.
>>>>
>>>> I have setup slave config as suggested in documentation:
>>>>
>>>> containerizers=docker,mesos
>>>> image_providers=docker \
>>>> isolation=filesystem/linux,docker/runtime
>>>>
>>>> However, when I execute my task with a volume I have an error:
>>>>
>>>> ....
>>>> + mount -n --rbind
>>>>
>>> /tmp/mesos/provisioner/containers/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/backends/copy/rootfses/f9f66bb2-308d-4555-ba77-49ec61cbeb4f
>>> /tmp/mesos/slaves/2a296daf-7419-4659-ade1-763c792cd522-S0/frameworks/aef1b0e3-ea2d-4770-baac-96d673ab88f9-0000/executors/51/runs/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/.rootfs
>>>> + mount -n --rbind
>>>>
>>> /home/osallou/Development/NOSAVE/go-docker/godshared/tasks/pairtree_root/us/er/_o/sa/ll/ou/task
>>> /tmp/mesos/slaves/2a296daf-7419-4659-ade1-763c792cd522-S0/frameworks/aef1b0e3-ea2d-4770-baac-96d673ab88f9-0000/executors/51/runs/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/.rootfs/mnt/god-data
>>>> mount: mount point
>>>>
>>> /tmp/mesos/slaves/2a296daf-7419-4659-ade1-763c792cd522-S0/frameworks/aef1b0e3-ea2d-4770-baac-96d673ab88f9-0000/executors/51/runs/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/.rootfs/mnt/god-data
>>>> does not exist
>>>> Failed to execute a preparation shell command
>>>>
>>>> Then, my task switches to FAILED.
>>>>
>>>> I define a local volume to bind mount in my "container"
>>>>
>>> /home/osallou/Development/NOSAVE/go-docker/godshared/tasks/pairtree_root/us/er/_o/sa/ll/ou/task
>>>> => /mnt/god-data
>>>> My directory exists on local server.
>>>> In mesos UI, I can see the .rootfs directory along stdout and stderr
>>>> files, and inside .rootfs, I can see /mnt/god-data (empty).
>>>>
>>>> Running the same using Docker containerizer instead of mesos
>>>> containerizer (with a Docker image) works fine.
>>>>
>>>> It seems it fails to mount my local directory in the container. Any idea
>>>> of what is going wrong or how to debug this?
>>>>
>>>>
>>>> Thanks
>>>>
>>> --
>>> Olivier Sallou
>>> IRISA / University of Rennes 1
>>> Campus de Beaulieu, 35000 RENNES - FRANCE
>>> Tel: 02.99.84.71.95
>>>
>>> gpg key id: 4096R/326D8438 (keyring.debian.org)
>>> Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D 8438
>>>
>>>
--
Olivier Sallou
IRISA / University of Rennes 1
Campus de Beaulieu, 35000 RENNES - FRANCE
Tel: 02.99.84.71.95
gpg key id: 4096R/326D8438 (keyring.debian.org)
Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D 8438
Re: volume / mount point error with Unified Containerizer
Posted by Guangya Liu <gy...@gmail.com>.
Just saw that you are working with 0.28.1, the "docker volume driver" code
was not in 0.28.1, can you please have a try with mesos master branch if
you are only doing some test?
Thanks,
Guangya
On Wed, May 18, 2016 at 8:28 PM, Guangya Liu <gy...@gmail.com> wrote:
> Hi Olivier,
>
> I think that you need to enable "docker volume isolator" if you want use
> external storage with unified container I was writing a document here
> https://reviews.apache.org/r/47511/, perhaps you can have a try according
> to the document and post some comments there if you find any issues.
>
> Also you can patch mesos-execute here https://reviews.apache.org/r/46762/ to
> have a try with mesos-execute.
>
> Thanks,
>
> Guangya
>
> On Wed, May 18, 2016 at 7:17 PM, Olivier Sallou <ol...@irisa.fr>
> wrote:
>
>> Answering (partially) to myself.
>>
>> I seems issue is container_path does not exists inside container. On
>> Docker, path is created and mounted. With pure mesos, container_path
>> must exists.
>>
>> mesos.proto says: "If the path is an absolute path, that path must
>> already exist."
>>
>> This is an issue however, using Docker images, the path I want to mount
>> does not exists, and it cannot be modified "on the fly".
>>
>> Is there a workaround for this ?
>>
>>
>> On 05/18/2016 12:24 PM, Olivier Sallou wrote:
>> > Hi,
>> > I am trying unified containerizer on a single server (master/slave) on
>> > mesos 0.28.1, to switch from docker containerizer to mesos+docker image
>> > container.
>> >
>> > I have setup slave config as suggested in documentation:
>> >
>> > containerizers=docker,mesos
>> > image_providers=docker \
>> > isolation=filesystem/linux,docker/runtime
>> >
>> > However, when I execute my task with a volume I have an error:
>> >
>> > ....
>> > + mount -n --rbind
>> >
>> /tmp/mesos/provisioner/containers/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/backends/copy/rootfses/f9f66bb2-308d-4555-ba77-49ec61cbeb4f
>> >
>> /tmp/mesos/slaves/2a296daf-7419-4659-ade1-763c792cd522-S0/frameworks/aef1b0e3-ea2d-4770-baac-96d673ab88f9-0000/executors/51/runs/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/.rootfs
>> > + mount -n --rbind
>> >
>> /home/osallou/Development/NOSAVE/go-docker/godshared/tasks/pairtree_root/us/er/_o/sa/ll/ou/task
>> >
>> /tmp/mesos/slaves/2a296daf-7419-4659-ade1-763c792cd522-S0/frameworks/aef1b0e3-ea2d-4770-baac-96d673ab88f9-0000/executors/51/runs/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/.rootfs/mnt/god-data
>> > mount: mount point
>> >
>> /tmp/mesos/slaves/2a296daf-7419-4659-ade1-763c792cd522-S0/frameworks/aef1b0e3-ea2d-4770-baac-96d673ab88f9-0000/executors/51/runs/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/.rootfs/mnt/god-data
>> > does not exist
>> > Failed to execute a preparation shell command
>> >
>> > Then, my task switches to FAILED.
>> >
>> > I define a local volume to bind mount in my "container"
>> >
>> /home/osallou/Development/NOSAVE/go-docker/godshared/tasks/pairtree_root/us/er/_o/sa/ll/ou/task
>> > => /mnt/god-data
>> > My directory exists on local server.
>> > In mesos UI, I can see the .rootfs directory along stdout and stderr
>> > files, and inside .rootfs, I can see /mnt/god-data (empty).
>> >
>> > Running the same using Docker containerizer instead of mesos
>> > containerizer (with a Docker image) works fine.
>> >
>> > It seems it fails to mount my local directory in the container. Any idea
>> > of what is going wrong or how to debug this?
>> >
>> >
>> > Thanks
>> >
>>
>> --
>> Olivier Sallou
>> IRISA / University of Rennes 1
>> Campus de Beaulieu, 35000 RENNES - FRANCE
>> Tel: 02.99.84.71.95
>>
>> gpg key id: 4096R/326D8438 (keyring.debian.org)
>> Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D 8438
>>
>>
>
Re: volume / mount point error with Unified Containerizer
Posted by Guangya Liu <gy...@gmail.com>.
Hi Olivier,
I think that you need to enable "docker volume isolator" if you want use
external storage with unified container I was writing a document here
https://reviews.apache.org/r/47511/, perhaps you can have a try according
to the document and post some comments there if you find any issues.
Also you can patch mesos-execute here https://reviews.apache.org/r/46762/ to
have a try with mesos-execute.
Thanks,
Guangya
On Wed, May 18, 2016 at 7:17 PM, Olivier Sallou <ol...@irisa.fr>
wrote:
> Answering (partially) to myself.
>
> I seems issue is container_path does not exists inside container. On
> Docker, path is created and mounted. With pure mesos, container_path
> must exists.
>
> mesos.proto says: "If the path is an absolute path, that path must
> already exist."
>
> This is an issue however, using Docker images, the path I want to mount
> does not exists, and it cannot be modified "on the fly".
>
> Is there a workaround for this ?
>
>
> On 05/18/2016 12:24 PM, Olivier Sallou wrote:
> > Hi,
> > I am trying unified containerizer on a single server (master/slave) on
> > mesos 0.28.1, to switch from docker containerizer to mesos+docker image
> > container.
> >
> > I have setup slave config as suggested in documentation:
> >
> > containerizers=docker,mesos
> > image_providers=docker \
> > isolation=filesystem/linux,docker/runtime
> >
> > However, when I execute my task with a volume I have an error:
> >
> > ....
> > + mount -n --rbind
> >
> /tmp/mesos/provisioner/containers/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/backends/copy/rootfses/f9f66bb2-308d-4555-ba77-49ec61cbeb4f
> >
> /tmp/mesos/slaves/2a296daf-7419-4659-ade1-763c792cd522-S0/frameworks/aef1b0e3-ea2d-4770-baac-96d673ab88f9-0000/executors/51/runs/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/.rootfs
> > + mount -n --rbind
> >
> /home/osallou/Development/NOSAVE/go-docker/godshared/tasks/pairtree_root/us/er/_o/sa/ll/ou/task
> >
> /tmp/mesos/slaves/2a296daf-7419-4659-ade1-763c792cd522-S0/frameworks/aef1b0e3-ea2d-4770-baac-96d673ab88f9-0000/executors/51/runs/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/.rootfs/mnt/god-data
> > mount: mount point
> >
> /tmp/mesos/slaves/2a296daf-7419-4659-ade1-763c792cd522-S0/frameworks/aef1b0e3-ea2d-4770-baac-96d673ab88f9-0000/executors/51/runs/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/.rootfs/mnt/god-data
> > does not exist
> > Failed to execute a preparation shell command
> >
> > Then, my task switches to FAILED.
> >
> > I define a local volume to bind mount in my "container"
> >
> /home/osallou/Development/NOSAVE/go-docker/godshared/tasks/pairtree_root/us/er/_o/sa/ll/ou/task
> > => /mnt/god-data
> > My directory exists on local server.
> > In mesos UI, I can see the .rootfs directory along stdout and stderr
> > files, and inside .rootfs, I can see /mnt/god-data (empty).
> >
> > Running the same using Docker containerizer instead of mesos
> > containerizer (with a Docker image) works fine.
> >
> > It seems it fails to mount my local directory in the container. Any idea
> > of what is going wrong or how to debug this?
> >
> >
> > Thanks
> >
>
> --
> Olivier Sallou
> IRISA / University of Rennes 1
> Campus de Beaulieu, 35000 RENNES - FRANCE
> Tel: 02.99.84.71.95
>
> gpg key id: 4096R/326D8438 (keyring.debian.org)
> Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D 8438
>
>
Re: volume / mount point error with Unified Containerizer
Posted by Olivier Sallou <ol...@irisa.fr>.
Answering (partially) to myself.
I seems issue is container_path does not exists inside container. On
Docker, path is created and mounted. With pure mesos, container_path
must exists.
mesos.proto says: "If the path is an absolute path, that path must
already exist."
This is an issue however, using Docker images, the path I want to mount
does not exists, and it cannot be modified "on the fly".
Is there a workaround for this ?
On 05/18/2016 12:24 PM, Olivier Sallou wrote:
> Hi,
> I am trying unified containerizer on a single server (master/slave) on
> mesos 0.28.1, to switch from docker containerizer to mesos+docker image
> container.
>
> I have setup slave config as suggested in documentation:
>
> containerizers=docker,mesos
> image_providers=docker \
> isolation=filesystem/linux,docker/runtime
>
> However, when I execute my task with a volume I have an error:
>
> ....
> + mount -n --rbind
> /tmp/mesos/provisioner/containers/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/backends/copy/rootfses/f9f66bb2-308d-4555-ba77-49ec61cbeb4f
> /tmp/mesos/slaves/2a296daf-7419-4659-ade1-763c792cd522-S0/frameworks/aef1b0e3-ea2d-4770-baac-96d673ab88f9-0000/executors/51/runs/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/.rootfs
> + mount -n --rbind
> /home/osallou/Development/NOSAVE/go-docker/godshared/tasks/pairtree_root/us/er/_o/sa/ll/ou/task
> /tmp/mesos/slaves/2a296daf-7419-4659-ade1-763c792cd522-S0/frameworks/aef1b0e3-ea2d-4770-baac-96d673ab88f9-0000/executors/51/runs/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/.rootfs/mnt/god-data
> mount: mount point
> /tmp/mesos/slaves/2a296daf-7419-4659-ade1-763c792cd522-S0/frameworks/aef1b0e3-ea2d-4770-baac-96d673ab88f9-0000/executors/51/runs/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/.rootfs/mnt/god-data
> does not exist
> Failed to execute a preparation shell command
>
> Then, my task switches to FAILED.
>
> I define a local volume to bind mount in my "container"
> /home/osallou/Development/NOSAVE/go-docker/godshared/tasks/pairtree_root/us/er/_o/sa/ll/ou/task
> => /mnt/god-data
> My directory exists on local server.
> In mesos UI, I can see the .rootfs directory along stdout and stderr
> files, and inside .rootfs, I can see /mnt/god-data (empty).
>
> Running the same using Docker containerizer instead of mesos
> containerizer (with a Docker image) works fine.
>
> It seems it fails to mount my local directory in the container. Any idea
> of what is going wrong or how to debug this?
>
>
> Thanks
>
--
Olivier Sallou
IRISA / University of Rennes 1
Campus de Beaulieu, 35000 RENNES - FRANCE
Tel: 02.99.84.71.95
gpg key id: 4096R/326D8438 (keyring.debian.org)
Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D 8438