You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@mesos.apache.org by Dmitriy Lyubimov <dl...@gmail.com> on 2013/06/13 01:26:40 UTC
Mesos, lxc and ubuntu 12
Hello,
is there anything speicific to ubuntu 12 that needs to be done to make
Mesos work with LCX?
I set things up according to ubuntu docs,
https://help.ubuntu.com/12.10/serverguide/lxc.html#lxc-creation
and all container examples there seem to be happily working.
However, some mesos unit tests are failing (which i suspect are relating to
lxc) as well as lxc isolation mode fails to spawn tasks.
(I am actually on ubuntu 12-04 LTS).
Is there any speicific way to troubleshoot this? Is LXC in Mesos even
working with Ubuntu 12?
thank you in advance. (slave output enclosed).
-d
I0612 16:24:20.682698 26452 slave.cpp:474] Got assigned task 0 for
framework 201306121623-16777343-5050-26417-0000
I0612 16:24:20.683425 26452 paths.hpp:234] Created executor directory
'/tmp/mesos/slaves/201306121623-16777343-5050-26417-0/frameworks/201306121623-16777343-5050-26417-0000/executors/Task
0 ("/home/dmitr...)/runs/9156d4fa-a177-464b-906f-fb62c8b9b363'
I0612 16:24:20.683630 26453 lxc_isolation_module.cpp:121] Launching Task 0
("/home/dmitr...) (/usr/local/libexec/mesos/mesos-executor) in
/tmp/mesos/slaves/201306121623-16777343-5050-26417-0/frameworks/201306121623-16777343-5050-26417-0000/executors/Task
0 ("/home/dmitr...)/runs/9156d4fa-a177-464b-906f-fb62c8b9b363 with
resources ' for framework 201306121623-16777343-5050-26417-0000
I0612 16:24:20.683945 26453 lxc_isolation_module.cpp:152] Forked executor
at = 26570
lxc-execute: No such file or directory - failed to create
'/sys/fs/cgroup/cpuset//lxc/mesos_executor_Task 0
("/home/dmitr...)_framework_201306121623-16777343-5050-26417-0000' directory
lxc-execute: failed to spawn 'mesos_executor_Task 0
("/home/dmitr...)_framework_201306121623-16777343-5050-26417-0000'
lxc-execute: No such file or directory - failed to remove cgroup
'/sys/fs/cgroup/cpuset//lxc/mesos_executor_Task 0
("/home/dmitr...)_framework_201306121623-16777343-5050-26417-0000'
I0612 16:24:21.451616 26452 lxc_isolation_module.cpp:322] Telling slave of
lost executor Task 0 ("/home/dmitr...) of framework
201306121623-16777343-5050-26417-0000
I0612 16:24:21.451709 26452 lxc_isolation_module.cpp:239] Stopping
container mesos_executor_Task 0
("/home/dmitr...)_framework_201306121623-16777343-5050-26417-0000
I0612 16:24:21.452199 26454 slave.cpp:998] Executor 'Task 0
("/home/dmitr...)' of framework 201306121623-16777343-5050-26417-0000 has
exited with status 255
sh: 1: Syntax error: "(" unexpected
E0612 16:24:21.453227 26452 lxc_isolation_module.cpp:248] Failed to stop
container mesos_executor_Task 0
("/home/dmitr...)_framework_201306121623-16777343-5050-26417-0000, lxc-stop
returned: 512
I0612 16:24:21.453385 26454 slave.cpp:829] Status update: task 0 of
framework 201306121623-16777343-5050-26417-0000 is now in state TASK_FAILED
E0612 16:24:21.453583 26453 lxc_isolation_module.cpp:273] ERROR! Asked to
update resources for an unknown executor!
I0612 16:24:21.453891 26451 gc.cpp:97] Scheduling
/tmp/mesos/slaves/201306121623-16777343-5050-26417-0/frameworks/201306121623-16777343-5050-26417-0000/executors/Task
0 ("/home/dmitr...)/runs/9156d4fa-a177-464b-906f-fb62c8b9b363 for removal
Re: Mesos, lxc and ubuntu 12
Posted by Dmitriy Lyubimov <dl...@gmail.com>.
tried also
cgroups_hierarchy_root=/sys/fs/cgroup/mesos
and then i don't get the aforementioned error but instead I am back at
square one:
F0613 12:46:12.833212 25419 cgroups_isolation_module.cpp:161] Required
subsystem 'freezer' is already in use
So there's something about that freezer and its being in use. Somebody
rummaging thru it.
On Thu, Jun 13, 2013 at 11:35 AM, Dmitriy Lyubimov <dl...@gmail.com>wrote:
> Thank you for your reply.
>
> Now getting this FATAL message :
>
>
> F0613 11:27:48.010704 24340 cgroups_isolation_module.cpp:147] Cannot
> create cgroups hierarchy root at /sys/fs/cgroup. Consider removing it
>
>
>
> On Wed, Jun 12, 2013 at 6:06 PM, Benjamin Mahler <
> benjamin.mahler@gmail.com> wrote:
>
>> Can you also pass --cgroups_hierarchy_root=/sys/fs/cgroup to the slave?
>>
>> In 0.11.0, we don't make an attempt to detect the cgroup hierarchy
>> location.
>>
>>
>> On Wed, Jun 12, 2013 at 6:01 PM, Dmitriy Lyubimov <dl...@gmail.com>wrote:
>>
>>> also, listing cgroup produces :
>>> dmitriy@BigHP:~$ lscgroup
>>> cpu:/
>>> cpu:/sysdefault
>>> cpuacct:/
>>> cpuacct:/sysdefault
>>> devices:/
>>> devices:/sysdefault
>>> memory:/
>>> memory:/sysdefault
>>> freezer:/
>>> freezer:/sysdefault
>>>
>>>
>>>
>>> On Wed, Jun 12, 2013 at 6:01 PM, Dmitriy Lyubimov <dl...@gmail.com>wrote:
>>>
>>>> also, listing cgroups produces
>>>>
>>>>
>>>>
>>>> On Wed, Jun 12, 2013 at 5:42 PM, Dmitriy Lyubimov <dl...@gmail.com>wrote:
>>>>
>>>>> and here's the list of subsystems if it is relevant.
>>>>> lssubsys -am
>>>>> cpuset
>>>>> cpu /sys/fs/cgroup/cpu
>>>>> cpuacct /sys/fs/cgroup/cpuacct
>>>>> memory /sys/fs/cgroup/memory
>>>>> devices /sys/fs/cgroup/devices
>>>>> freezer /sys/fs/cgroup/freezer
>>>>> blkio
>>>>> perf_event
>>>>>
>>>>>
>>>>>
>>>>> On Wed, Jun 12, 2013 at 5:34 PM, Dmitriy Lyubimov <dl...@gmail.com>wrote:
>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Wed, Jun 12, 2013 at 5:24 PM, Benjamin Mahler <
>>>>>> benjamin.mahler@gmail.com> wrote:
>>>>>>
>>>>>>> Cgroups does not allow one to mount the same subsystem across cgroup
>>>>>>> hierarchies. Do you have multiple cgroup hierarchies present on your
>>>>>>> machine?
>>>>>>>
>>>>>>
>>>>>> I frankly know nothing of cgroups. how do i check? not that i know of
>>>>>> .
>>>>>>
>>>>>>>
>>>>>>> Ideally this will work with a stock ubuntu 12 OS, but it's possible
>>>>>>> that ubuntu already mounts a cgroup hierarchy with the freezer subsystem in
>>>>>>> a location we did not expect.
>>>>>>>
>>>>>>> What are the contents of the root directory on that machine?
>>>>>>>
>>>>>> bin cdrom etc home initrd.img.old lib64 media
>>>>>> opt root sbin srv tmp var vmlinuz.old
>>>>>> boot dev hadoop initrd.img lib lost+found mnt
>>>>>> proc run selinux sys usr vmlinuz
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> On Wed, Jun 12, 2013 at 5:19 PM, Dmitriy Lyubimov <dlieu.7@gmail.com
>>>>>>> > wrote:
>>>>>>>
>>>>>>>> ok thanks.
>>>>>>>>
>>>>>>>> now i switched to cgroups and can't get slave to start. The fatal
>>>>>>>> error says
>>>>>>>>
>>>>>>>> F0612 17:17:04.053773 10059 cgroups_isolation_module.cpp:161]
>>>>>>>> Required subsystem 'freezer' is already in use
>>>>>>>>
>>>>>>>> Any hints appreciated.
>>>>>>>>
>>>>>>>> thank you.
>>>>>>>>
>>>>>>>>
>>>>>>>> On Wed, Jun 12, 2013 at 4:59 PM, Vinod Kone <vi...@gmail.com>wrote:
>>>>>>>>
>>>>>>>>> No problem. Instead of giving --isolation=lxc, you could give
>>>>>>>>> --isolation=cgroups. Also for more flags, start mesos slave with --help.
>>>>>>>>> Unfortunately, we have been a bit behind on the documentation, so the only
>>>>>>>>> place you can look at are our header files (e.g.,
>>>>>>>>> src/slave/cgroups_isolation.hpp). That said, if your kernel supports it
>>>>>>>>> cgroups should work out of the box with mesos.
>>>>>>>>>
>>>>>>>>> HTH,
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> -- Vinod
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Wed, Jun 12, 2013 at 4:52 PM, Dmitriy Lyubimov <
>>>>>>>>> dlieu.7@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Oops. I am just starting with this. I see it clearly not
>>>>>>>>>> working.. I just downloaded 0.11 and trying to set up spark 0.7.2 with it.
>>>>>>>>>> it works ok with "process" isolation. I assumed lxc would be preferrable
>>>>>>>>>> since it is being advertised feature on the Mesos home page.
>>>>>>>>>>
>>>>>>>>>> I will snoop around the docs looking for cgroups isolation. If
>>>>>>>>>> you can point me to manual, i'd be grateful too.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Wed, Jun 12, 2013 at 4:48 PM, Vinod Kone <vi...@gmail.com>wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi Dmitry,
>>>>>>>>>>>
>>>>>>>>>>> What version of mesos are you using? Lxc support has been
>>>>>>>>>>> deprecated for a while now. You should use the new cgroups isolation.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Wed, Jun 12, 2013 at 4:26 PM, Dmitriy Lyubimov <
>>>>>>>>>>> dlieu.7@gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hello,
>>>>>>>>>>>>
>>>>>>>>>>>> is there anything speicific to ubuntu 12 that needs to be done
>>>>>>>>>>>> to make Mesos work with LCX?
>>>>>>>>>>>>
>>>>>>>>>>>> I set things up according to ubuntu docs,
>>>>>>>>>>>> https://help.ubuntu.com/12.10/serverguide/lxc.html#lxc-creation
>>>>>>>>>>>>
>>>>>>>>>>>> and all container examples there seem to be happily working.
>>>>>>>>>>>>
>>>>>>>>>>>> However, some mesos unit tests are failing (which i suspect are
>>>>>>>>>>>> relating to lxc) as well as lxc isolation mode fails to spawn tasks.
>>>>>>>>>>>>
>>>>>>>>>>>> (I am actually on ubuntu 12-04 LTS).
>>>>>>>>>>>>
>>>>>>>>>>>> Is there any speicific way to troubleshoot this? Is LXC in
>>>>>>>>>>>> Mesos even working with Ubuntu 12?
>>>>>>>>>>>>
>>>>>>>>>>>> thank you in advance. (slave output enclosed).
>>>>>>>>>>>> -d
>>>>>>>>>>>>
>>>>>>>>>>>> I0612 16:24:20.682698 26452 slave.cpp:474] Got assigned task 0
>>>>>>>>>>>> for framework 201306121623-16777343-5050-26417-0000
>>>>>>>>>>>> I0612 16:24:20.683425 26452 paths.hpp:234] Created executor
>>>>>>>>>>>> directory
>>>>>>>>>>>> '/tmp/mesos/slaves/201306121623-16777343-5050-26417-0/frameworks/201306121623-16777343-5050-26417-0000/executors/Task
>>>>>>>>>>>> 0 ("/home/dmitr...)/runs/9156d4fa-a177-464b-906f-fb62c8b9b363'
>>>>>>>>>>>> I0612 16:24:20.683630 26453 lxc_isolation_module.cpp:121]
>>>>>>>>>>>> Launching Task 0 ("/home/dmitr...)
>>>>>>>>>>>> (/usr/local/libexec/mesos/mesos-executor) in
>>>>>>>>>>>> /tmp/mesos/slaves/201306121623-16777343-5050-26417-0/frameworks/201306121623-16777343-5050-26417-0000/executors/Task
>>>>>>>>>>>> 0 ("/home/dmitr...)/runs/9156d4fa-a177-464b-906f-fb62c8b9b363 with
>>>>>>>>>>>> resources ' for framework 201306121623-16777343-5050-26417-0000
>>>>>>>>>>>> I0612 16:24:20.683945 26453 lxc_isolation_module.cpp:152]
>>>>>>>>>>>> Forked executor at = 26570
>>>>>>>>>>>> lxc-execute: No such file or directory - failed to create
>>>>>>>>>>>> '/sys/fs/cgroup/cpuset//lxc/mesos_executor_Task 0
>>>>>>>>>>>> ("/home/dmitr...)_framework_201306121623-16777343-5050-26417-0000' directory
>>>>>>>>>>>> lxc-execute: failed to spawn 'mesos_executor_Task 0
>>>>>>>>>>>> ("/home/dmitr...)_framework_201306121623-16777343-5050-26417-0000'
>>>>>>>>>>>> lxc-execute: No such file or directory - failed to remove
>>>>>>>>>>>> cgroup '/sys/fs/cgroup/cpuset//lxc/mesos_executor_Task 0
>>>>>>>>>>>> ("/home/dmitr...)_framework_201306121623-16777343-5050-26417-0000'
>>>>>>>>>>>> I0612 16:24:21.451616 26452 lxc_isolation_module.cpp:322]
>>>>>>>>>>>> Telling slave of lost executor Task 0 ("/home/dmitr...) of framework
>>>>>>>>>>>> 201306121623-16777343-5050-26417-0000
>>>>>>>>>>>> I0612 16:24:21.451709 26452 lxc_isolation_module.cpp:239]
>>>>>>>>>>>> Stopping container mesos_executor_Task 0
>>>>>>>>>>>> ("/home/dmitr...)_framework_201306121623-16777343-5050-26417-0000
>>>>>>>>>>>> I0612 16:24:21.452199 26454 slave.cpp:998] Executor 'Task 0
>>>>>>>>>>>> ("/home/dmitr...)' of framework 201306121623-16777343-5050-26417-0000 has
>>>>>>>>>>>> exited with status 255
>>>>>>>>>>>> sh: 1: Syntax error: "(" unexpected
>>>>>>>>>>>> E0612 16:24:21.453227 26452 lxc_isolation_module.cpp:248]
>>>>>>>>>>>> Failed to stop container mesos_executor_Task 0
>>>>>>>>>>>> ("/home/dmitr...)_framework_201306121623-16777343-5050-26417-0000, lxc-stop
>>>>>>>>>>>> returned: 512
>>>>>>>>>>>> I0612 16:24:21.453385 26454 slave.cpp:829] Status update: task
>>>>>>>>>>>> 0 of framework 201306121623-16777343-5050-26417-0000 is now in state
>>>>>>>>>>>> TASK_FAILED
>>>>>>>>>>>> E0612 16:24:21.453583 26453 lxc_isolation_module.cpp:273]
>>>>>>>>>>>> ERROR! Asked to update resources for an unknown executor!
>>>>>>>>>>>> I0612 16:24:21.453891 26451 gc.cpp:97] Scheduling
>>>>>>>>>>>> /tmp/mesos/slaves/201306121623-16777343-5050-26417-0/frameworks/201306121623-16777343-5050-26417-0000/executors/Task
>>>>>>>>>>>> 0 ("/home/dmitr...)/runs/9156d4fa-a177-464b-906f-fb62c8b9b363 for removal
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Mesos, lxc and ubuntu 12
Posted by Dmitriy Lyubimov <dl...@gmail.com>.
Thank you for your reply.
Now getting this FATAL message :
F0613 11:27:48.010704 24340 cgroups_isolation_module.cpp:147] Cannot create
cgroups hierarchy root at /sys/fs/cgroup. Consider removing it
On Wed, Jun 12, 2013 at 6:06 PM, Benjamin Mahler
<be...@gmail.com>wrote:
> Can you also pass --cgroups_hierarchy_root=/sys/fs/cgroup to the slave?
>
> In 0.11.0, we don't make an attempt to detect the cgroup hierarchy
> location.
>
>
> On Wed, Jun 12, 2013 at 6:01 PM, Dmitriy Lyubimov <dl...@gmail.com>wrote:
>
>> also, listing cgroup produces :
>> dmitriy@BigHP:~$ lscgroup
>> cpu:/
>> cpu:/sysdefault
>> cpuacct:/
>> cpuacct:/sysdefault
>> devices:/
>> devices:/sysdefault
>> memory:/
>> memory:/sysdefault
>> freezer:/
>> freezer:/sysdefault
>>
>>
>>
>> On Wed, Jun 12, 2013 at 6:01 PM, Dmitriy Lyubimov <dl...@gmail.com>wrote:
>>
>>> also, listing cgroups produces
>>>
>>>
>>>
>>> On Wed, Jun 12, 2013 at 5:42 PM, Dmitriy Lyubimov <dl...@gmail.com>wrote:
>>>
>>>> and here's the list of subsystems if it is relevant.
>>>> lssubsys -am
>>>> cpuset
>>>> cpu /sys/fs/cgroup/cpu
>>>> cpuacct /sys/fs/cgroup/cpuacct
>>>> memory /sys/fs/cgroup/memory
>>>> devices /sys/fs/cgroup/devices
>>>> freezer /sys/fs/cgroup/freezer
>>>> blkio
>>>> perf_event
>>>>
>>>>
>>>>
>>>> On Wed, Jun 12, 2013 at 5:34 PM, Dmitriy Lyubimov <dl...@gmail.com>wrote:
>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Wed, Jun 12, 2013 at 5:24 PM, Benjamin Mahler <
>>>>> benjamin.mahler@gmail.com> wrote:
>>>>>
>>>>>> Cgroups does not allow one to mount the same subsystem across cgroup
>>>>>> hierarchies. Do you have multiple cgroup hierarchies present on your
>>>>>> machine?
>>>>>>
>>>>>
>>>>> I frankly know nothing of cgroups. how do i check? not that i know of
>>>>> .
>>>>>
>>>>>>
>>>>>> Ideally this will work with a stock ubuntu 12 OS, but it's possible
>>>>>> that ubuntu already mounts a cgroup hierarchy with the freezer subsystem in
>>>>>> a location we did not expect.
>>>>>>
>>>>>> What are the contents of the root directory on that machine?
>>>>>>
>>>>> bin cdrom etc home initrd.img.old lib64 media
>>>>> opt root sbin srv tmp var vmlinuz.old
>>>>> boot dev hadoop initrd.img lib lost+found mnt
>>>>> proc run selinux sys usr vmlinuz
>>>>>
>>>>>
>>>>>>
>>>>>> On Wed, Jun 12, 2013 at 5:19 PM, Dmitriy Lyubimov <dl...@gmail.com>wrote:
>>>>>>
>>>>>>> ok thanks.
>>>>>>>
>>>>>>> now i switched to cgroups and can't get slave to start. The fatal
>>>>>>> error says
>>>>>>>
>>>>>>> F0612 17:17:04.053773 10059 cgroups_isolation_module.cpp:161]
>>>>>>> Required subsystem 'freezer' is already in use
>>>>>>>
>>>>>>> Any hints appreciated.
>>>>>>>
>>>>>>> thank you.
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Jun 12, 2013 at 4:59 PM, Vinod Kone <vi...@gmail.com>wrote:
>>>>>>>
>>>>>>>> No problem. Instead of giving --isolation=lxc, you could give
>>>>>>>> --isolation=cgroups. Also for more flags, start mesos slave with --help.
>>>>>>>> Unfortunately, we have been a bit behind on the documentation, so the only
>>>>>>>> place you can look at are our header files (e.g.,
>>>>>>>> src/slave/cgroups_isolation.hpp). That said, if your kernel supports it
>>>>>>>> cgroups should work out of the box with mesos.
>>>>>>>>
>>>>>>>> HTH,
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> -- Vinod
>>>>>>>>
>>>>>>>>
>>>>>>>> On Wed, Jun 12, 2013 at 4:52 PM, Dmitriy Lyubimov <
>>>>>>>> dlieu.7@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Oops. I am just starting with this. I see it clearly not working..
>>>>>>>>> I just downloaded 0.11 and trying to set up spark 0.7.2 with it. it works
>>>>>>>>> ok with "process" isolation. I assumed lxc would be preferrable since it is
>>>>>>>>> being advertised feature on the Mesos home page.
>>>>>>>>>
>>>>>>>>> I will snoop around the docs looking for cgroups isolation. If you
>>>>>>>>> can point me to manual, i'd be grateful too.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Wed, Jun 12, 2013 at 4:48 PM, Vinod Kone <vi...@gmail.com>wrote:
>>>>>>>>>
>>>>>>>>>> Hi Dmitry,
>>>>>>>>>>
>>>>>>>>>> What version of mesos are you using? Lxc support has been
>>>>>>>>>> deprecated for a while now. You should use the new cgroups isolation.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Wed, Jun 12, 2013 at 4:26 PM, Dmitriy Lyubimov <
>>>>>>>>>> dlieu.7@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hello,
>>>>>>>>>>>
>>>>>>>>>>> is there anything speicific to ubuntu 12 that needs to be done
>>>>>>>>>>> to make Mesos work with LCX?
>>>>>>>>>>>
>>>>>>>>>>> I set things up according to ubuntu docs,
>>>>>>>>>>> https://help.ubuntu.com/12.10/serverguide/lxc.html#lxc-creation
>>>>>>>>>>>
>>>>>>>>>>> and all container examples there seem to be happily working.
>>>>>>>>>>>
>>>>>>>>>>> However, some mesos unit tests are failing (which i suspect are
>>>>>>>>>>> relating to lxc) as well as lxc isolation mode fails to spawn tasks.
>>>>>>>>>>>
>>>>>>>>>>> (I am actually on ubuntu 12-04 LTS).
>>>>>>>>>>>
>>>>>>>>>>> Is there any speicific way to troubleshoot this? Is LXC in Mesos
>>>>>>>>>>> even working with Ubuntu 12?
>>>>>>>>>>>
>>>>>>>>>>> thank you in advance. (slave output enclosed).
>>>>>>>>>>> -d
>>>>>>>>>>>
>>>>>>>>>>> I0612 16:24:20.682698 26452 slave.cpp:474] Got assigned task 0
>>>>>>>>>>> for framework 201306121623-16777343-5050-26417-0000
>>>>>>>>>>> I0612 16:24:20.683425 26452 paths.hpp:234] Created executor
>>>>>>>>>>> directory
>>>>>>>>>>> '/tmp/mesos/slaves/201306121623-16777343-5050-26417-0/frameworks/201306121623-16777343-5050-26417-0000/executors/Task
>>>>>>>>>>> 0 ("/home/dmitr...)/runs/9156d4fa-a177-464b-906f-fb62c8b9b363'
>>>>>>>>>>> I0612 16:24:20.683630 26453 lxc_isolation_module.cpp:121]
>>>>>>>>>>> Launching Task 0 ("/home/dmitr...)
>>>>>>>>>>> (/usr/local/libexec/mesos/mesos-executor) in
>>>>>>>>>>> /tmp/mesos/slaves/201306121623-16777343-5050-26417-0/frameworks/201306121623-16777343-5050-26417-0000/executors/Task
>>>>>>>>>>> 0 ("/home/dmitr...)/runs/9156d4fa-a177-464b-906f-fb62c8b9b363 with
>>>>>>>>>>> resources ' for framework 201306121623-16777343-5050-26417-0000
>>>>>>>>>>> I0612 16:24:20.683945 26453 lxc_isolation_module.cpp:152] Forked
>>>>>>>>>>> executor at = 26570
>>>>>>>>>>> lxc-execute: No such file or directory - failed to create
>>>>>>>>>>> '/sys/fs/cgroup/cpuset//lxc/mesos_executor_Task 0
>>>>>>>>>>> ("/home/dmitr...)_framework_201306121623-16777343-5050-26417-0000' directory
>>>>>>>>>>> lxc-execute: failed to spawn 'mesos_executor_Task 0
>>>>>>>>>>> ("/home/dmitr...)_framework_201306121623-16777343-5050-26417-0000'
>>>>>>>>>>> lxc-execute: No such file or directory - failed to remove cgroup
>>>>>>>>>>> '/sys/fs/cgroup/cpuset//lxc/mesos_executor_Task 0
>>>>>>>>>>> ("/home/dmitr...)_framework_201306121623-16777343-5050-26417-0000'
>>>>>>>>>>> I0612 16:24:21.451616 26452 lxc_isolation_module.cpp:322]
>>>>>>>>>>> Telling slave of lost executor Task 0 ("/home/dmitr...) of framework
>>>>>>>>>>> 201306121623-16777343-5050-26417-0000
>>>>>>>>>>> I0612 16:24:21.451709 26452 lxc_isolation_module.cpp:239]
>>>>>>>>>>> Stopping container mesos_executor_Task 0
>>>>>>>>>>> ("/home/dmitr...)_framework_201306121623-16777343-5050-26417-0000
>>>>>>>>>>> I0612 16:24:21.452199 26454 slave.cpp:998] Executor 'Task 0
>>>>>>>>>>> ("/home/dmitr...)' of framework 201306121623-16777343-5050-26417-0000 has
>>>>>>>>>>> exited with status 255
>>>>>>>>>>> sh: 1: Syntax error: "(" unexpected
>>>>>>>>>>> E0612 16:24:21.453227 26452 lxc_isolation_module.cpp:248] Failed
>>>>>>>>>>> to stop container mesos_executor_Task 0
>>>>>>>>>>> ("/home/dmitr...)_framework_201306121623-16777343-5050-26417-0000, lxc-stop
>>>>>>>>>>> returned: 512
>>>>>>>>>>> I0612 16:24:21.453385 26454 slave.cpp:829] Status update: task 0
>>>>>>>>>>> of framework 201306121623-16777343-5050-26417-0000 is now in state
>>>>>>>>>>> TASK_FAILED
>>>>>>>>>>> E0612 16:24:21.453583 26453 lxc_isolation_module.cpp:273] ERROR!
>>>>>>>>>>> Asked to update resources for an unknown executor!
>>>>>>>>>>> I0612 16:24:21.453891 26451 gc.cpp:97] Scheduling
>>>>>>>>>>> /tmp/mesos/slaves/201306121623-16777343-5050-26417-0/frameworks/201306121623-16777343-5050-26417-0000/executors/Task
>>>>>>>>>>> 0 ("/home/dmitr...)/runs/9156d4fa-a177-464b-906f-fb62c8b9b363 for removal
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Mesos, lxc and ubuntu 12
Posted by Benjamin Mahler <be...@gmail.com>.
Can you also pass --cgroups_hierarchy_root=/sys/fs/cgroup to the slave?
In 0.11.0, we don't make an attempt to detect the cgroup hierarchy location.
On Wed, Jun 12, 2013 at 6:01 PM, Dmitriy Lyubimov <dl...@gmail.com> wrote:
> also, listing cgroup produces :
> dmitriy@BigHP:~$ lscgroup
> cpu:/
> cpu:/sysdefault
> cpuacct:/
> cpuacct:/sysdefault
> devices:/
> devices:/sysdefault
> memory:/
> memory:/sysdefault
> freezer:/
> freezer:/sysdefault
>
>
>
> On Wed, Jun 12, 2013 at 6:01 PM, Dmitriy Lyubimov <dl...@gmail.com>wrote:
>
>> also, listing cgroups produces
>>
>>
>>
>> On Wed, Jun 12, 2013 at 5:42 PM, Dmitriy Lyubimov <dl...@gmail.com>wrote:
>>
>>> and here's the list of subsystems if it is relevant.
>>> lssubsys -am
>>> cpuset
>>> cpu /sys/fs/cgroup/cpu
>>> cpuacct /sys/fs/cgroup/cpuacct
>>> memory /sys/fs/cgroup/memory
>>> devices /sys/fs/cgroup/devices
>>> freezer /sys/fs/cgroup/freezer
>>> blkio
>>> perf_event
>>>
>>>
>>>
>>> On Wed, Jun 12, 2013 at 5:34 PM, Dmitriy Lyubimov <dl...@gmail.com>wrote:
>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Wed, Jun 12, 2013 at 5:24 PM, Benjamin Mahler <
>>>> benjamin.mahler@gmail.com> wrote:
>>>>
>>>>> Cgroups does not allow one to mount the same subsystem across cgroup
>>>>> hierarchies. Do you have multiple cgroup hierarchies present on your
>>>>> machine?
>>>>>
>>>>
>>>> I frankly know nothing of cgroups. how do i check? not that i know of .
>>>>
>>>>>
>>>>> Ideally this will work with a stock ubuntu 12 OS, but it's possible
>>>>> that ubuntu already mounts a cgroup hierarchy with the freezer subsystem in
>>>>> a location we did not expect.
>>>>>
>>>>> What are the contents of the root directory on that machine?
>>>>>
>>>> bin cdrom etc home initrd.img.old lib64 media
>>>> opt root sbin srv tmp var vmlinuz.old
>>>> boot dev hadoop initrd.img lib lost+found mnt
>>>> proc run selinux sys usr vmlinuz
>>>>
>>>>
>>>>>
>>>>> On Wed, Jun 12, 2013 at 5:19 PM, Dmitriy Lyubimov <dl...@gmail.com>wrote:
>>>>>
>>>>>> ok thanks.
>>>>>>
>>>>>> now i switched to cgroups and can't get slave to start. The fatal
>>>>>> error says
>>>>>>
>>>>>> F0612 17:17:04.053773 10059 cgroups_isolation_module.cpp:161]
>>>>>> Required subsystem 'freezer' is already in use
>>>>>>
>>>>>> Any hints appreciated.
>>>>>>
>>>>>> thank you.
>>>>>>
>>>>>>
>>>>>> On Wed, Jun 12, 2013 at 4:59 PM, Vinod Kone <vi...@gmail.com>wrote:
>>>>>>
>>>>>>> No problem. Instead of giving --isolation=lxc, you could give
>>>>>>> --isolation=cgroups. Also for more flags, start mesos slave with --help.
>>>>>>> Unfortunately, we have been a bit behind on the documentation, so the only
>>>>>>> place you can look at are our header files (e.g.,
>>>>>>> src/slave/cgroups_isolation.hpp). That said, if your kernel supports it
>>>>>>> cgroups should work out of the box with mesos.
>>>>>>>
>>>>>>> HTH,
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> -- Vinod
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Jun 12, 2013 at 4:52 PM, Dmitriy Lyubimov <dlieu.7@gmail.com
>>>>>>> > wrote:
>>>>>>>
>>>>>>>> Oops. I am just starting with this. I see it clearly not working..
>>>>>>>> I just downloaded 0.11 and trying to set up spark 0.7.2 with it. it works
>>>>>>>> ok with "process" isolation. I assumed lxc would be preferrable since it is
>>>>>>>> being advertised feature on the Mesos home page.
>>>>>>>>
>>>>>>>> I will snoop around the docs looking for cgroups isolation. If you
>>>>>>>> can point me to manual, i'd be grateful too.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Wed, Jun 12, 2013 at 4:48 PM, Vinod Kone <vi...@gmail.com>wrote:
>>>>>>>>
>>>>>>>>> Hi Dmitry,
>>>>>>>>>
>>>>>>>>> What version of mesos are you using? Lxc support has been
>>>>>>>>> deprecated for a while now. You should use the new cgroups isolation.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Wed, Jun 12, 2013 at 4:26 PM, Dmitriy Lyubimov <
>>>>>>>>> dlieu.7@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Hello,
>>>>>>>>>>
>>>>>>>>>> is there anything speicific to ubuntu 12 that needs to be done to
>>>>>>>>>> make Mesos work with LCX?
>>>>>>>>>>
>>>>>>>>>> I set things up according to ubuntu docs,
>>>>>>>>>> https://help.ubuntu.com/12.10/serverguide/lxc.html#lxc-creation
>>>>>>>>>>
>>>>>>>>>> and all container examples there seem to be happily working.
>>>>>>>>>>
>>>>>>>>>> However, some mesos unit tests are failing (which i suspect are
>>>>>>>>>> relating to lxc) as well as lxc isolation mode fails to spawn tasks.
>>>>>>>>>>
>>>>>>>>>> (I am actually on ubuntu 12-04 LTS).
>>>>>>>>>>
>>>>>>>>>> Is there any speicific way to troubleshoot this? Is LXC in Mesos
>>>>>>>>>> even working with Ubuntu 12?
>>>>>>>>>>
>>>>>>>>>> thank you in advance. (slave output enclosed).
>>>>>>>>>> -d
>>>>>>>>>>
>>>>>>>>>> I0612 16:24:20.682698 26452 slave.cpp:474] Got assigned task 0
>>>>>>>>>> for framework 201306121623-16777343-5050-26417-0000
>>>>>>>>>> I0612 16:24:20.683425 26452 paths.hpp:234] Created executor
>>>>>>>>>> directory
>>>>>>>>>> '/tmp/mesos/slaves/201306121623-16777343-5050-26417-0/frameworks/201306121623-16777343-5050-26417-0000/executors/Task
>>>>>>>>>> 0 ("/home/dmitr...)/runs/9156d4fa-a177-464b-906f-fb62c8b9b363'
>>>>>>>>>> I0612 16:24:20.683630 26453 lxc_isolation_module.cpp:121]
>>>>>>>>>> Launching Task 0 ("/home/dmitr...)
>>>>>>>>>> (/usr/local/libexec/mesos/mesos-executor) in
>>>>>>>>>> /tmp/mesos/slaves/201306121623-16777343-5050-26417-0/frameworks/201306121623-16777343-5050-26417-0000/executors/Task
>>>>>>>>>> 0 ("/home/dmitr...)/runs/9156d4fa-a177-464b-906f-fb62c8b9b363 with
>>>>>>>>>> resources ' for framework 201306121623-16777343-5050-26417-0000
>>>>>>>>>> I0612 16:24:20.683945 26453 lxc_isolation_module.cpp:152] Forked
>>>>>>>>>> executor at = 26570
>>>>>>>>>> lxc-execute: No such file or directory - failed to create
>>>>>>>>>> '/sys/fs/cgroup/cpuset//lxc/mesos_executor_Task 0
>>>>>>>>>> ("/home/dmitr...)_framework_201306121623-16777343-5050-26417-0000' directory
>>>>>>>>>> lxc-execute: failed to spawn 'mesos_executor_Task 0
>>>>>>>>>> ("/home/dmitr...)_framework_201306121623-16777343-5050-26417-0000'
>>>>>>>>>> lxc-execute: No such file or directory - failed to remove cgroup
>>>>>>>>>> '/sys/fs/cgroup/cpuset//lxc/mesos_executor_Task 0
>>>>>>>>>> ("/home/dmitr...)_framework_201306121623-16777343-5050-26417-0000'
>>>>>>>>>> I0612 16:24:21.451616 26452 lxc_isolation_module.cpp:322] Telling
>>>>>>>>>> slave of lost executor Task 0 ("/home/dmitr...) of framework
>>>>>>>>>> 201306121623-16777343-5050-26417-0000
>>>>>>>>>> I0612 16:24:21.451709 26452 lxc_isolation_module.cpp:239]
>>>>>>>>>> Stopping container mesos_executor_Task 0
>>>>>>>>>> ("/home/dmitr...)_framework_201306121623-16777343-5050-26417-0000
>>>>>>>>>> I0612 16:24:21.452199 26454 slave.cpp:998] Executor 'Task 0
>>>>>>>>>> ("/home/dmitr...)' of framework 201306121623-16777343-5050-26417-0000 has
>>>>>>>>>> exited with status 255
>>>>>>>>>> sh: 1: Syntax error: "(" unexpected
>>>>>>>>>> E0612 16:24:21.453227 26452 lxc_isolation_module.cpp:248] Failed
>>>>>>>>>> to stop container mesos_executor_Task 0
>>>>>>>>>> ("/home/dmitr...)_framework_201306121623-16777343-5050-26417-0000, lxc-stop
>>>>>>>>>> returned: 512
>>>>>>>>>> I0612 16:24:21.453385 26454 slave.cpp:829] Status update: task 0
>>>>>>>>>> of framework 201306121623-16777343-5050-26417-0000 is now in state
>>>>>>>>>> TASK_FAILED
>>>>>>>>>> E0612 16:24:21.453583 26453 lxc_isolation_module.cpp:273] ERROR!
>>>>>>>>>> Asked to update resources for an unknown executor!
>>>>>>>>>> I0612 16:24:21.453891 26451 gc.cpp:97] Scheduling
>>>>>>>>>> /tmp/mesos/slaves/201306121623-16777343-5050-26417-0/frameworks/201306121623-16777343-5050-26417-0000/executors/Task
>>>>>>>>>> 0 ("/home/dmitr...)/runs/9156d4fa-a177-464b-906f-fb62c8b9b363 for removal
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Mesos, lxc and ubuntu 12
Posted by Dmitriy Lyubimov <dl...@gmail.com>.
also, listing cgroup produces :
dmitriy@BigHP:~$ lscgroup
cpu:/
cpu:/sysdefault
cpuacct:/
cpuacct:/sysdefault
devices:/
devices:/sysdefault
memory:/
memory:/sysdefault
freezer:/
freezer:/sysdefault
On Wed, Jun 12, 2013 at 6:01 PM, Dmitriy Lyubimov <dl...@gmail.com> wrote:
> also, listing cgroups produces
>
>
>
> On Wed, Jun 12, 2013 at 5:42 PM, Dmitriy Lyubimov <dl...@gmail.com>wrote:
>
>> and here's the list of subsystems if it is relevant.
>> lssubsys -am
>> cpuset
>> cpu /sys/fs/cgroup/cpu
>> cpuacct /sys/fs/cgroup/cpuacct
>> memory /sys/fs/cgroup/memory
>> devices /sys/fs/cgroup/devices
>> freezer /sys/fs/cgroup/freezer
>> blkio
>> perf_event
>>
>>
>>
>> On Wed, Jun 12, 2013 at 5:34 PM, Dmitriy Lyubimov <dl...@gmail.com>wrote:
>>
>>>
>>>
>>>
>>>
>>>
>>> On Wed, Jun 12, 2013 at 5:24 PM, Benjamin Mahler <
>>> benjamin.mahler@gmail.com> wrote:
>>>
>>>> Cgroups does not allow one to mount the same subsystem across cgroup
>>>> hierarchies. Do you have multiple cgroup hierarchies present on your
>>>> machine?
>>>>
>>>
>>> I frankly know nothing of cgroups. how do i check? not that i know of .
>>>
>>>>
>>>> Ideally this will work with a stock ubuntu 12 OS, but it's possible
>>>> that ubuntu already mounts a cgroup hierarchy with the freezer subsystem in
>>>> a location we did not expect.
>>>>
>>>> What are the contents of the root directory on that machine?
>>>>
>>> bin cdrom etc home initrd.img.old lib64 media opt
>>> root sbin srv tmp var vmlinuz.old
>>> boot dev hadoop initrd.img lib lost+found mnt proc
>>> run selinux sys usr vmlinuz
>>>
>>>
>>>>
>>>> On Wed, Jun 12, 2013 at 5:19 PM, Dmitriy Lyubimov <dl...@gmail.com>wrote:
>>>>
>>>>> ok thanks.
>>>>>
>>>>> now i switched to cgroups and can't get slave to start. The fatal
>>>>> error says
>>>>>
>>>>> F0612 17:17:04.053773 10059 cgroups_isolation_module.cpp:161] Required
>>>>> subsystem 'freezer' is already in use
>>>>>
>>>>> Any hints appreciated.
>>>>>
>>>>> thank you.
>>>>>
>>>>>
>>>>> On Wed, Jun 12, 2013 at 4:59 PM, Vinod Kone <vi...@gmail.com>wrote:
>>>>>
>>>>>> No problem. Instead of giving --isolation=lxc, you could give
>>>>>> --isolation=cgroups. Also for more flags, start mesos slave with --help.
>>>>>> Unfortunately, we have been a bit behind on the documentation, so the only
>>>>>> place you can look at are our header files (e.g.,
>>>>>> src/slave/cgroups_isolation.hpp). That said, if your kernel supports it
>>>>>> cgroups should work out of the box with mesos.
>>>>>>
>>>>>> HTH,
>>>>>>
>>>>>>
>>>>>>
>>>>>> -- Vinod
>>>>>>
>>>>>>
>>>>>> On Wed, Jun 12, 2013 at 4:52 PM, Dmitriy Lyubimov <dl...@gmail.com>wrote:
>>>>>>
>>>>>>> Oops. I am just starting with this. I see it clearly not working..
>>>>>>> I just downloaded 0.11 and trying to set up spark 0.7.2 with it. it works
>>>>>>> ok with "process" isolation. I assumed lxc would be preferrable since it is
>>>>>>> being advertised feature on the Mesos home page.
>>>>>>>
>>>>>>> I will snoop around the docs looking for cgroups isolation. If you
>>>>>>> can point me to manual, i'd be grateful too.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Jun 12, 2013 at 4:48 PM, Vinod Kone <vi...@gmail.com>wrote:
>>>>>>>
>>>>>>>> Hi Dmitry,
>>>>>>>>
>>>>>>>> What version of mesos are you using? Lxc support has been
>>>>>>>> deprecated for a while now. You should use the new cgroups isolation.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Wed, Jun 12, 2013 at 4:26 PM, Dmitriy Lyubimov <
>>>>>>>> dlieu.7@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Hello,
>>>>>>>>>
>>>>>>>>> is there anything speicific to ubuntu 12 that needs to be done to
>>>>>>>>> make Mesos work with LCX?
>>>>>>>>>
>>>>>>>>> I set things up according to ubuntu docs,
>>>>>>>>> https://help.ubuntu.com/12.10/serverguide/lxc.html#lxc-creation
>>>>>>>>>
>>>>>>>>> and all container examples there seem to be happily working.
>>>>>>>>>
>>>>>>>>> However, some mesos unit tests are failing (which i suspect are
>>>>>>>>> relating to lxc) as well as lxc isolation mode fails to spawn tasks.
>>>>>>>>>
>>>>>>>>> (I am actually on ubuntu 12-04 LTS).
>>>>>>>>>
>>>>>>>>> Is there any speicific way to troubleshoot this? Is LXC in Mesos
>>>>>>>>> even working with Ubuntu 12?
>>>>>>>>>
>>>>>>>>> thank you in advance. (slave output enclosed).
>>>>>>>>> -d
>>>>>>>>>
>>>>>>>>> I0612 16:24:20.682698 26452 slave.cpp:474] Got assigned task 0 for
>>>>>>>>> framework 201306121623-16777343-5050-26417-0000
>>>>>>>>> I0612 16:24:20.683425 26452 paths.hpp:234] Created executor
>>>>>>>>> directory
>>>>>>>>> '/tmp/mesos/slaves/201306121623-16777343-5050-26417-0/frameworks/201306121623-16777343-5050-26417-0000/executors/Task
>>>>>>>>> 0 ("/home/dmitr...)/runs/9156d4fa-a177-464b-906f-fb62c8b9b363'
>>>>>>>>> I0612 16:24:20.683630 26453 lxc_isolation_module.cpp:121]
>>>>>>>>> Launching Task 0 ("/home/dmitr...)
>>>>>>>>> (/usr/local/libexec/mesos/mesos-executor) in
>>>>>>>>> /tmp/mesos/slaves/201306121623-16777343-5050-26417-0/frameworks/201306121623-16777343-5050-26417-0000/executors/Task
>>>>>>>>> 0 ("/home/dmitr...)/runs/9156d4fa-a177-464b-906f-fb62c8b9b363 with
>>>>>>>>> resources ' for framework 201306121623-16777343-5050-26417-0000
>>>>>>>>> I0612 16:24:20.683945 26453 lxc_isolation_module.cpp:152] Forked
>>>>>>>>> executor at = 26570
>>>>>>>>> lxc-execute: No such file or directory - failed to create
>>>>>>>>> '/sys/fs/cgroup/cpuset//lxc/mesos_executor_Task 0
>>>>>>>>> ("/home/dmitr...)_framework_201306121623-16777343-5050-26417-0000' directory
>>>>>>>>> lxc-execute: failed to spawn 'mesos_executor_Task 0
>>>>>>>>> ("/home/dmitr...)_framework_201306121623-16777343-5050-26417-0000'
>>>>>>>>> lxc-execute: No such file or directory - failed to remove cgroup
>>>>>>>>> '/sys/fs/cgroup/cpuset//lxc/mesos_executor_Task 0
>>>>>>>>> ("/home/dmitr...)_framework_201306121623-16777343-5050-26417-0000'
>>>>>>>>> I0612 16:24:21.451616 26452 lxc_isolation_module.cpp:322] Telling
>>>>>>>>> slave of lost executor Task 0 ("/home/dmitr...) of framework
>>>>>>>>> 201306121623-16777343-5050-26417-0000
>>>>>>>>> I0612 16:24:21.451709 26452 lxc_isolation_module.cpp:239] Stopping
>>>>>>>>> container mesos_executor_Task 0
>>>>>>>>> ("/home/dmitr...)_framework_201306121623-16777343-5050-26417-0000
>>>>>>>>> I0612 16:24:21.452199 26454 slave.cpp:998] Executor 'Task 0
>>>>>>>>> ("/home/dmitr...)' of framework 201306121623-16777343-5050-26417-0000 has
>>>>>>>>> exited with status 255
>>>>>>>>> sh: 1: Syntax error: "(" unexpected
>>>>>>>>> E0612 16:24:21.453227 26452 lxc_isolation_module.cpp:248] Failed
>>>>>>>>> to stop container mesos_executor_Task 0
>>>>>>>>> ("/home/dmitr...)_framework_201306121623-16777343-5050-26417-0000, lxc-stop
>>>>>>>>> returned: 512
>>>>>>>>> I0612 16:24:21.453385 26454 slave.cpp:829] Status update: task 0
>>>>>>>>> of framework 201306121623-16777343-5050-26417-0000 is now in state
>>>>>>>>> TASK_FAILED
>>>>>>>>> E0612 16:24:21.453583 26453 lxc_isolation_module.cpp:273] ERROR!
>>>>>>>>> Asked to update resources for an unknown executor!
>>>>>>>>> I0612 16:24:21.453891 26451 gc.cpp:97] Scheduling
>>>>>>>>> /tmp/mesos/slaves/201306121623-16777343-5050-26417-0/frameworks/201306121623-16777343-5050-26417-0000/executors/Task
>>>>>>>>> 0 ("/home/dmitr...)/runs/9156d4fa-a177-464b-906f-fb62c8b9b363 for removal
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Mesos, lxc and ubuntu 12
Posted by Dmitriy Lyubimov <dl...@gmail.com>.
also, listing cgroups produces
On Wed, Jun 12, 2013 at 5:42 PM, Dmitriy Lyubimov <dl...@gmail.com> wrote:
> and here's the list of subsystems if it is relevant.
> lssubsys -am
> cpuset
> cpu /sys/fs/cgroup/cpu
> cpuacct /sys/fs/cgroup/cpuacct
> memory /sys/fs/cgroup/memory
> devices /sys/fs/cgroup/devices
> freezer /sys/fs/cgroup/freezer
> blkio
> perf_event
>
>
>
> On Wed, Jun 12, 2013 at 5:34 PM, Dmitriy Lyubimov <dl...@gmail.com>wrote:
>
>>
>>
>>
>>
>>
>> On Wed, Jun 12, 2013 at 5:24 PM, Benjamin Mahler <
>> benjamin.mahler@gmail.com> wrote:
>>
>>> Cgroups does not allow one to mount the same subsystem across cgroup
>>> hierarchies. Do you have multiple cgroup hierarchies present on your
>>> machine?
>>>
>>
>> I frankly know nothing of cgroups. how do i check? not that i know of .
>>
>>>
>>> Ideally this will work with a stock ubuntu 12 OS, but it's possible that
>>> ubuntu already mounts a cgroup hierarchy with the freezer subsystem in a
>>> location we did not expect.
>>>
>>> What are the contents of the root directory on that machine?
>>>
>> bin cdrom etc home initrd.img.old lib64 media opt
>> root sbin srv tmp var vmlinuz.old
>> boot dev hadoop initrd.img lib lost+found mnt proc
>> run selinux sys usr vmlinuz
>>
>>
>>>
>>> On Wed, Jun 12, 2013 at 5:19 PM, Dmitriy Lyubimov <dl...@gmail.com>wrote:
>>>
>>>> ok thanks.
>>>>
>>>> now i switched to cgroups and can't get slave to start. The fatal error
>>>> says
>>>>
>>>> F0612 17:17:04.053773 10059 cgroups_isolation_module.cpp:161] Required
>>>> subsystem 'freezer' is already in use
>>>>
>>>> Any hints appreciated.
>>>>
>>>> thank you.
>>>>
>>>>
>>>> On Wed, Jun 12, 2013 at 4:59 PM, Vinod Kone <vi...@gmail.com>wrote:
>>>>
>>>>> No problem. Instead of giving --isolation=lxc, you could give
>>>>> --isolation=cgroups. Also for more flags, start mesos slave with --help.
>>>>> Unfortunately, we have been a bit behind on the documentation, so the only
>>>>> place you can look at are our header files (e.g.,
>>>>> src/slave/cgroups_isolation.hpp). That said, if your kernel supports it
>>>>> cgroups should work out of the box with mesos.
>>>>>
>>>>> HTH,
>>>>>
>>>>>
>>>>>
>>>>> -- Vinod
>>>>>
>>>>>
>>>>> On Wed, Jun 12, 2013 at 4:52 PM, Dmitriy Lyubimov <dl...@gmail.com>wrote:
>>>>>
>>>>>> Oops. I am just starting with this. I see it clearly not working.. I
>>>>>> just downloaded 0.11 and trying to set up spark 0.7.2 with it. it works ok
>>>>>> with "process" isolation. I assumed lxc would be preferrable since it is
>>>>>> being advertised feature on the Mesos home page.
>>>>>>
>>>>>> I will snoop around the docs looking for cgroups isolation. If you
>>>>>> can point me to manual, i'd be grateful too.
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Wed, Jun 12, 2013 at 4:48 PM, Vinod Kone <vi...@gmail.com>wrote:
>>>>>>
>>>>>>> Hi Dmitry,
>>>>>>>
>>>>>>> What version of mesos are you using? Lxc support has been deprecated
>>>>>>> for a while now. You should use the new cgroups isolation.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Jun 12, 2013 at 4:26 PM, Dmitriy Lyubimov <dlieu.7@gmail.com
>>>>>>> > wrote:
>>>>>>>
>>>>>>>> Hello,
>>>>>>>>
>>>>>>>> is there anything speicific to ubuntu 12 that needs to be done to
>>>>>>>> make Mesos work with LCX?
>>>>>>>>
>>>>>>>> I set things up according to ubuntu docs,
>>>>>>>> https://help.ubuntu.com/12.10/serverguide/lxc.html#lxc-creation
>>>>>>>>
>>>>>>>> and all container examples there seem to be happily working.
>>>>>>>>
>>>>>>>> However, some mesos unit tests are failing (which i suspect are
>>>>>>>> relating to lxc) as well as lxc isolation mode fails to spawn tasks.
>>>>>>>>
>>>>>>>> (I am actually on ubuntu 12-04 LTS).
>>>>>>>>
>>>>>>>> Is there any speicific way to troubleshoot this? Is LXC in Mesos
>>>>>>>> even working with Ubuntu 12?
>>>>>>>>
>>>>>>>> thank you in advance. (slave output enclosed).
>>>>>>>> -d
>>>>>>>>
>>>>>>>> I0612 16:24:20.682698 26452 slave.cpp:474] Got assigned task 0 for
>>>>>>>> framework 201306121623-16777343-5050-26417-0000
>>>>>>>> I0612 16:24:20.683425 26452 paths.hpp:234] Created executor
>>>>>>>> directory
>>>>>>>> '/tmp/mesos/slaves/201306121623-16777343-5050-26417-0/frameworks/201306121623-16777343-5050-26417-0000/executors/Task
>>>>>>>> 0 ("/home/dmitr...)/runs/9156d4fa-a177-464b-906f-fb62c8b9b363'
>>>>>>>> I0612 16:24:20.683630 26453 lxc_isolation_module.cpp:121] Launching
>>>>>>>> Task 0 ("/home/dmitr...) (/usr/local/libexec/mesos/mesos-executor) in
>>>>>>>> /tmp/mesos/slaves/201306121623-16777343-5050-26417-0/frameworks/201306121623-16777343-5050-26417-0000/executors/Task
>>>>>>>> 0 ("/home/dmitr...)/runs/9156d4fa-a177-464b-906f-fb62c8b9b363 with
>>>>>>>> resources ' for framework 201306121623-16777343-5050-26417-0000
>>>>>>>> I0612 16:24:20.683945 26453 lxc_isolation_module.cpp:152] Forked
>>>>>>>> executor at = 26570
>>>>>>>> lxc-execute: No such file or directory - failed to create
>>>>>>>> '/sys/fs/cgroup/cpuset//lxc/mesos_executor_Task 0
>>>>>>>> ("/home/dmitr...)_framework_201306121623-16777343-5050-26417-0000' directory
>>>>>>>> lxc-execute: failed to spawn 'mesos_executor_Task 0
>>>>>>>> ("/home/dmitr...)_framework_201306121623-16777343-5050-26417-0000'
>>>>>>>> lxc-execute: No such file or directory - failed to remove cgroup
>>>>>>>> '/sys/fs/cgroup/cpuset//lxc/mesos_executor_Task 0
>>>>>>>> ("/home/dmitr...)_framework_201306121623-16777343-5050-26417-0000'
>>>>>>>> I0612 16:24:21.451616 26452 lxc_isolation_module.cpp:322] Telling
>>>>>>>> slave of lost executor Task 0 ("/home/dmitr...) of framework
>>>>>>>> 201306121623-16777343-5050-26417-0000
>>>>>>>> I0612 16:24:21.451709 26452 lxc_isolation_module.cpp:239] Stopping
>>>>>>>> container mesos_executor_Task 0
>>>>>>>> ("/home/dmitr...)_framework_201306121623-16777343-5050-26417-0000
>>>>>>>> I0612 16:24:21.452199 26454 slave.cpp:998] Executor 'Task 0
>>>>>>>> ("/home/dmitr...)' of framework 201306121623-16777343-5050-26417-0000 has
>>>>>>>> exited with status 255
>>>>>>>> sh: 1: Syntax error: "(" unexpected
>>>>>>>> E0612 16:24:21.453227 26452 lxc_isolation_module.cpp:248] Failed to
>>>>>>>> stop container mesos_executor_Task 0
>>>>>>>> ("/home/dmitr...)_framework_201306121623-16777343-5050-26417-0000, lxc-stop
>>>>>>>> returned: 512
>>>>>>>> I0612 16:24:21.453385 26454 slave.cpp:829] Status update: task 0 of
>>>>>>>> framework 201306121623-16777343-5050-26417-0000 is now in state TASK_FAILED
>>>>>>>> E0612 16:24:21.453583 26453 lxc_isolation_module.cpp:273] ERROR!
>>>>>>>> Asked to update resources for an unknown executor!
>>>>>>>> I0612 16:24:21.453891 26451 gc.cpp:97] Scheduling
>>>>>>>> /tmp/mesos/slaves/201306121623-16777343-5050-26417-0/frameworks/201306121623-16777343-5050-26417-0000/executors/Task
>>>>>>>> 0 ("/home/dmitr...)/runs/9156d4fa-a177-464b-906f-fb62c8b9b363 for removal
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Mesos, lxc and ubuntu 12
Posted by Dmitriy Lyubimov <dl...@gmail.com>.
and here's the list of subsystems if it is relevant.
lssubsys -am
cpuset
cpu /sys/fs/cgroup/cpu
cpuacct /sys/fs/cgroup/cpuacct
memory /sys/fs/cgroup/memory
devices /sys/fs/cgroup/devices
freezer /sys/fs/cgroup/freezer
blkio
perf_event
On Wed, Jun 12, 2013 at 5:34 PM, Dmitriy Lyubimov <dl...@gmail.com> wrote:
>
>
>
>
>
> On Wed, Jun 12, 2013 at 5:24 PM, Benjamin Mahler <
> benjamin.mahler@gmail.com> wrote:
>
>> Cgroups does not allow one to mount the same subsystem across cgroup
>> hierarchies. Do you have multiple cgroup hierarchies present on your
>> machine?
>>
>
> I frankly know nothing of cgroups. how do i check? not that i know of .
>
>>
>> Ideally this will work with a stock ubuntu 12 OS, but it's possible that
>> ubuntu already mounts a cgroup hierarchy with the freezer subsystem in a
>> location we did not expect.
>>
>> What are the contents of the root directory on that machine?
>>
> bin cdrom etc home initrd.img.old lib64 media opt
> root sbin srv tmp var vmlinuz.old
> boot dev hadoop initrd.img lib lost+found mnt proc
> run selinux sys usr vmlinuz
>
>
>>
>> On Wed, Jun 12, 2013 at 5:19 PM, Dmitriy Lyubimov <dl...@gmail.com>wrote:
>>
>>> ok thanks.
>>>
>>> now i switched to cgroups and can't get slave to start. The fatal error
>>> says
>>>
>>> F0612 17:17:04.053773 10059 cgroups_isolation_module.cpp:161] Required
>>> subsystem 'freezer' is already in use
>>>
>>> Any hints appreciated.
>>>
>>> thank you.
>>>
>>>
>>> On Wed, Jun 12, 2013 at 4:59 PM, Vinod Kone <vi...@gmail.com> wrote:
>>>
>>>> No problem. Instead of giving --isolation=lxc, you could give
>>>> --isolation=cgroups. Also for more flags, start mesos slave with --help.
>>>> Unfortunately, we have been a bit behind on the documentation, so the only
>>>> place you can look at are our header files (e.g.,
>>>> src/slave/cgroups_isolation.hpp). That said, if your kernel supports it
>>>> cgroups should work out of the box with mesos.
>>>>
>>>> HTH,
>>>>
>>>>
>>>>
>>>> -- Vinod
>>>>
>>>>
>>>> On Wed, Jun 12, 2013 at 4:52 PM, Dmitriy Lyubimov <dl...@gmail.com>wrote:
>>>>
>>>>> Oops. I am just starting with this. I see it clearly not working.. I
>>>>> just downloaded 0.11 and trying to set up spark 0.7.2 with it. it works ok
>>>>> with "process" isolation. I assumed lxc would be preferrable since it is
>>>>> being advertised feature on the Mesos home page.
>>>>>
>>>>> I will snoop around the docs looking for cgroups isolation. If you can
>>>>> point me to manual, i'd be grateful too.
>>>>>
>>>>>
>>>>>
>>>>> On Wed, Jun 12, 2013 at 4:48 PM, Vinod Kone <vi...@gmail.com>wrote:
>>>>>
>>>>>> Hi Dmitry,
>>>>>>
>>>>>> What version of mesos are you using? Lxc support has been deprecated
>>>>>> for a while now. You should use the new cgroups isolation.
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Wed, Jun 12, 2013 at 4:26 PM, Dmitriy Lyubimov <dl...@gmail.com>wrote:
>>>>>>
>>>>>>> Hello,
>>>>>>>
>>>>>>> is there anything speicific to ubuntu 12 that needs to be done to
>>>>>>> make Mesos work with LCX?
>>>>>>>
>>>>>>> I set things up according to ubuntu docs,
>>>>>>> https://help.ubuntu.com/12.10/serverguide/lxc.html#lxc-creation
>>>>>>>
>>>>>>> and all container examples there seem to be happily working.
>>>>>>>
>>>>>>> However, some mesos unit tests are failing (which i suspect are
>>>>>>> relating to lxc) as well as lxc isolation mode fails to spawn tasks.
>>>>>>>
>>>>>>> (I am actually on ubuntu 12-04 LTS).
>>>>>>>
>>>>>>> Is there any speicific way to troubleshoot this? Is LXC in Mesos
>>>>>>> even working with Ubuntu 12?
>>>>>>>
>>>>>>> thank you in advance. (slave output enclosed).
>>>>>>> -d
>>>>>>>
>>>>>>> I0612 16:24:20.682698 26452 slave.cpp:474] Got assigned task 0 for
>>>>>>> framework 201306121623-16777343-5050-26417-0000
>>>>>>> I0612 16:24:20.683425 26452 paths.hpp:234] Created executor
>>>>>>> directory
>>>>>>> '/tmp/mesos/slaves/201306121623-16777343-5050-26417-0/frameworks/201306121623-16777343-5050-26417-0000/executors/Task
>>>>>>> 0 ("/home/dmitr...)/runs/9156d4fa-a177-464b-906f-fb62c8b9b363'
>>>>>>> I0612 16:24:20.683630 26453 lxc_isolation_module.cpp:121] Launching
>>>>>>> Task 0 ("/home/dmitr...) (/usr/local/libexec/mesos/mesos-executor) in
>>>>>>> /tmp/mesos/slaves/201306121623-16777343-5050-26417-0/frameworks/201306121623-16777343-5050-26417-0000/executors/Task
>>>>>>> 0 ("/home/dmitr...)/runs/9156d4fa-a177-464b-906f-fb62c8b9b363 with
>>>>>>> resources ' for framework 201306121623-16777343-5050-26417-0000
>>>>>>> I0612 16:24:20.683945 26453 lxc_isolation_module.cpp:152] Forked
>>>>>>> executor at = 26570
>>>>>>> lxc-execute: No such file or directory - failed to create
>>>>>>> '/sys/fs/cgroup/cpuset//lxc/mesos_executor_Task 0
>>>>>>> ("/home/dmitr...)_framework_201306121623-16777343-5050-26417-0000' directory
>>>>>>> lxc-execute: failed to spawn 'mesos_executor_Task 0
>>>>>>> ("/home/dmitr...)_framework_201306121623-16777343-5050-26417-0000'
>>>>>>> lxc-execute: No such file or directory - failed to remove cgroup
>>>>>>> '/sys/fs/cgroup/cpuset//lxc/mesos_executor_Task 0
>>>>>>> ("/home/dmitr...)_framework_201306121623-16777343-5050-26417-0000'
>>>>>>> I0612 16:24:21.451616 26452 lxc_isolation_module.cpp:322] Telling
>>>>>>> slave of lost executor Task 0 ("/home/dmitr...) of framework
>>>>>>> 201306121623-16777343-5050-26417-0000
>>>>>>> I0612 16:24:21.451709 26452 lxc_isolation_module.cpp:239] Stopping
>>>>>>> container mesos_executor_Task 0
>>>>>>> ("/home/dmitr...)_framework_201306121623-16777343-5050-26417-0000
>>>>>>> I0612 16:24:21.452199 26454 slave.cpp:998] Executor 'Task 0
>>>>>>> ("/home/dmitr...)' of framework 201306121623-16777343-5050-26417-0000 has
>>>>>>> exited with status 255
>>>>>>> sh: 1: Syntax error: "(" unexpected
>>>>>>> E0612 16:24:21.453227 26452 lxc_isolation_module.cpp:248] Failed to
>>>>>>> stop container mesos_executor_Task 0
>>>>>>> ("/home/dmitr...)_framework_201306121623-16777343-5050-26417-0000, lxc-stop
>>>>>>> returned: 512
>>>>>>> I0612 16:24:21.453385 26454 slave.cpp:829] Status update: task 0 of
>>>>>>> framework 201306121623-16777343-5050-26417-0000 is now in state TASK_FAILED
>>>>>>> E0612 16:24:21.453583 26453 lxc_isolation_module.cpp:273] ERROR!
>>>>>>> Asked to update resources for an unknown executor!
>>>>>>> I0612 16:24:21.453891 26451 gc.cpp:97] Scheduling
>>>>>>> /tmp/mesos/slaves/201306121623-16777343-5050-26417-0/frameworks/201306121623-16777343-5050-26417-0000/executors/Task
>>>>>>> 0 ("/home/dmitr...)/runs/9156d4fa-a177-464b-906f-fb62c8b9b363 for removal
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Mesos, lxc and ubuntu 12
Posted by Dmitriy Lyubimov <dl...@gmail.com>.
On Wed, Jun 12, 2013 at 5:24 PM, Benjamin Mahler
<be...@gmail.com>wrote:
> Cgroups does not allow one to mount the same subsystem across cgroup
> hierarchies. Do you have multiple cgroup hierarchies present on your
> machine?
>
I frankly know nothing of cgroups. how do i check? not that i know of .
>
> Ideally this will work with a stock ubuntu 12 OS, but it's possible that
> ubuntu already mounts a cgroup hierarchy with the freezer subsystem in a
> location we did not expect.
>
> What are the contents of the root directory on that machine?
>
bin cdrom etc home initrd.img.old lib64 media opt
root sbin srv tmp var vmlinuz.old
boot dev hadoop initrd.img lib lost+found mnt proc
run selinux sys usr vmlinuz
>
> On Wed, Jun 12, 2013 at 5:19 PM, Dmitriy Lyubimov <dl...@gmail.com>wrote:
>
>> ok thanks.
>>
>> now i switched to cgroups and can't get slave to start. The fatal error
>> says
>>
>> F0612 17:17:04.053773 10059 cgroups_isolation_module.cpp:161] Required
>> subsystem 'freezer' is already in use
>>
>> Any hints appreciated.
>>
>> thank you.
>>
>>
>> On Wed, Jun 12, 2013 at 4:59 PM, Vinod Kone <vi...@gmail.com> wrote:
>>
>>> No problem. Instead of giving --isolation=lxc, you could give
>>> --isolation=cgroups. Also for more flags, start mesos slave with --help.
>>> Unfortunately, we have been a bit behind on the documentation, so the only
>>> place you can look at are our header files (e.g.,
>>> src/slave/cgroups_isolation.hpp). That said, if your kernel supports it
>>> cgroups should work out of the box with mesos.
>>>
>>> HTH,
>>>
>>>
>>>
>>> -- Vinod
>>>
>>>
>>> On Wed, Jun 12, 2013 at 4:52 PM, Dmitriy Lyubimov <dl...@gmail.com>wrote:
>>>
>>>> Oops. I am just starting with this. I see it clearly not working.. I
>>>> just downloaded 0.11 and trying to set up spark 0.7.2 with it. it works ok
>>>> with "process" isolation. I assumed lxc would be preferrable since it is
>>>> being advertised feature on the Mesos home page.
>>>>
>>>> I will snoop around the docs looking for cgroups isolation. If you can
>>>> point me to manual, i'd be grateful too.
>>>>
>>>>
>>>>
>>>> On Wed, Jun 12, 2013 at 4:48 PM, Vinod Kone <vi...@gmail.com>wrote:
>>>>
>>>>> Hi Dmitry,
>>>>>
>>>>> What version of mesos are you using? Lxc support has been deprecated
>>>>> for a while now. You should use the new cgroups isolation.
>>>>>
>>>>>
>>>>>
>>>>> On Wed, Jun 12, 2013 at 4:26 PM, Dmitriy Lyubimov <dl...@gmail.com>wrote:
>>>>>
>>>>>> Hello,
>>>>>>
>>>>>> is there anything speicific to ubuntu 12 that needs to be done to
>>>>>> make Mesos work with LCX?
>>>>>>
>>>>>> I set things up according to ubuntu docs,
>>>>>> https://help.ubuntu.com/12.10/serverguide/lxc.html#lxc-creation
>>>>>>
>>>>>> and all container examples there seem to be happily working.
>>>>>>
>>>>>> However, some mesos unit tests are failing (which i suspect are
>>>>>> relating to lxc) as well as lxc isolation mode fails to spawn tasks.
>>>>>>
>>>>>> (I am actually on ubuntu 12-04 LTS).
>>>>>>
>>>>>> Is there any speicific way to troubleshoot this? Is LXC in Mesos even
>>>>>> working with Ubuntu 12?
>>>>>>
>>>>>> thank you in advance. (slave output enclosed).
>>>>>> -d
>>>>>>
>>>>>> I0612 16:24:20.682698 26452 slave.cpp:474] Got assigned task 0 for
>>>>>> framework 201306121623-16777343-5050-26417-0000
>>>>>> I0612 16:24:20.683425 26452 paths.hpp:234] Created executor directory
>>>>>> '/tmp/mesos/slaves/201306121623-16777343-5050-26417-0/frameworks/201306121623-16777343-5050-26417-0000/executors/Task
>>>>>> 0 ("/home/dmitr...)/runs/9156d4fa-a177-464b-906f-fb62c8b9b363'
>>>>>> I0612 16:24:20.683630 26453 lxc_isolation_module.cpp:121] Launching
>>>>>> Task 0 ("/home/dmitr...) (/usr/local/libexec/mesos/mesos-executor) in
>>>>>> /tmp/mesos/slaves/201306121623-16777343-5050-26417-0/frameworks/201306121623-16777343-5050-26417-0000/executors/Task
>>>>>> 0 ("/home/dmitr...)/runs/9156d4fa-a177-464b-906f-fb62c8b9b363 with
>>>>>> resources ' for framework 201306121623-16777343-5050-26417-0000
>>>>>> I0612 16:24:20.683945 26453 lxc_isolation_module.cpp:152] Forked
>>>>>> executor at = 26570
>>>>>> lxc-execute: No such file or directory - failed to create
>>>>>> '/sys/fs/cgroup/cpuset//lxc/mesos_executor_Task 0
>>>>>> ("/home/dmitr...)_framework_201306121623-16777343-5050-26417-0000' directory
>>>>>> lxc-execute: failed to spawn 'mesos_executor_Task 0
>>>>>> ("/home/dmitr...)_framework_201306121623-16777343-5050-26417-0000'
>>>>>> lxc-execute: No such file or directory - failed to remove cgroup
>>>>>> '/sys/fs/cgroup/cpuset//lxc/mesos_executor_Task 0
>>>>>> ("/home/dmitr...)_framework_201306121623-16777343-5050-26417-0000'
>>>>>> I0612 16:24:21.451616 26452 lxc_isolation_module.cpp:322] Telling
>>>>>> slave of lost executor Task 0 ("/home/dmitr...) of framework
>>>>>> 201306121623-16777343-5050-26417-0000
>>>>>> I0612 16:24:21.451709 26452 lxc_isolation_module.cpp:239] Stopping
>>>>>> container mesos_executor_Task 0
>>>>>> ("/home/dmitr...)_framework_201306121623-16777343-5050-26417-0000
>>>>>> I0612 16:24:21.452199 26454 slave.cpp:998] Executor 'Task 0
>>>>>> ("/home/dmitr...)' of framework 201306121623-16777343-5050-26417-0000 has
>>>>>> exited with status 255
>>>>>> sh: 1: Syntax error: "(" unexpected
>>>>>> E0612 16:24:21.453227 26452 lxc_isolation_module.cpp:248] Failed to
>>>>>> stop container mesos_executor_Task 0
>>>>>> ("/home/dmitr...)_framework_201306121623-16777343-5050-26417-0000, lxc-stop
>>>>>> returned: 512
>>>>>> I0612 16:24:21.453385 26454 slave.cpp:829] Status update: task 0 of
>>>>>> framework 201306121623-16777343-5050-26417-0000 is now in state TASK_FAILED
>>>>>> E0612 16:24:21.453583 26453 lxc_isolation_module.cpp:273] ERROR!
>>>>>> Asked to update resources for an unknown executor!
>>>>>> I0612 16:24:21.453891 26451 gc.cpp:97] Scheduling
>>>>>> /tmp/mesos/slaves/201306121623-16777343-5050-26417-0/frameworks/201306121623-16777343-5050-26417-0000/executors/Task
>>>>>> 0 ("/home/dmitr...)/runs/9156d4fa-a177-464b-906f-fb62c8b9b363 for removal
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Mesos, lxc and ubuntu 12
Posted by Benjamin Mahler <be...@gmail.com>.
Cgroups does not allow one to mount the same subsystem across cgroup
hierarchies. Do you have multiple cgroup hierarchies present on your
machine?
Ideally this will work with a stock ubuntu 12 OS, but it's possible that
ubuntu already mounts a cgroup hierarchy with the freezer subsystem in a
location we did not expect.
What are the contents of the root directory on that machine?
On Wed, Jun 12, 2013 at 5:19 PM, Dmitriy Lyubimov <dl...@gmail.com> wrote:
> ok thanks.
>
> now i switched to cgroups and can't get slave to start. The fatal error
> says
>
> F0612 17:17:04.053773 10059 cgroups_isolation_module.cpp:161] Required
> subsystem 'freezer' is already in use
>
> Any hints appreciated.
>
> thank you.
>
>
> On Wed, Jun 12, 2013 at 4:59 PM, Vinod Kone <vi...@gmail.com> wrote:
>
>> No problem. Instead of giving --isolation=lxc, you could give
>> --isolation=cgroups. Also for more flags, start mesos slave with --help.
>> Unfortunately, we have been a bit behind on the documentation, so the only
>> place you can look at are our header files (e.g.,
>> src/slave/cgroups_isolation.hpp). That said, if your kernel supports it
>> cgroups should work out of the box with mesos.
>>
>> HTH,
>>
>>
>>
>> -- Vinod
>>
>>
>> On Wed, Jun 12, 2013 at 4:52 PM, Dmitriy Lyubimov <dl...@gmail.com>wrote:
>>
>>> Oops. I am just starting with this. I see it clearly not working.. I
>>> just downloaded 0.11 and trying to set up spark 0.7.2 with it. it works ok
>>> with "process" isolation. I assumed lxc would be preferrable since it is
>>> being advertised feature on the Mesos home page.
>>>
>>> I will snoop around the docs looking for cgroups isolation. If you can
>>> point me to manual, i'd be grateful too.
>>>
>>>
>>>
>>> On Wed, Jun 12, 2013 at 4:48 PM, Vinod Kone <vi...@gmail.com> wrote:
>>>
>>>> Hi Dmitry,
>>>>
>>>> What version of mesos are you using? Lxc support has been deprecated
>>>> for a while now. You should use the new cgroups isolation.
>>>>
>>>>
>>>>
>>>> On Wed, Jun 12, 2013 at 4:26 PM, Dmitriy Lyubimov <dl...@gmail.com>wrote:
>>>>
>>>>> Hello,
>>>>>
>>>>> is there anything speicific to ubuntu 12 that needs to be done to make
>>>>> Mesos work with LCX?
>>>>>
>>>>> I set things up according to ubuntu docs,
>>>>> https://help.ubuntu.com/12.10/serverguide/lxc.html#lxc-creation
>>>>>
>>>>> and all container examples there seem to be happily working.
>>>>>
>>>>> However, some mesos unit tests are failing (which i suspect are
>>>>> relating to lxc) as well as lxc isolation mode fails to spawn tasks.
>>>>>
>>>>> (I am actually on ubuntu 12-04 LTS).
>>>>>
>>>>> Is there any speicific way to troubleshoot this? Is LXC in Mesos even
>>>>> working with Ubuntu 12?
>>>>>
>>>>> thank you in advance. (slave output enclosed).
>>>>> -d
>>>>>
>>>>> I0612 16:24:20.682698 26452 slave.cpp:474] Got assigned task 0 for
>>>>> framework 201306121623-16777343-5050-26417-0000
>>>>> I0612 16:24:20.683425 26452 paths.hpp:234] Created executor directory
>>>>> '/tmp/mesos/slaves/201306121623-16777343-5050-26417-0/frameworks/201306121623-16777343-5050-26417-0000/executors/Task
>>>>> 0 ("/home/dmitr...)/runs/9156d4fa-a177-464b-906f-fb62c8b9b363'
>>>>> I0612 16:24:20.683630 26453 lxc_isolation_module.cpp:121] Launching
>>>>> Task 0 ("/home/dmitr...) (/usr/local/libexec/mesos/mesos-executor) in
>>>>> /tmp/mesos/slaves/201306121623-16777343-5050-26417-0/frameworks/201306121623-16777343-5050-26417-0000/executors/Task
>>>>> 0 ("/home/dmitr...)/runs/9156d4fa-a177-464b-906f-fb62c8b9b363 with
>>>>> resources ' for framework 201306121623-16777343-5050-26417-0000
>>>>> I0612 16:24:20.683945 26453 lxc_isolation_module.cpp:152] Forked
>>>>> executor at = 26570
>>>>> lxc-execute: No such file or directory - failed to create
>>>>> '/sys/fs/cgroup/cpuset//lxc/mesos_executor_Task 0
>>>>> ("/home/dmitr...)_framework_201306121623-16777343-5050-26417-0000' directory
>>>>> lxc-execute: failed to spawn 'mesos_executor_Task 0
>>>>> ("/home/dmitr...)_framework_201306121623-16777343-5050-26417-0000'
>>>>> lxc-execute: No such file or directory - failed to remove cgroup
>>>>> '/sys/fs/cgroup/cpuset//lxc/mesos_executor_Task 0
>>>>> ("/home/dmitr...)_framework_201306121623-16777343-5050-26417-0000'
>>>>> I0612 16:24:21.451616 26452 lxc_isolation_module.cpp:322] Telling
>>>>> slave of lost executor Task 0 ("/home/dmitr...) of framework
>>>>> 201306121623-16777343-5050-26417-0000
>>>>> I0612 16:24:21.451709 26452 lxc_isolation_module.cpp:239] Stopping
>>>>> container mesos_executor_Task 0
>>>>> ("/home/dmitr...)_framework_201306121623-16777343-5050-26417-0000
>>>>> I0612 16:24:21.452199 26454 slave.cpp:998] Executor 'Task 0
>>>>> ("/home/dmitr...)' of framework 201306121623-16777343-5050-26417-0000 has
>>>>> exited with status 255
>>>>> sh: 1: Syntax error: "(" unexpected
>>>>> E0612 16:24:21.453227 26452 lxc_isolation_module.cpp:248] Failed to
>>>>> stop container mesos_executor_Task 0
>>>>> ("/home/dmitr...)_framework_201306121623-16777343-5050-26417-0000, lxc-stop
>>>>> returned: 512
>>>>> I0612 16:24:21.453385 26454 slave.cpp:829] Status update: task 0 of
>>>>> framework 201306121623-16777343-5050-26417-0000 is now in state TASK_FAILED
>>>>> E0612 16:24:21.453583 26453 lxc_isolation_module.cpp:273] ERROR! Asked
>>>>> to update resources for an unknown executor!
>>>>> I0612 16:24:21.453891 26451 gc.cpp:97] Scheduling
>>>>> /tmp/mesos/slaves/201306121623-16777343-5050-26417-0/frameworks/201306121623-16777343-5050-26417-0000/executors/Task
>>>>> 0 ("/home/dmitr...)/runs/9156d4fa-a177-464b-906f-fb62c8b9b363 for removal
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Mesos, lxc and ubuntu 12
Posted by Dmitriy Lyubimov <dl...@gmail.com>.
ok thanks.
now i switched to cgroups and can't get slave to start. The fatal error
says
F0612 17:17:04.053773 10059 cgroups_isolation_module.cpp:161] Required
subsystem 'freezer' is already in use
Any hints appreciated.
thank you.
On Wed, Jun 12, 2013 at 4:59 PM, Vinod Kone <vi...@gmail.com> wrote:
> No problem. Instead of giving --isolation=lxc, you could give
> --isolation=cgroups. Also for more flags, start mesos slave with --help.
> Unfortunately, we have been a bit behind on the documentation, so the only
> place you can look at are our header files (e.g.,
> src/slave/cgroups_isolation.hpp). That said, if your kernel supports it
> cgroups should work out of the box with mesos.
>
> HTH,
>
>
>
> -- Vinod
>
>
> On Wed, Jun 12, 2013 at 4:52 PM, Dmitriy Lyubimov <dl...@gmail.com>wrote:
>
>> Oops. I am just starting with this. I see it clearly not working.. I
>> just downloaded 0.11 and trying to set up spark 0.7.2 with it. it works ok
>> with "process" isolation. I assumed lxc would be preferrable since it is
>> being advertised feature on the Mesos home page.
>>
>> I will snoop around the docs looking for cgroups isolation. If you can
>> point me to manual, i'd be grateful too.
>>
>>
>>
>> On Wed, Jun 12, 2013 at 4:48 PM, Vinod Kone <vi...@gmail.com> wrote:
>>
>>> Hi Dmitry,
>>>
>>> What version of mesos are you using? Lxc support has been deprecated for
>>> a while now. You should use the new cgroups isolation.
>>>
>>>
>>>
>>> On Wed, Jun 12, 2013 at 4:26 PM, Dmitriy Lyubimov <dl...@gmail.com>wrote:
>>>
>>>> Hello,
>>>>
>>>> is there anything speicific to ubuntu 12 that needs to be done to make
>>>> Mesos work with LCX?
>>>>
>>>> I set things up according to ubuntu docs,
>>>> https://help.ubuntu.com/12.10/serverguide/lxc.html#lxc-creation
>>>>
>>>> and all container examples there seem to be happily working.
>>>>
>>>> However, some mesos unit tests are failing (which i suspect are
>>>> relating to lxc) as well as lxc isolation mode fails to spawn tasks.
>>>>
>>>> (I am actually on ubuntu 12-04 LTS).
>>>>
>>>> Is there any speicific way to troubleshoot this? Is LXC in Mesos even
>>>> working with Ubuntu 12?
>>>>
>>>> thank you in advance. (slave output enclosed).
>>>> -d
>>>>
>>>> I0612 16:24:20.682698 26452 slave.cpp:474] Got assigned task 0 for
>>>> framework 201306121623-16777343-5050-26417-0000
>>>> I0612 16:24:20.683425 26452 paths.hpp:234] Created executor directory
>>>> '/tmp/mesos/slaves/201306121623-16777343-5050-26417-0/frameworks/201306121623-16777343-5050-26417-0000/executors/Task
>>>> 0 ("/home/dmitr...)/runs/9156d4fa-a177-464b-906f-fb62c8b9b363'
>>>> I0612 16:24:20.683630 26453 lxc_isolation_module.cpp:121] Launching
>>>> Task 0 ("/home/dmitr...) (/usr/local/libexec/mesos/mesos-executor) in
>>>> /tmp/mesos/slaves/201306121623-16777343-5050-26417-0/frameworks/201306121623-16777343-5050-26417-0000/executors/Task
>>>> 0 ("/home/dmitr...)/runs/9156d4fa-a177-464b-906f-fb62c8b9b363 with
>>>> resources ' for framework 201306121623-16777343-5050-26417-0000
>>>> I0612 16:24:20.683945 26453 lxc_isolation_module.cpp:152] Forked
>>>> executor at = 26570
>>>> lxc-execute: No such file or directory - failed to create
>>>> '/sys/fs/cgroup/cpuset//lxc/mesos_executor_Task 0
>>>> ("/home/dmitr...)_framework_201306121623-16777343-5050-26417-0000' directory
>>>> lxc-execute: failed to spawn 'mesos_executor_Task 0
>>>> ("/home/dmitr...)_framework_201306121623-16777343-5050-26417-0000'
>>>> lxc-execute: No such file or directory - failed to remove cgroup
>>>> '/sys/fs/cgroup/cpuset//lxc/mesos_executor_Task 0
>>>> ("/home/dmitr...)_framework_201306121623-16777343-5050-26417-0000'
>>>> I0612 16:24:21.451616 26452 lxc_isolation_module.cpp:322] Telling slave
>>>> of lost executor Task 0 ("/home/dmitr...) of framework
>>>> 201306121623-16777343-5050-26417-0000
>>>> I0612 16:24:21.451709 26452 lxc_isolation_module.cpp:239] Stopping
>>>> container mesos_executor_Task 0
>>>> ("/home/dmitr...)_framework_201306121623-16777343-5050-26417-0000
>>>> I0612 16:24:21.452199 26454 slave.cpp:998] Executor 'Task 0
>>>> ("/home/dmitr...)' of framework 201306121623-16777343-5050-26417-0000 has
>>>> exited with status 255
>>>> sh: 1: Syntax error: "(" unexpected
>>>> E0612 16:24:21.453227 26452 lxc_isolation_module.cpp:248] Failed to
>>>> stop container mesos_executor_Task 0
>>>> ("/home/dmitr...)_framework_201306121623-16777343-5050-26417-0000, lxc-stop
>>>> returned: 512
>>>> I0612 16:24:21.453385 26454 slave.cpp:829] Status update: task 0 of
>>>> framework 201306121623-16777343-5050-26417-0000 is now in state TASK_FAILED
>>>> E0612 16:24:21.453583 26453 lxc_isolation_module.cpp:273] ERROR! Asked
>>>> to update resources for an unknown executor!
>>>> I0612 16:24:21.453891 26451 gc.cpp:97] Scheduling
>>>> /tmp/mesos/slaves/201306121623-16777343-5050-26417-0/frameworks/201306121623-16777343-5050-26417-0000/executors/Task
>>>> 0 ("/home/dmitr...)/runs/9156d4fa-a177-464b-906f-fb62c8b9b363 for removal
>>>>
>>>>
>>>>
>>>>
>>>
>>
>
Re: Mesos, lxc and ubuntu 12
Posted by Vinod Kone <vi...@gmail.com>.
No problem. Instead of giving --isolation=lxc, you could give
--isolation=cgroups. Also for more flags, start mesos slave with --help.
Unfortunately, we have been a bit behind on the documentation, so the only
place you can look at are our header files (e.g.,
src/slave/cgroups_isolation.hpp). That said, if your kernel supports it
cgroups should work out of the box with mesos.
HTH,
-- Vinod
On Wed, Jun 12, 2013 at 4:52 PM, Dmitriy Lyubimov <dl...@gmail.com> wrote:
> Oops. I am just starting with this. I see it clearly not working.. I just
> downloaded 0.11 and trying to set up spark 0.7.2 with it. it works ok with
> "process" isolation. I assumed lxc would be preferrable since it is being
> advertised feature on the Mesos home page.
>
> I will snoop around the docs looking for cgroups isolation. If you can
> point me to manual, i'd be grateful too.
>
>
>
> On Wed, Jun 12, 2013 at 4:48 PM, Vinod Kone <vi...@gmail.com> wrote:
>
>> Hi Dmitry,
>>
>> What version of mesos are you using? Lxc support has been deprecated for
>> a while now. You should use the new cgroups isolation.
>>
>>
>>
>> On Wed, Jun 12, 2013 at 4:26 PM, Dmitriy Lyubimov <dl...@gmail.com>wrote:
>>
>>> Hello,
>>>
>>> is there anything speicific to ubuntu 12 that needs to be done to make
>>> Mesos work with LCX?
>>>
>>> I set things up according to ubuntu docs,
>>> https://help.ubuntu.com/12.10/serverguide/lxc.html#lxc-creation
>>>
>>> and all container examples there seem to be happily working.
>>>
>>> However, some mesos unit tests are failing (which i suspect are relating
>>> to lxc) as well as lxc isolation mode fails to spawn tasks.
>>>
>>> (I am actually on ubuntu 12-04 LTS).
>>>
>>> Is there any speicific way to troubleshoot this? Is LXC in Mesos even
>>> working with Ubuntu 12?
>>>
>>> thank you in advance. (slave output enclosed).
>>> -d
>>>
>>> I0612 16:24:20.682698 26452 slave.cpp:474] Got assigned task 0 for
>>> framework 201306121623-16777343-5050-26417-0000
>>> I0612 16:24:20.683425 26452 paths.hpp:234] Created executor directory
>>> '/tmp/mesos/slaves/201306121623-16777343-5050-26417-0/frameworks/201306121623-16777343-5050-26417-0000/executors/Task
>>> 0 ("/home/dmitr...)/runs/9156d4fa-a177-464b-906f-fb62c8b9b363'
>>> I0612 16:24:20.683630 26453 lxc_isolation_module.cpp:121] Launching Task
>>> 0 ("/home/dmitr...) (/usr/local/libexec/mesos/mesos-executor) in
>>> /tmp/mesos/slaves/201306121623-16777343-5050-26417-0/frameworks/201306121623-16777343-5050-26417-0000/executors/Task
>>> 0 ("/home/dmitr...)/runs/9156d4fa-a177-464b-906f-fb62c8b9b363 with
>>> resources ' for framework 201306121623-16777343-5050-26417-0000
>>> I0612 16:24:20.683945 26453 lxc_isolation_module.cpp:152] Forked
>>> executor at = 26570
>>> lxc-execute: No such file or directory - failed to create
>>> '/sys/fs/cgroup/cpuset//lxc/mesos_executor_Task 0
>>> ("/home/dmitr...)_framework_201306121623-16777343-5050-26417-0000' directory
>>> lxc-execute: failed to spawn 'mesos_executor_Task 0
>>> ("/home/dmitr...)_framework_201306121623-16777343-5050-26417-0000'
>>> lxc-execute: No such file or directory - failed to remove cgroup
>>> '/sys/fs/cgroup/cpuset//lxc/mesos_executor_Task 0
>>> ("/home/dmitr...)_framework_201306121623-16777343-5050-26417-0000'
>>> I0612 16:24:21.451616 26452 lxc_isolation_module.cpp:322] Telling slave
>>> of lost executor Task 0 ("/home/dmitr...) of framework
>>> 201306121623-16777343-5050-26417-0000
>>> I0612 16:24:21.451709 26452 lxc_isolation_module.cpp:239] Stopping
>>> container mesos_executor_Task 0
>>> ("/home/dmitr...)_framework_201306121623-16777343-5050-26417-0000
>>> I0612 16:24:21.452199 26454 slave.cpp:998] Executor 'Task 0
>>> ("/home/dmitr...)' of framework 201306121623-16777343-5050-26417-0000 has
>>> exited with status 255
>>> sh: 1: Syntax error: "(" unexpected
>>> E0612 16:24:21.453227 26452 lxc_isolation_module.cpp:248] Failed to stop
>>> container mesos_executor_Task 0
>>> ("/home/dmitr...)_framework_201306121623-16777343-5050-26417-0000, lxc-stop
>>> returned: 512
>>> I0612 16:24:21.453385 26454 slave.cpp:829] Status update: task 0 of
>>> framework 201306121623-16777343-5050-26417-0000 is now in state TASK_FAILED
>>> E0612 16:24:21.453583 26453 lxc_isolation_module.cpp:273] ERROR! Asked
>>> to update resources for an unknown executor!
>>> I0612 16:24:21.453891 26451 gc.cpp:97] Scheduling
>>> /tmp/mesos/slaves/201306121623-16777343-5050-26417-0/frameworks/201306121623-16777343-5050-26417-0000/executors/Task
>>> 0 ("/home/dmitr...)/runs/9156d4fa-a177-464b-906f-fb62c8b9b363 for removal
>>>
>>>
>>>
>>>
>>
>
Re: Mesos, lxc and ubuntu 12
Posted by Dmitriy Lyubimov <dl...@gmail.com>.
Oops. I am just starting with this. I see it clearly not working.. I just
downloaded 0.11 and trying to set up spark 0.7.2 with it. it works ok with
"process" isolation. I assumed lxc would be preferrable since it is being
advertised feature on the Mesos home page.
I will snoop around the docs looking for cgroups isolation. If you can
point me to manual, i'd be grateful too.
On Wed, Jun 12, 2013 at 4:48 PM, Vinod Kone <vi...@gmail.com> wrote:
> Hi Dmitry,
>
> What version of mesos are you using? Lxc support has been deprecated for a
> while now. You should use the new cgroups isolation.
>
>
>
> On Wed, Jun 12, 2013 at 4:26 PM, Dmitriy Lyubimov <dl...@gmail.com>wrote:
>
>> Hello,
>>
>> is there anything speicific to ubuntu 12 that needs to be done to make
>> Mesos work with LCX?
>>
>> I set things up according to ubuntu docs,
>> https://help.ubuntu.com/12.10/serverguide/lxc.html#lxc-creation
>>
>> and all container examples there seem to be happily working.
>>
>> However, some mesos unit tests are failing (which i suspect are relating
>> to lxc) as well as lxc isolation mode fails to spawn tasks.
>>
>> (I am actually on ubuntu 12-04 LTS).
>>
>> Is there any speicific way to troubleshoot this? Is LXC in Mesos even
>> working with Ubuntu 12?
>>
>> thank you in advance. (slave output enclosed).
>> -d
>>
>> I0612 16:24:20.682698 26452 slave.cpp:474] Got assigned task 0 for
>> framework 201306121623-16777343-5050-26417-0000
>> I0612 16:24:20.683425 26452 paths.hpp:234] Created executor directory
>> '/tmp/mesos/slaves/201306121623-16777343-5050-26417-0/frameworks/201306121623-16777343-5050-26417-0000/executors/Task
>> 0 ("/home/dmitr...)/runs/9156d4fa-a177-464b-906f-fb62c8b9b363'
>> I0612 16:24:20.683630 26453 lxc_isolation_module.cpp:121] Launching Task
>> 0 ("/home/dmitr...) (/usr/local/libexec/mesos/mesos-executor) in
>> /tmp/mesos/slaves/201306121623-16777343-5050-26417-0/frameworks/201306121623-16777343-5050-26417-0000/executors/Task
>> 0 ("/home/dmitr...)/runs/9156d4fa-a177-464b-906f-fb62c8b9b363 with
>> resources ' for framework 201306121623-16777343-5050-26417-0000
>> I0612 16:24:20.683945 26453 lxc_isolation_module.cpp:152] Forked executor
>> at = 26570
>> lxc-execute: No such file or directory - failed to create
>> '/sys/fs/cgroup/cpuset//lxc/mesos_executor_Task 0
>> ("/home/dmitr...)_framework_201306121623-16777343-5050-26417-0000' directory
>> lxc-execute: failed to spawn 'mesos_executor_Task 0
>> ("/home/dmitr...)_framework_201306121623-16777343-5050-26417-0000'
>> lxc-execute: No such file or directory - failed to remove cgroup
>> '/sys/fs/cgroup/cpuset//lxc/mesos_executor_Task 0
>> ("/home/dmitr...)_framework_201306121623-16777343-5050-26417-0000'
>> I0612 16:24:21.451616 26452 lxc_isolation_module.cpp:322] Telling slave
>> of lost executor Task 0 ("/home/dmitr...) of framework
>> 201306121623-16777343-5050-26417-0000
>> I0612 16:24:21.451709 26452 lxc_isolation_module.cpp:239] Stopping
>> container mesos_executor_Task 0
>> ("/home/dmitr...)_framework_201306121623-16777343-5050-26417-0000
>> I0612 16:24:21.452199 26454 slave.cpp:998] Executor 'Task 0
>> ("/home/dmitr...)' of framework 201306121623-16777343-5050-26417-0000 has
>> exited with status 255
>> sh: 1: Syntax error: "(" unexpected
>> E0612 16:24:21.453227 26452 lxc_isolation_module.cpp:248] Failed to stop
>> container mesos_executor_Task 0
>> ("/home/dmitr...)_framework_201306121623-16777343-5050-26417-0000, lxc-stop
>> returned: 512
>> I0612 16:24:21.453385 26454 slave.cpp:829] Status update: task 0 of
>> framework 201306121623-16777343-5050-26417-0000 is now in state TASK_FAILED
>> E0612 16:24:21.453583 26453 lxc_isolation_module.cpp:273] ERROR! Asked to
>> update resources for an unknown executor!
>> I0612 16:24:21.453891 26451 gc.cpp:97] Scheduling
>> /tmp/mesos/slaves/201306121623-16777343-5050-26417-0/frameworks/201306121623-16777343-5050-26417-0000/executors/Task
>> 0 ("/home/dmitr...)/runs/9156d4fa-a177-464b-906f-fb62c8b9b363 for removal
>>
>>
>>
>>
>
Re: Mesos, lxc and ubuntu 12
Posted by Vinod Kone <vi...@gmail.com>.
Hi Dmitry,
What version of mesos are you using? Lxc support has been deprecated for a
while now. You should use the new cgroups isolation.
On Wed, Jun 12, 2013 at 4:26 PM, Dmitriy Lyubimov <dl...@gmail.com> wrote:
> Hello,
>
> is there anything speicific to ubuntu 12 that needs to be done to make
> Mesos work with LCX?
>
> I set things up according to ubuntu docs,
> https://help.ubuntu.com/12.10/serverguide/lxc.html#lxc-creation
>
> and all container examples there seem to be happily working.
>
> However, some mesos unit tests are failing (which i suspect are relating
> to lxc) as well as lxc isolation mode fails to spawn tasks.
>
> (I am actually on ubuntu 12-04 LTS).
>
> Is there any speicific way to troubleshoot this? Is LXC in Mesos even
> working with Ubuntu 12?
>
> thank you in advance. (slave output enclosed).
> -d
>
> I0612 16:24:20.682698 26452 slave.cpp:474] Got assigned task 0 for
> framework 201306121623-16777343-5050-26417-0000
> I0612 16:24:20.683425 26452 paths.hpp:234] Created executor directory
> '/tmp/mesos/slaves/201306121623-16777343-5050-26417-0/frameworks/201306121623-16777343-5050-26417-0000/executors/Task
> 0 ("/home/dmitr...)/runs/9156d4fa-a177-464b-906f-fb62c8b9b363'
> I0612 16:24:20.683630 26453 lxc_isolation_module.cpp:121] Launching Task 0
> ("/home/dmitr...) (/usr/local/libexec/mesos/mesos-executor) in
> /tmp/mesos/slaves/201306121623-16777343-5050-26417-0/frameworks/201306121623-16777343-5050-26417-0000/executors/Task
> 0 ("/home/dmitr...)/runs/9156d4fa-a177-464b-906f-fb62c8b9b363 with
> resources ' for framework 201306121623-16777343-5050-26417-0000
> I0612 16:24:20.683945 26453 lxc_isolation_module.cpp:152] Forked executor
> at = 26570
> lxc-execute: No such file or directory - failed to create
> '/sys/fs/cgroup/cpuset//lxc/mesos_executor_Task 0
> ("/home/dmitr...)_framework_201306121623-16777343-5050-26417-0000' directory
> lxc-execute: failed to spawn 'mesos_executor_Task 0
> ("/home/dmitr...)_framework_201306121623-16777343-5050-26417-0000'
> lxc-execute: No such file or directory - failed to remove cgroup
> '/sys/fs/cgroup/cpuset//lxc/mesos_executor_Task 0
> ("/home/dmitr...)_framework_201306121623-16777343-5050-26417-0000'
> I0612 16:24:21.451616 26452 lxc_isolation_module.cpp:322] Telling slave of
> lost executor Task 0 ("/home/dmitr...) of framework
> 201306121623-16777343-5050-26417-0000
> I0612 16:24:21.451709 26452 lxc_isolation_module.cpp:239] Stopping
> container mesos_executor_Task 0
> ("/home/dmitr...)_framework_201306121623-16777343-5050-26417-0000
> I0612 16:24:21.452199 26454 slave.cpp:998] Executor 'Task 0
> ("/home/dmitr...)' of framework 201306121623-16777343-5050-26417-0000 has
> exited with status 255
> sh: 1: Syntax error: "(" unexpected
> E0612 16:24:21.453227 26452 lxc_isolation_module.cpp:248] Failed to stop
> container mesos_executor_Task 0
> ("/home/dmitr...)_framework_201306121623-16777343-5050-26417-0000, lxc-stop
> returned: 512
> I0612 16:24:21.453385 26454 slave.cpp:829] Status update: task 0 of
> framework 201306121623-16777343-5050-26417-0000 is now in state TASK_FAILED
> E0612 16:24:21.453583 26453 lxc_isolation_module.cpp:273] ERROR! Asked to
> update resources for an unknown executor!
> I0612 16:24:21.453891 26451 gc.cpp:97] Scheduling
> /tmp/mesos/slaves/201306121623-16777343-5050-26417-0/frameworks/201306121623-16777343-5050-26417-0000/executors/Task
> 0 ("/home/dmitr...)/runs/9156d4fa-a177-464b-906f-fb62c8b9b363 for removal
>
>
>
>