You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mesos.apache.org by Kevin Lyda <ke...@ie.suberic.net> on 2013/08/01 17:59:20 UTC
make check failure...
Running make check on master from github (bfdfdce) and I'm getting a failure:
[ OK ] AllocatorZooKeeperTest/0.SlaveReregistersFirst (2098 ms)
[----------] 2 tests from AllocatorZooKeeperTest/0 (4242 ms total)
[----------] Global test environment tear-down
[==========] 240 tests from 46 test cases ran. (91604 ms total)
[ PASSED ] 239 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] CgroupsIsolatorTest.ROOT_CGROUPS_BalloonFramework
1 FAILED TEST
make[3]: *** [check-local] Error 1
make[3]: Leaving directory `/usr/src/mesos/build/src'
make[2]: *** [check-am] Error 2
make[2]: Leaving directory `/usr/src/mesos/build/src'
make[1]: *** [check] Error 2
make[1]: Leaving directory `/usr/src/mesos/build/src'
make: *** [check-recursive] Error 1
This is on an up-to-date Ubuntu 12.04.2. Ah. I realise I hadn't
installed lxc. OK, trying again.
Nope. Now the errors are worse (built from a clean tree, same commit):
[----------] Global test environment tear-down
[==========] 239 tests from 45 test cases ran. (91234 ms total)
[ PASSED ] 216 tests.
[ FAILED ] 23 tests, listed below:
[ FAILED ] SlaveRecoveryTest/1.RecoverSlaveState, where TypeParam =
mesos::internal::slave::CgroupsIsolator
[ FAILED ] SlaveRecoveryTest/1.RecoverStatusUpdateManager, where
TypeParam = mesos::internal::slave::CgroupsIsolator
[ FAILED ] SlaveRecoveryTest/1.ReconnectExecutor, where TypeParam =
mesos::internal::slave::CgroupsIsolator
[ FAILED ] SlaveRecoveryTest/1.RecoverUnregisteredExecutor, where
TypeParam = mesos::internal::slave::CgroupsIsolator
[ FAILED ] SlaveRecoveryTest/1.RecoverTerminatedExecutor, where
TypeParam = mesos::internal::slave::CgroupsIsolator
[ FAILED ] SlaveRecoveryTest/1.CleanupExecutor, where TypeParam =
mesos::internal::slave::CgroupsIsolator
[ FAILED ] SlaveRecoveryTest/1.RemoveNonCheckpointingFramework,
where TypeParam = mesos::internal::slave::CgroupsIsolator
[ FAILED ] SlaveRecoveryTest/1.NonCheckpointingFramework, where
TypeParam = mesos::internal::slave::CgroupsIsolator
[ FAILED ] SlaveRecoveryTest/1.NonCheckpointingSlave, where
TypeParam = mesos::internal::slave::CgroupsIsolator
[ FAILED ] SlaveRecoveryTest/1.KillTask, where TypeParam =
mesos::internal::slave::CgroupsIsolator
[ FAILED ] SlaveRecoveryTest/1.GCExecutor, where TypeParam =
mesos::internal::slave::CgroupsIsolator
[ FAILED ] SlaveRecoveryTest/1.ShutdownSlave, where TypeParam =
mesos::internal::slave::CgroupsIsolator
[ FAILED ] IsolatorTest/1.Usage, where TypeParam =
mesos::internal::slave::CgroupsIsolator
[ FAILED ] CgroupsIsolatorTest.ROOT_CGROUPS_BalloonFramework
[ FAILED ] CgroupsAnyHierarchyWithCpuMemoryTest.ROOT_CGROUPS_Busy
[ FAILED ] CgroupsAnyHierarchyWithCpuMemoryTest.ROOT_CGROUPS_SubsystemsHierarchy
[ FAILED ] CgroupsAnyHierarchyWithCpuMemoryTest.ROOT_CGROUPS_MountedSubsystems
[ FAILED ] CgroupsAnyHierarchyWithCpuMemoryTest.ROOT_CGROUPS_CreateRemove
[ FAILED ] CgroupsAnyHierarchyWithCpuMemoryTest.ROOT_CGROUPS_Listen
[ FAILED ] CgroupsAnyHierarchyWithCpuAcctMemoryTest.ROOT_CGROUPS_Stat
[ FAILED ] CgroupsAnyHierarchyWithCpuMemoryFreezerTest.ROOT_CGROUPS_Freeze
[ FAILED ] CgroupsAnyHierarchyWithCpuMemoryFreezerTest.ROOT_CGROUPS_Kill
[ FAILED ] CgroupsAnyHierarchyWithCpuMemoryFreezerTest.ROOT_CGROUPS_Destroy
23 FAILED TESTS
make[3]: *** [check-local] Error 1
make[3]: Leaving directory `/usr/src/mesos/build/src'
make[2]: *** [check-am] Error 2
make[2]: Leaving directory `/usr/src/mesos/build/src'
make[1]: *** [check] Error 2
make[1]: Leaving directory `/usr/src/mesos/build/src'
make: *** [check-recursive] Error 1
Kevin
--
Kevin Lyda
Galway, Ireland
US Citizen overseas? We can vote.
Register now: http://www.votefromabroad.org/
Re: make check failure...
Posted by Ian Downes <id...@twitter.com>.
The failing test is an integration test that seems to be triggering a kernel bug. All other tests are passing so you should just ignore it for now; Mesos will run.
There is some discussion here resulting from this same test failing: https://lkml.org/lkml/2013/7/28/74
Does running the test leave an unkillable process as described in the post? If so, can you check the /proc/<pid>/stack and /proc/<pid>/status?
Thanks,
Ian
On Aug 1, 2013, at 1:18 PM, Kevin Lyda <ke...@ie.suberic.net> wrote:
> Not running systemd. Ubuntu uses upstart.
>
> cgroups are mounted by /etc/init/cgconfig.conf which is installed by
> cgroup-bin (which I need for lscgroup).
>
> I turned that off by doing:
>
> service cgconfig stop
> echo manual > /etc/init/cgconfig.override
>
> The echo line turns it off permanently.
>
> OK, ran make check and am back to the original error! Hurrah!
>
> [----------] Global test environment tear-down
> [==========] 240 tests from 46 test cases ran. (91790 ms total)
> [ PASSED ] 239 tests.
> [ FAILED ] 1 test, listed below:
> [ FAILED ] CgroupsIsolatorTest.ROOT_CGROUPS_BalloonFramework
>
> 1 FAILED TEST
> make[3]: *** [check-local] Error 1
> make[3]: Leaving directory `/usr/src/mesos/build/src'
> make[2]: *** [check-am] Error 2
> make[2]: Leaving directory `/usr/src/mesos/build/src'
> make[1]: *** [check] Error 2
> make[1]: Leaving directory `/usr/src/mesos/build/src'
> make: *** [check-recursive] Error 1
>
> # lscgroup
> cgroups can't be listed: Cgroup is not mounted
>
> # mount
> /dev/sda1 on / type ext4 (rw,errors=remount-ro)
> proc on /proc type proc (rw,noexec,nosuid,nodev)
> sysfs on /sys type sysfs (rw,noexec,nosuid,nodev)
> none on /sys/fs/fuse/connections type fusectl (rw)
> none on /sys/kernel/debug type debugfs (rw)
> none on /sys/kernel/security type securityfs (rw)
> udev on /dev type devtmpfs (rw,mode=0755)
> devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620)
> tmpfs on /run type tmpfs (rw,noexec,nosuid,size=10%,mode=0755)
> none on /run/lock type tmpfs (rw,noexec,nosuid,nodev,size=5242880)
> none on /run/shm type tmpfs (rw,nosuid,nodev)
> /dev/sdd1 on /u4 type ext4 (rw)
> /dev/sdb1 on /u2 type ext4 (rw)
> /dev/sdc1 on /u3 type ext4 (rw)
> /dev/sda6 on /u1 type ext4 (rw)
>
>
> On Thu, Aug 1, 2013 at 8:25 PM, Ian Downes <id...@twitter.com> wrote:
>> Looks like cgroups are mounted on /sys/fs/cgroup so I'd expect the tests to fail again. Are you running systemd? If the tests do fail, and you're not running systemd, you should try manually umount'ing cgroups and re-running the tests (no need to rebuild, just run `make check` again).
>>
>> Ian
>> On Aug 1, 2013, at 12:18 PM, Kevin Lyda <ke...@ie.suberic.net> wrote:
>>
>>> [Initial reply not sent to list]
>>>
>>> Will remove lxc and make sure cgroups are unmounted. This is not in a
>>> VM; it's on an x86_64 machine. Replying via phone so will get the
>>> other info in a bit.
>>>
>>> [Followup]
>>>
>>> Right, removed lxc. Noted that there's a cgroups-lite package that
>>> might be an issue. Hm. No, that mounts cgroups.
>>>
>>> OK, lscgroup is in the cgroup-bin package. That also mounts cgroups.
>>> But perhaps it isn't busy w/o lxc installed?
>>>
>>> Trying a build now. Output of requested commands follows; will report
>>> build results when they're done.
>>>
>>> # lscgroup
>>> cpu:/
>>> cpu:/sysdefault
>>> cpuacct:/
>>> cpuacct:/sysdefault
>>> devices:/
>>> devices:/sysdefault
>>> memory:/
>>> memory:/sysdefault
>>> freezer:/
>>> freezer:/sysdefault
>>> # uname -a
>>> Linux node29 3.5.0-23-generic #35~precise1-Ubuntu SMP Fri Jan 25
>>> 17:13:26 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
>>> # mount
>>> /dev/sda1 on / type ext4 (rw,errors=remount-ro)
>>> proc on /proc type proc (rw,noexec,nosuid,nodev)
>>> sysfs on /sys type sysfs (rw,noexec,nosuid,nodev)
>>> none on /sys/fs/fuse/connections type fusectl (rw)
>>> none on /sys/kernel/debug type debugfs (rw)
>>> none on /sys/kernel/security type securityfs (rw)
>>> udev on /dev type devtmpfs (rw,mode=0755)
>>> devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620)
>>> tmpfs on /run type tmpfs (rw,noexec,nosuid,size=10%,mode=0755)
>>> none on /run/lock type tmpfs (rw,noexec,nosuid,nodev,size=5242880)
>>> none on /run/shm type tmpfs (rw,nosuid,nodev)
>>> /dev/sdd1 on /u4 type ext4 (rw)
>>> /dev/sdb1 on /u2 type ext4 (rw)
>>> /dev/sdc1 on /u3 type ext4 (rw)
>>> /dev/sda6 on /u1 type ext4 (rw)
>>> cgroups on /sys/fs/cgroup type tmpfs (rw,uid=0,gid=0,mode=0755)
>>>
>>> Thanks for the feedback,
>>>
>>> Kevin
>>>
>>>
>>> On Thu, Aug 1, 2013 at 7:43 PM, Ian Downes <id...@twitter.com> wrote:
>>>> The documentation is out of date and lxc has been deprecated; cgroups are used directly instead. We'll be updating the documentation shortly to reflect this.
>>>>
>>>> Installing lxc possibly mounted cgroups and led to the slew of busy errors. Could you please post the output of `mount` and, if you have libcgroup installed, `lscgroup`, ideally before and after lxc is installed? And also `uname -a`.
>>>>
>>>> The failed BalloonFramework test may be related to a bug that we're investigating. Are you running this inside a VM by any chance?
>>>>
>>>> Ian
>>>>
>>>> On Aug 1, 2013, at 8:59 AM, Kevin Lyda <ke...@ie.suberic.net> wrote:
>>>>
>>>>> Running make check on master from github (bfdfdce) and I'm getting a failure:
>>>>>
>>>>> [ OK ] AllocatorZooKeeperTest/0.SlaveReregistersFirst (2098 ms)
>>>>> [----------] 2 tests from AllocatorZooKeeperTest/0 (4242 ms total)
>>>>>
>>>>> [----------] Global test environment tear-down
>>>>> [==========] 240 tests from 46 test cases ran. (91604 ms total)
>>>>> [ PASSED ] 239 tests.
>>>>> [ FAILED ] 1 test, listed below:
>>>>> [ FAILED ] CgroupsIsolatorTest.ROOT_CGROUPS_BalloonFramework
>>>>>
>>>>> 1 FAILED TEST
>>>>> make[3]: *** [check-local] Error 1
>>>>> make[3]: Leaving directory `/usr/src/mesos/build/src'
>>>>> make[2]: *** [check-am] Error 2
>>>>> make[2]: Leaving directory `/usr/src/mesos/build/src'
>>>>> make[1]: *** [check] Error 2
>>>>> make[1]: Leaving directory `/usr/src/mesos/build/src'
>>>>> make: *** [check-recursive] Error 1
>>>>>
>>>>> This is on an up-to-date Ubuntu 12.04.2. Ah. I realise I hadn't
>>>>> installed lxc. OK, trying again.
>>>>>
>>>>> Nope. Now the errors are worse (built from a clean tree, same commit):
>>>>>
>>>>> [----------] Global test environment tear-down
>>>>> [==========] 239 tests from 45 test cases ran. (91234 ms total)
>>>>> [ PASSED ] 216 tests.
>>>>> [ FAILED ] 23 tests, listed below:
>>>>> [ FAILED ] SlaveRecoveryTest/1.RecoverSlaveState, where TypeParam =
>>>>> mesos::internal::slave::CgroupsIsolator
>>>>> [ FAILED ] SlaveRecoveryTest/1.RecoverStatusUpdateManager, where
>>>>> TypeParam = mesos::internal::slave::CgroupsIsolator
>>>>> [ FAILED ] SlaveRecoveryTest/1.ReconnectExecutor, where TypeParam =
>>>>> mesos::internal::slave::CgroupsIsolator
>>>>> [ FAILED ] SlaveRecoveryTest/1.RecoverUnregisteredExecutor, where
>>>>> TypeParam = mesos::internal::slave::CgroupsIsolator
>>>>> [ FAILED ] SlaveRecoveryTest/1.RecoverTerminatedExecutor, where
>>>>> TypeParam = mesos::internal::slave::CgroupsIsolator
>>>>> [ FAILED ] SlaveRecoveryTest/1.CleanupExecutor, where TypeParam =
>>>>> mesos::internal::slave::CgroupsIsolator
>>>>> [ FAILED ] SlaveRecoveryTest/1.RemoveNonCheckpointingFramework,
>>>>> where TypeParam = mesos::internal::slave::CgroupsIsolator
>>>>> [ FAILED ] SlaveRecoveryTest/1.NonCheckpointingFramework, where
>>>>> TypeParam = mesos::internal::slave::CgroupsIsolator
>>>>> [ FAILED ] SlaveRecoveryTest/1.NonCheckpointingSlave, where
>>>>> TypeParam = mesos::internal::slave::CgroupsIsolator
>>>>> [ FAILED ] SlaveRecoveryTest/1.KillTask, where TypeParam =
>>>>> mesos::internal::slave::CgroupsIsolator
>>>>> [ FAILED ] SlaveRecoveryTest/1.GCExecutor, where TypeParam =
>>>>> mesos::internal::slave::CgroupsIsolator
>>>>> [ FAILED ] SlaveRecoveryTest/1.ShutdownSlave, where TypeParam =
>>>>> mesos::internal::slave::CgroupsIsolator
>>>>> [ FAILED ] IsolatorTest/1.Usage, where TypeParam =
>>>>> mesos::internal::slave::CgroupsIsolator
>>>>> [ FAILED ] CgroupsIsolatorTest.ROOT_CGROUPS_BalloonFramework
>>>>> [ FAILED ] CgroupsAnyHierarchyWithCpuMemoryTest.ROOT_CGROUPS_Busy
>>>>> [ FAILED ] CgroupsAnyHierarchyWithCpuMemoryTest.ROOT_CGROUPS_SubsystemsHierarchy
>>>>> [ FAILED ] CgroupsAnyHierarchyWithCpuMemoryTest.ROOT_CGROUPS_MountedSubsystems
>>>>> [ FAILED ] CgroupsAnyHierarchyWithCpuMemoryTest.ROOT_CGROUPS_CreateRemove
>>>>> [ FAILED ] CgroupsAnyHierarchyWithCpuMemoryTest.ROOT_CGROUPS_Listen
>>>>> [ FAILED ] CgroupsAnyHierarchyWithCpuAcctMemoryTest.ROOT_CGROUPS_Stat
>>>>> [ FAILED ] CgroupsAnyHierarchyWithCpuMemoryFreezerTest.ROOT_CGROUPS_Freeze
>>>>> [ FAILED ] CgroupsAnyHierarchyWithCpuMemoryFreezerTest.ROOT_CGROUPS_Kill
>>>>> [ FAILED ] CgroupsAnyHierarchyWithCpuMemoryFreezerTest.ROOT_CGROUPS_Destroy
>>>>>
>>>>> 23 FAILED TESTS
>>>>> make[3]: *** [check-local] Error 1
>>>>> make[3]: Leaving directory `/usr/src/mesos/build/src'
>>>>> make[2]: *** [check-am] Error 2
>>>>> make[2]: Leaving directory `/usr/src/mesos/build/src'
>>>>> make[1]: *** [check] Error 2
>>>>> make[1]: Leaving directory `/usr/src/mesos/build/src'
>>>>> make: *** [check-recursive] Error 1
>>>>>
>>>>> Kevin
>>>>>
>>>>> --
>>>>> Kevin Lyda
>>>>> Galway, Ireland
>>>>> US Citizen overseas? We can vote.
>>>>> Register now: http://www.votefromabroad.org/
>>>>
>>>
>>>
>>>
>>> --
>>> Kevin Lyda
>>> Galway, Ireland
>>> US Citizen overseas? We can vote.
>>> Register now: http://www.votefromabroad.org/
>>
>
>
>
> --
> Kevin Lyda
> Galway, Ireland
> US Citizen overseas? We can vote.
> Register now: http://www.votefromabroad.org/
Re: make check failure...
Posted by "Eric W. Biederman" <eb...@xmission.com>.
Benjamin Mahler <be...@gmail.com> writes:
> That test is broken on master currently, the ticket is here:
> MESOS-487<https://issues.apache.org/jira/browse/MESOS-487>
And the fix for the broken test is in:
https://reviews.apache.org/r/13034/
Kevin
Your first run of the tests is current expected until that test is fixed.
With respect to subsequent runs I have seen that before simply with
mounting and unmounting cgroupfs. There are weird races in play and
weird checks going on, and the unit tests exercise the kernel bugs quite
well.
You can look at /proc/cgroups and /proc/<pid>/cgroups to have some idea
of what is going on.
For myself when I do not wind up with unkillable processes or orphan
processes I only had to wait a while. Possibly coupled with echo 3 >
/proc/sys/vm/drop_caches and it was possible to mount cgroup filesystems
again.
I intend to look into these kernel bugs soonish but they aren't exactly
deterministic.
mesos-slave in a running configuration instead of a test configuration
leaves cgroupfs mounted so you are not likely to hit these kernel
problems if you actually start running mesos.
Do becareful about running with a fixed balloon test though. With an
unfixed kernel and system with swap enabled it creates effectively
unkillable processes for me.
If you are a curious you can find more about how the tests are failing
by running them with MESOS_VERBOSE=1 make check.
Eric
Re: make check failure...
Posted by Benjamin Mahler <be...@gmail.com>.
That test is broken on master currently, the ticket is here:
MESOS-487<https://issues.apache.org/jira/browse/MESOS-487>
On Thu, Aug 1, 2013 at 1:18 PM, Kevin Lyda <ke...@ie.suberic.net> wrote:
> Not running systemd. Ubuntu uses upstart.
>
> cgroups are mounted by /etc/init/cgconfig.conf which is installed by
> cgroup-bin (which I need for lscgroup).
>
> I turned that off by doing:
>
> service cgconfig stop
> echo manual > /etc/init/cgconfig.override
>
> The echo line turns it off permanently.
>
> OK, ran make check and am back to the original error! Hurrah!
>
> [----------] Global test environment tear-down
> [==========] 240 tests from 46 test cases ran. (91790 ms total)
> [ PASSED ] 239 tests.
> [ FAILED ] 1 test, listed below:
> [ FAILED ] CgroupsIsolatorTest.ROOT_CGROUPS_BalloonFramework
>
> 1 FAILED TEST
> make[3]: *** [check-local] Error 1
> make[3]: Leaving directory `/usr/src/mesos/build/src'
> make[2]: *** [check-am] Error 2
> make[2]: Leaving directory `/usr/src/mesos/build/src'
> make[1]: *** [check] Error 2
> make[1]: Leaving directory `/usr/src/mesos/build/src'
> make: *** [check-recursive] Error 1
>
> # lscgroup
> cgroups can't be listed: Cgroup is not mounted
>
> # mount
> /dev/sda1 on / type ext4 (rw,errors=remount-ro)
> proc on /proc type proc (rw,noexec,nosuid,nodev)
> sysfs on /sys type sysfs (rw,noexec,nosuid,nodev)
> none on /sys/fs/fuse/connections type fusectl (rw)
> none on /sys/kernel/debug type debugfs (rw)
> none on /sys/kernel/security type securityfs (rw)
> udev on /dev type devtmpfs (rw,mode=0755)
> devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620)
> tmpfs on /run type tmpfs (rw,noexec,nosuid,size=10%,mode=0755)
> none on /run/lock type tmpfs (rw,noexec,nosuid,nodev,size=5242880)
> none on /run/shm type tmpfs (rw,nosuid,nodev)
> /dev/sdd1 on /u4 type ext4 (rw)
> /dev/sdb1 on /u2 type ext4 (rw)
> /dev/sdc1 on /u3 type ext4 (rw)
> /dev/sda6 on /u1 type ext4 (rw)
>
>
> On Thu, Aug 1, 2013 at 8:25 PM, Ian Downes <id...@twitter.com> wrote:
> > Looks like cgroups are mounted on /sys/fs/cgroup so I'd expect the tests
> to fail again. Are you running systemd? If the tests do fail, and you're
> not running systemd, you should try manually umount'ing cgroups and
> re-running the tests (no need to rebuild, just run `make check` again).
> >
> > Ian
> > On Aug 1, 2013, at 12:18 PM, Kevin Lyda <ke...@ie.suberic.net> wrote:
> >
> >> [Initial reply not sent to list]
> >>
> >> Will remove lxc and make sure cgroups are unmounted. This is not in a
> >> VM; it's on an x86_64 machine. Replying via phone so will get the
> >> other info in a bit.
> >>
> >> [Followup]
> >>
> >> Right, removed lxc. Noted that there's a cgroups-lite package that
> >> might be an issue. Hm. No, that mounts cgroups.
> >>
> >> OK, lscgroup is in the cgroup-bin package. That also mounts cgroups.
> >> But perhaps it isn't busy w/o lxc installed?
> >>
> >> Trying a build now. Output of requested commands follows; will report
> >> build results when they're done.
> >>
> >> # lscgroup
> >> cpu:/
> >> cpu:/sysdefault
> >> cpuacct:/
> >> cpuacct:/sysdefault
> >> devices:/
> >> devices:/sysdefault
> >> memory:/
> >> memory:/sysdefault
> >> freezer:/
> >> freezer:/sysdefault
> >> # uname -a
> >> Linux node29 3.5.0-23-generic #35~precise1-Ubuntu SMP Fri Jan 25
> >> 17:13:26 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
> >> # mount
> >> /dev/sda1 on / type ext4 (rw,errors=remount-ro)
> >> proc on /proc type proc (rw,noexec,nosuid,nodev)
> >> sysfs on /sys type sysfs (rw,noexec,nosuid,nodev)
> >> none on /sys/fs/fuse/connections type fusectl (rw)
> >> none on /sys/kernel/debug type debugfs (rw)
> >> none on /sys/kernel/security type securityfs (rw)
> >> udev on /dev type devtmpfs (rw,mode=0755)
> >> devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620)
> >> tmpfs on /run type tmpfs (rw,noexec,nosuid,size=10%,mode=0755)
> >> none on /run/lock type tmpfs (rw,noexec,nosuid,nodev,size=5242880)
> >> none on /run/shm type tmpfs (rw,nosuid,nodev)
> >> /dev/sdd1 on /u4 type ext4 (rw)
> >> /dev/sdb1 on /u2 type ext4 (rw)
> >> /dev/sdc1 on /u3 type ext4 (rw)
> >> /dev/sda6 on /u1 type ext4 (rw)
> >> cgroups on /sys/fs/cgroup type tmpfs (rw,uid=0,gid=0,mode=0755)
> >>
> >> Thanks for the feedback,
> >>
> >> Kevin
> >>
> >>
> >> On Thu, Aug 1, 2013 at 7:43 PM, Ian Downes <id...@twitter.com> wrote:
> >>> The documentation is out of date and lxc has been deprecated; cgroups
> are used directly instead. We'll be updating the documentation shortly to
> reflect this.
> >>>
> >>> Installing lxc possibly mounted cgroups and led to the slew of busy
> errors. Could you please post the output of `mount` and, if you have
> libcgroup installed, `lscgroup`, ideally before and after lxc is installed?
> And also `uname -a`.
> >>>
> >>> The failed BalloonFramework test may be related to a bug that we're
> investigating. Are you running this inside a VM by any chance?
> >>>
> >>> Ian
> >>>
> >>> On Aug 1, 2013, at 8:59 AM, Kevin Lyda <ke...@ie.suberic.net> wrote:
> >>>
> >>>> Running make check on master from github (bfdfdce) and I'm getting a
> failure:
> >>>>
> >>>> [ OK ] AllocatorZooKeeperTest/0.SlaveReregistersFirst (2098 ms)
> >>>> [----------] 2 tests from AllocatorZooKeeperTest/0 (4242 ms total)
> >>>>
> >>>> [----------] Global test environment tear-down
> >>>> [==========] 240 tests from 46 test cases ran. (91604 ms total)
> >>>> [ PASSED ] 239 tests.
> >>>> [ FAILED ] 1 test, listed below:
> >>>> [ FAILED ] CgroupsIsolatorTest.ROOT_CGROUPS_BalloonFramework
> >>>>
> >>>> 1 FAILED TEST
> >>>> make[3]: *** [check-local] Error 1
> >>>> make[3]: Leaving directory `/usr/src/mesos/build/src'
> >>>> make[2]: *** [check-am] Error 2
> >>>> make[2]: Leaving directory `/usr/src/mesos/build/src'
> >>>> make[1]: *** [check] Error 2
> >>>> make[1]: Leaving directory `/usr/src/mesos/build/src'
> >>>> make: *** [check-recursive] Error 1
> >>>>
> >>>> This is on an up-to-date Ubuntu 12.04.2. Ah. I realise I hadn't
> >>>> installed lxc. OK, trying again.
> >>>>
> >>>> Nope. Now the errors are worse (built from a clean tree, same commit):
> >>>>
> >>>> [----------] Global test environment tear-down
> >>>> [==========] 239 tests from 45 test cases ran. (91234 ms total)
> >>>> [ PASSED ] 216 tests.
> >>>> [ FAILED ] 23 tests, listed below:
> >>>> [ FAILED ] SlaveRecoveryTest/1.RecoverSlaveState, where TypeParam =
> >>>> mesos::internal::slave::CgroupsIsolator
> >>>> [ FAILED ] SlaveRecoveryTest/1.RecoverStatusUpdateManager, where
> >>>> TypeParam = mesos::internal::slave::CgroupsIsolator
> >>>> [ FAILED ] SlaveRecoveryTest/1.ReconnectExecutor, where TypeParam =
> >>>> mesos::internal::slave::CgroupsIsolator
> >>>> [ FAILED ] SlaveRecoveryTest/1.RecoverUnregisteredExecutor, where
> >>>> TypeParam = mesos::internal::slave::CgroupsIsolator
> >>>> [ FAILED ] SlaveRecoveryTest/1.RecoverTerminatedExecutor, where
> >>>> TypeParam = mesos::internal::slave::CgroupsIsolator
> >>>> [ FAILED ] SlaveRecoveryTest/1.CleanupExecutor, where TypeParam =
> >>>> mesos::internal::slave::CgroupsIsolator
> >>>> [ FAILED ] SlaveRecoveryTest/1.RemoveNonCheckpointingFramework,
> >>>> where TypeParam = mesos::internal::slave::CgroupsIsolator
> >>>> [ FAILED ] SlaveRecoveryTest/1.NonCheckpointingFramework, where
> >>>> TypeParam = mesos::internal::slave::CgroupsIsolator
> >>>> [ FAILED ] SlaveRecoveryTest/1.NonCheckpointingSlave, where
> >>>> TypeParam = mesos::internal::slave::CgroupsIsolator
> >>>> [ FAILED ] SlaveRecoveryTest/1.KillTask, where TypeParam =
> >>>> mesos::internal::slave::CgroupsIsolator
> >>>> [ FAILED ] SlaveRecoveryTest/1.GCExecutor, where TypeParam =
> >>>> mesos::internal::slave::CgroupsIsolator
> >>>> [ FAILED ] SlaveRecoveryTest/1.ShutdownSlave, where TypeParam =
> >>>> mesos::internal::slave::CgroupsIsolator
> >>>> [ FAILED ] IsolatorTest/1.Usage, where TypeParam =
> >>>> mesos::internal::slave::CgroupsIsolator
> >>>> [ FAILED ] CgroupsIsolatorTest.ROOT_CGROUPS_BalloonFramework
> >>>> [ FAILED ] CgroupsAnyHierarchyWithCpuMemoryTest.ROOT_CGROUPS_Busy
> >>>> [ FAILED ]
> CgroupsAnyHierarchyWithCpuMemoryTest.ROOT_CGROUPS_SubsystemsHierarchy
> >>>> [ FAILED ]
> CgroupsAnyHierarchyWithCpuMemoryTest.ROOT_CGROUPS_MountedSubsystems
> >>>> [ FAILED ]
> CgroupsAnyHierarchyWithCpuMemoryTest.ROOT_CGROUPS_CreateRemove
> >>>> [ FAILED ] CgroupsAnyHierarchyWithCpuMemoryTest.ROOT_CGROUPS_Listen
> >>>> [ FAILED ]
> CgroupsAnyHierarchyWithCpuAcctMemoryTest.ROOT_CGROUPS_Stat
> >>>> [ FAILED ]
> CgroupsAnyHierarchyWithCpuMemoryFreezerTest.ROOT_CGROUPS_Freeze
> >>>> [ FAILED ]
> CgroupsAnyHierarchyWithCpuMemoryFreezerTest.ROOT_CGROUPS_Kill
> >>>> [ FAILED ]
> CgroupsAnyHierarchyWithCpuMemoryFreezerTest.ROOT_CGROUPS_Destroy
> >>>>
> >>>> 23 FAILED TESTS
> >>>> make[3]: *** [check-local] Error 1
> >>>> make[3]: Leaving directory `/usr/src/mesos/build/src'
> >>>> make[2]: *** [check-am] Error 2
> >>>> make[2]: Leaving directory `/usr/src/mesos/build/src'
> >>>> make[1]: *** [check] Error 2
> >>>> make[1]: Leaving directory `/usr/src/mesos/build/src'
> >>>> make: *** [check-recursive] Error 1
> >>>>
> >>>> Kevin
> >>>>
> >>>> --
> >>>> Kevin Lyda
> >>>> Galway, Ireland
> >>>> US Citizen overseas? We can vote.
> >>>> Register now: http://www.votefromabroad.org/
> >>>
> >>
> >>
> >>
> >> --
> >> Kevin Lyda
> >> Galway, Ireland
> >> US Citizen overseas? We can vote.
> >> Register now: http://www.votefromabroad.org/
> >
>
>
>
> --
> Kevin Lyda
> Galway, Ireland
> US Citizen overseas? We can vote.
> Register now: http://www.votefromabroad.org/
>
Re: make check failure...
Posted by Kevin Lyda <ke...@ie.suberic.net>.
Not running systemd. Ubuntu uses upstart.
cgroups are mounted by /etc/init/cgconfig.conf which is installed by
cgroup-bin (which I need for lscgroup).
I turned that off by doing:
service cgconfig stop
echo manual > /etc/init/cgconfig.override
The echo line turns it off permanently.
OK, ran make check and am back to the original error! Hurrah!
[----------] Global test environment tear-down
[==========] 240 tests from 46 test cases ran. (91790 ms total)
[ PASSED ] 239 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] CgroupsIsolatorTest.ROOT_CGROUPS_BalloonFramework
1 FAILED TEST
make[3]: *** [check-local] Error 1
make[3]: Leaving directory `/usr/src/mesos/build/src'
make[2]: *** [check-am] Error 2
make[2]: Leaving directory `/usr/src/mesos/build/src'
make[1]: *** [check] Error 2
make[1]: Leaving directory `/usr/src/mesos/build/src'
make: *** [check-recursive] Error 1
# lscgroup
cgroups can't be listed: Cgroup is not mounted
# mount
/dev/sda1 on / type ext4 (rw,errors=remount-ro)
proc on /proc type proc (rw,noexec,nosuid,nodev)
sysfs on /sys type sysfs (rw,noexec,nosuid,nodev)
none on /sys/fs/fuse/connections type fusectl (rw)
none on /sys/kernel/debug type debugfs (rw)
none on /sys/kernel/security type securityfs (rw)
udev on /dev type devtmpfs (rw,mode=0755)
devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620)
tmpfs on /run type tmpfs (rw,noexec,nosuid,size=10%,mode=0755)
none on /run/lock type tmpfs (rw,noexec,nosuid,nodev,size=5242880)
none on /run/shm type tmpfs (rw,nosuid,nodev)
/dev/sdd1 on /u4 type ext4 (rw)
/dev/sdb1 on /u2 type ext4 (rw)
/dev/sdc1 on /u3 type ext4 (rw)
/dev/sda6 on /u1 type ext4 (rw)
On Thu, Aug 1, 2013 at 8:25 PM, Ian Downes <id...@twitter.com> wrote:
> Looks like cgroups are mounted on /sys/fs/cgroup so I'd expect the tests to fail again. Are you running systemd? If the tests do fail, and you're not running systemd, you should try manually umount'ing cgroups and re-running the tests (no need to rebuild, just run `make check` again).
>
> Ian
> On Aug 1, 2013, at 12:18 PM, Kevin Lyda <ke...@ie.suberic.net> wrote:
>
>> [Initial reply not sent to list]
>>
>> Will remove lxc and make sure cgroups are unmounted. This is not in a
>> VM; it's on an x86_64 machine. Replying via phone so will get the
>> other info in a bit.
>>
>> [Followup]
>>
>> Right, removed lxc. Noted that there's a cgroups-lite package that
>> might be an issue. Hm. No, that mounts cgroups.
>>
>> OK, lscgroup is in the cgroup-bin package. That also mounts cgroups.
>> But perhaps it isn't busy w/o lxc installed?
>>
>> Trying a build now. Output of requested commands follows; will report
>> build results when they're done.
>>
>> # lscgroup
>> cpu:/
>> cpu:/sysdefault
>> cpuacct:/
>> cpuacct:/sysdefault
>> devices:/
>> devices:/sysdefault
>> memory:/
>> memory:/sysdefault
>> freezer:/
>> freezer:/sysdefault
>> # uname -a
>> Linux node29 3.5.0-23-generic #35~precise1-Ubuntu SMP Fri Jan 25
>> 17:13:26 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
>> # mount
>> /dev/sda1 on / type ext4 (rw,errors=remount-ro)
>> proc on /proc type proc (rw,noexec,nosuid,nodev)
>> sysfs on /sys type sysfs (rw,noexec,nosuid,nodev)
>> none on /sys/fs/fuse/connections type fusectl (rw)
>> none on /sys/kernel/debug type debugfs (rw)
>> none on /sys/kernel/security type securityfs (rw)
>> udev on /dev type devtmpfs (rw,mode=0755)
>> devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620)
>> tmpfs on /run type tmpfs (rw,noexec,nosuid,size=10%,mode=0755)
>> none on /run/lock type tmpfs (rw,noexec,nosuid,nodev,size=5242880)
>> none on /run/shm type tmpfs (rw,nosuid,nodev)
>> /dev/sdd1 on /u4 type ext4 (rw)
>> /dev/sdb1 on /u2 type ext4 (rw)
>> /dev/sdc1 on /u3 type ext4 (rw)
>> /dev/sda6 on /u1 type ext4 (rw)
>> cgroups on /sys/fs/cgroup type tmpfs (rw,uid=0,gid=0,mode=0755)
>>
>> Thanks for the feedback,
>>
>> Kevin
>>
>>
>> On Thu, Aug 1, 2013 at 7:43 PM, Ian Downes <id...@twitter.com> wrote:
>>> The documentation is out of date and lxc has been deprecated; cgroups are used directly instead. We'll be updating the documentation shortly to reflect this.
>>>
>>> Installing lxc possibly mounted cgroups and led to the slew of busy errors. Could you please post the output of `mount` and, if you have libcgroup installed, `lscgroup`, ideally before and after lxc is installed? And also `uname -a`.
>>>
>>> The failed BalloonFramework test may be related to a bug that we're investigating. Are you running this inside a VM by any chance?
>>>
>>> Ian
>>>
>>> On Aug 1, 2013, at 8:59 AM, Kevin Lyda <ke...@ie.suberic.net> wrote:
>>>
>>>> Running make check on master from github (bfdfdce) and I'm getting a failure:
>>>>
>>>> [ OK ] AllocatorZooKeeperTest/0.SlaveReregistersFirst (2098 ms)
>>>> [----------] 2 tests from AllocatorZooKeeperTest/0 (4242 ms total)
>>>>
>>>> [----------] Global test environment tear-down
>>>> [==========] 240 tests from 46 test cases ran. (91604 ms total)
>>>> [ PASSED ] 239 tests.
>>>> [ FAILED ] 1 test, listed below:
>>>> [ FAILED ] CgroupsIsolatorTest.ROOT_CGROUPS_BalloonFramework
>>>>
>>>> 1 FAILED TEST
>>>> make[3]: *** [check-local] Error 1
>>>> make[3]: Leaving directory `/usr/src/mesos/build/src'
>>>> make[2]: *** [check-am] Error 2
>>>> make[2]: Leaving directory `/usr/src/mesos/build/src'
>>>> make[1]: *** [check] Error 2
>>>> make[1]: Leaving directory `/usr/src/mesos/build/src'
>>>> make: *** [check-recursive] Error 1
>>>>
>>>> This is on an up-to-date Ubuntu 12.04.2. Ah. I realise I hadn't
>>>> installed lxc. OK, trying again.
>>>>
>>>> Nope. Now the errors are worse (built from a clean tree, same commit):
>>>>
>>>> [----------] Global test environment tear-down
>>>> [==========] 239 tests from 45 test cases ran. (91234 ms total)
>>>> [ PASSED ] 216 tests.
>>>> [ FAILED ] 23 tests, listed below:
>>>> [ FAILED ] SlaveRecoveryTest/1.RecoverSlaveState, where TypeParam =
>>>> mesos::internal::slave::CgroupsIsolator
>>>> [ FAILED ] SlaveRecoveryTest/1.RecoverStatusUpdateManager, where
>>>> TypeParam = mesos::internal::slave::CgroupsIsolator
>>>> [ FAILED ] SlaveRecoveryTest/1.ReconnectExecutor, where TypeParam =
>>>> mesos::internal::slave::CgroupsIsolator
>>>> [ FAILED ] SlaveRecoveryTest/1.RecoverUnregisteredExecutor, where
>>>> TypeParam = mesos::internal::slave::CgroupsIsolator
>>>> [ FAILED ] SlaveRecoveryTest/1.RecoverTerminatedExecutor, where
>>>> TypeParam = mesos::internal::slave::CgroupsIsolator
>>>> [ FAILED ] SlaveRecoveryTest/1.CleanupExecutor, where TypeParam =
>>>> mesos::internal::slave::CgroupsIsolator
>>>> [ FAILED ] SlaveRecoveryTest/1.RemoveNonCheckpointingFramework,
>>>> where TypeParam = mesos::internal::slave::CgroupsIsolator
>>>> [ FAILED ] SlaveRecoveryTest/1.NonCheckpointingFramework, where
>>>> TypeParam = mesos::internal::slave::CgroupsIsolator
>>>> [ FAILED ] SlaveRecoveryTest/1.NonCheckpointingSlave, where
>>>> TypeParam = mesos::internal::slave::CgroupsIsolator
>>>> [ FAILED ] SlaveRecoveryTest/1.KillTask, where TypeParam =
>>>> mesos::internal::slave::CgroupsIsolator
>>>> [ FAILED ] SlaveRecoveryTest/1.GCExecutor, where TypeParam =
>>>> mesos::internal::slave::CgroupsIsolator
>>>> [ FAILED ] SlaveRecoveryTest/1.ShutdownSlave, where TypeParam =
>>>> mesos::internal::slave::CgroupsIsolator
>>>> [ FAILED ] IsolatorTest/1.Usage, where TypeParam =
>>>> mesos::internal::slave::CgroupsIsolator
>>>> [ FAILED ] CgroupsIsolatorTest.ROOT_CGROUPS_BalloonFramework
>>>> [ FAILED ] CgroupsAnyHierarchyWithCpuMemoryTest.ROOT_CGROUPS_Busy
>>>> [ FAILED ] CgroupsAnyHierarchyWithCpuMemoryTest.ROOT_CGROUPS_SubsystemsHierarchy
>>>> [ FAILED ] CgroupsAnyHierarchyWithCpuMemoryTest.ROOT_CGROUPS_MountedSubsystems
>>>> [ FAILED ] CgroupsAnyHierarchyWithCpuMemoryTest.ROOT_CGROUPS_CreateRemove
>>>> [ FAILED ] CgroupsAnyHierarchyWithCpuMemoryTest.ROOT_CGROUPS_Listen
>>>> [ FAILED ] CgroupsAnyHierarchyWithCpuAcctMemoryTest.ROOT_CGROUPS_Stat
>>>> [ FAILED ] CgroupsAnyHierarchyWithCpuMemoryFreezerTest.ROOT_CGROUPS_Freeze
>>>> [ FAILED ] CgroupsAnyHierarchyWithCpuMemoryFreezerTest.ROOT_CGROUPS_Kill
>>>> [ FAILED ] CgroupsAnyHierarchyWithCpuMemoryFreezerTest.ROOT_CGROUPS_Destroy
>>>>
>>>> 23 FAILED TESTS
>>>> make[3]: *** [check-local] Error 1
>>>> make[3]: Leaving directory `/usr/src/mesos/build/src'
>>>> make[2]: *** [check-am] Error 2
>>>> make[2]: Leaving directory `/usr/src/mesos/build/src'
>>>> make[1]: *** [check] Error 2
>>>> make[1]: Leaving directory `/usr/src/mesos/build/src'
>>>> make: *** [check-recursive] Error 1
>>>>
>>>> Kevin
>>>>
>>>> --
>>>> Kevin Lyda
>>>> Galway, Ireland
>>>> US Citizen overseas? We can vote.
>>>> Register now: http://www.votefromabroad.org/
>>>
>>
>>
>>
>> --
>> Kevin Lyda
>> Galway, Ireland
>> US Citizen overseas? We can vote.
>> Register now: http://www.votefromabroad.org/
>
--
Kevin Lyda
Galway, Ireland
US Citizen overseas? We can vote.
Register now: http://www.votefromabroad.org/
Re: make check failure...
Posted by Ian Downes <id...@twitter.com>.
Looks like cgroups are mounted on /sys/fs/cgroup so I'd expect the tests to fail again. Are you running systemd? If the tests do fail, and you're not running systemd, you should try manually umount'ing cgroups and re-running the tests (no need to rebuild, just run `make check` again).
Ian
On Aug 1, 2013, at 12:18 PM, Kevin Lyda <ke...@ie.suberic.net> wrote:
> [Initial reply not sent to list]
>
> Will remove lxc and make sure cgroups are unmounted. This is not in a
> VM; it's on an x86_64 machine. Replying via phone so will get the
> other info in a bit.
>
> [Followup]
>
> Right, removed lxc. Noted that there's a cgroups-lite package that
> might be an issue. Hm. No, that mounts cgroups.
>
> OK, lscgroup is in the cgroup-bin package. That also mounts cgroups.
> But perhaps it isn't busy w/o lxc installed?
>
> Trying a build now. Output of requested commands follows; will report
> build results when they're done.
>
> # lscgroup
> cpu:/
> cpu:/sysdefault
> cpuacct:/
> cpuacct:/sysdefault
> devices:/
> devices:/sysdefault
> memory:/
> memory:/sysdefault
> freezer:/
> freezer:/sysdefault
> # uname -a
> Linux node29 3.5.0-23-generic #35~precise1-Ubuntu SMP Fri Jan 25
> 17:13:26 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
> # mount
> /dev/sda1 on / type ext4 (rw,errors=remount-ro)
> proc on /proc type proc (rw,noexec,nosuid,nodev)
> sysfs on /sys type sysfs (rw,noexec,nosuid,nodev)
> none on /sys/fs/fuse/connections type fusectl (rw)
> none on /sys/kernel/debug type debugfs (rw)
> none on /sys/kernel/security type securityfs (rw)
> udev on /dev type devtmpfs (rw,mode=0755)
> devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620)
> tmpfs on /run type tmpfs (rw,noexec,nosuid,size=10%,mode=0755)
> none on /run/lock type tmpfs (rw,noexec,nosuid,nodev,size=5242880)
> none on /run/shm type tmpfs (rw,nosuid,nodev)
> /dev/sdd1 on /u4 type ext4 (rw)
> /dev/sdb1 on /u2 type ext4 (rw)
> /dev/sdc1 on /u3 type ext4 (rw)
> /dev/sda6 on /u1 type ext4 (rw)
> cgroups on /sys/fs/cgroup type tmpfs (rw,uid=0,gid=0,mode=0755)
>
> Thanks for the feedback,
>
> Kevin
>
>
> On Thu, Aug 1, 2013 at 7:43 PM, Ian Downes <id...@twitter.com> wrote:
>> The documentation is out of date and lxc has been deprecated; cgroups are used directly instead. We'll be updating the documentation shortly to reflect this.
>>
>> Installing lxc possibly mounted cgroups and led to the slew of busy errors. Could you please post the output of `mount` and, if you have libcgroup installed, `lscgroup`, ideally before and after lxc is installed? And also `uname -a`.
>>
>> The failed BalloonFramework test may be related to a bug that we're investigating. Are you running this inside a VM by any chance?
>>
>> Ian
>>
>> On Aug 1, 2013, at 8:59 AM, Kevin Lyda <ke...@ie.suberic.net> wrote:
>>
>>> Running make check on master from github (bfdfdce) and I'm getting a failure:
>>>
>>> [ OK ] AllocatorZooKeeperTest/0.SlaveReregistersFirst (2098 ms)
>>> [----------] 2 tests from AllocatorZooKeeperTest/0 (4242 ms total)
>>>
>>> [----------] Global test environment tear-down
>>> [==========] 240 tests from 46 test cases ran. (91604 ms total)
>>> [ PASSED ] 239 tests.
>>> [ FAILED ] 1 test, listed below:
>>> [ FAILED ] CgroupsIsolatorTest.ROOT_CGROUPS_BalloonFramework
>>>
>>> 1 FAILED TEST
>>> make[3]: *** [check-local] Error 1
>>> make[3]: Leaving directory `/usr/src/mesos/build/src'
>>> make[2]: *** [check-am] Error 2
>>> make[2]: Leaving directory `/usr/src/mesos/build/src'
>>> make[1]: *** [check] Error 2
>>> make[1]: Leaving directory `/usr/src/mesos/build/src'
>>> make: *** [check-recursive] Error 1
>>>
>>> This is on an up-to-date Ubuntu 12.04.2. Ah. I realise I hadn't
>>> installed lxc. OK, trying again.
>>>
>>> Nope. Now the errors are worse (built from a clean tree, same commit):
>>>
>>> [----------] Global test environment tear-down
>>> [==========] 239 tests from 45 test cases ran. (91234 ms total)
>>> [ PASSED ] 216 tests.
>>> [ FAILED ] 23 tests, listed below:
>>> [ FAILED ] SlaveRecoveryTest/1.RecoverSlaveState, where TypeParam =
>>> mesos::internal::slave::CgroupsIsolator
>>> [ FAILED ] SlaveRecoveryTest/1.RecoverStatusUpdateManager, where
>>> TypeParam = mesos::internal::slave::CgroupsIsolator
>>> [ FAILED ] SlaveRecoveryTest/1.ReconnectExecutor, where TypeParam =
>>> mesos::internal::slave::CgroupsIsolator
>>> [ FAILED ] SlaveRecoveryTest/1.RecoverUnregisteredExecutor, where
>>> TypeParam = mesos::internal::slave::CgroupsIsolator
>>> [ FAILED ] SlaveRecoveryTest/1.RecoverTerminatedExecutor, where
>>> TypeParam = mesos::internal::slave::CgroupsIsolator
>>> [ FAILED ] SlaveRecoveryTest/1.CleanupExecutor, where TypeParam =
>>> mesos::internal::slave::CgroupsIsolator
>>> [ FAILED ] SlaveRecoveryTest/1.RemoveNonCheckpointingFramework,
>>> where TypeParam = mesos::internal::slave::CgroupsIsolator
>>> [ FAILED ] SlaveRecoveryTest/1.NonCheckpointingFramework, where
>>> TypeParam = mesos::internal::slave::CgroupsIsolator
>>> [ FAILED ] SlaveRecoveryTest/1.NonCheckpointingSlave, where
>>> TypeParam = mesos::internal::slave::CgroupsIsolator
>>> [ FAILED ] SlaveRecoveryTest/1.KillTask, where TypeParam =
>>> mesos::internal::slave::CgroupsIsolator
>>> [ FAILED ] SlaveRecoveryTest/1.GCExecutor, where TypeParam =
>>> mesos::internal::slave::CgroupsIsolator
>>> [ FAILED ] SlaveRecoveryTest/1.ShutdownSlave, where TypeParam =
>>> mesos::internal::slave::CgroupsIsolator
>>> [ FAILED ] IsolatorTest/1.Usage, where TypeParam =
>>> mesos::internal::slave::CgroupsIsolator
>>> [ FAILED ] CgroupsIsolatorTest.ROOT_CGROUPS_BalloonFramework
>>> [ FAILED ] CgroupsAnyHierarchyWithCpuMemoryTest.ROOT_CGROUPS_Busy
>>> [ FAILED ] CgroupsAnyHierarchyWithCpuMemoryTest.ROOT_CGROUPS_SubsystemsHierarchy
>>> [ FAILED ] CgroupsAnyHierarchyWithCpuMemoryTest.ROOT_CGROUPS_MountedSubsystems
>>> [ FAILED ] CgroupsAnyHierarchyWithCpuMemoryTest.ROOT_CGROUPS_CreateRemove
>>> [ FAILED ] CgroupsAnyHierarchyWithCpuMemoryTest.ROOT_CGROUPS_Listen
>>> [ FAILED ] CgroupsAnyHierarchyWithCpuAcctMemoryTest.ROOT_CGROUPS_Stat
>>> [ FAILED ] CgroupsAnyHierarchyWithCpuMemoryFreezerTest.ROOT_CGROUPS_Freeze
>>> [ FAILED ] CgroupsAnyHierarchyWithCpuMemoryFreezerTest.ROOT_CGROUPS_Kill
>>> [ FAILED ] CgroupsAnyHierarchyWithCpuMemoryFreezerTest.ROOT_CGROUPS_Destroy
>>>
>>> 23 FAILED TESTS
>>> make[3]: *** [check-local] Error 1
>>> make[3]: Leaving directory `/usr/src/mesos/build/src'
>>> make[2]: *** [check-am] Error 2
>>> make[2]: Leaving directory `/usr/src/mesos/build/src'
>>> make[1]: *** [check] Error 2
>>> make[1]: Leaving directory `/usr/src/mesos/build/src'
>>> make: *** [check-recursive] Error 1
>>>
>>> Kevin
>>>
>>> --
>>> Kevin Lyda
>>> Galway, Ireland
>>> US Citizen overseas? We can vote.
>>> Register now: http://www.votefromabroad.org/
>>
>
>
>
> --
> Kevin Lyda
> Galway, Ireland
> US Citizen overseas? We can vote.
> Register now: http://www.votefromabroad.org/
Re: make check failure...
Posted by Kevin Lyda <ke...@ie.suberic.net>.
[Initial reply not sent to list]
Will remove lxc and make sure cgroups are unmounted. This is not in a
VM; it's on an x86_64 machine. Replying via phone so will get the
other info in a bit.
[Followup]
Right, removed lxc. Noted that there's a cgroups-lite package that
might be an issue. Hm. No, that mounts cgroups.
OK, lscgroup is in the cgroup-bin package. That also mounts cgroups.
But perhaps it isn't busy w/o lxc installed?
Trying a build now. Output of requested commands follows; will report
build results when they're done.
# lscgroup
cpu:/
cpu:/sysdefault
cpuacct:/
cpuacct:/sysdefault
devices:/
devices:/sysdefault
memory:/
memory:/sysdefault
freezer:/
freezer:/sysdefault
# uname -a
Linux node29 3.5.0-23-generic #35~precise1-Ubuntu SMP Fri Jan 25
17:13:26 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
# mount
/dev/sda1 on / type ext4 (rw,errors=remount-ro)
proc on /proc type proc (rw,noexec,nosuid,nodev)
sysfs on /sys type sysfs (rw,noexec,nosuid,nodev)
none on /sys/fs/fuse/connections type fusectl (rw)
none on /sys/kernel/debug type debugfs (rw)
none on /sys/kernel/security type securityfs (rw)
udev on /dev type devtmpfs (rw,mode=0755)
devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620)
tmpfs on /run type tmpfs (rw,noexec,nosuid,size=10%,mode=0755)
none on /run/lock type tmpfs (rw,noexec,nosuid,nodev,size=5242880)
none on /run/shm type tmpfs (rw,nosuid,nodev)
/dev/sdd1 on /u4 type ext4 (rw)
/dev/sdb1 on /u2 type ext4 (rw)
/dev/sdc1 on /u3 type ext4 (rw)
/dev/sda6 on /u1 type ext4 (rw)
cgroups on /sys/fs/cgroup type tmpfs (rw,uid=0,gid=0,mode=0755)
Thanks for the feedback,
Kevin
On Thu, Aug 1, 2013 at 7:43 PM, Ian Downes <id...@twitter.com> wrote:
> The documentation is out of date and lxc has been deprecated; cgroups are used directly instead. We'll be updating the documentation shortly to reflect this.
>
> Installing lxc possibly mounted cgroups and led to the slew of busy errors. Could you please post the output of `mount` and, if you have libcgroup installed, `lscgroup`, ideally before and after lxc is installed? And also `uname -a`.
>
> The failed BalloonFramework test may be related to a bug that we're investigating. Are you running this inside a VM by any chance?
>
> Ian
>
> On Aug 1, 2013, at 8:59 AM, Kevin Lyda <ke...@ie.suberic.net> wrote:
>
>> Running make check on master from github (bfdfdce) and I'm getting a failure:
>>
>> [ OK ] AllocatorZooKeeperTest/0.SlaveReregistersFirst (2098 ms)
>> [----------] 2 tests from AllocatorZooKeeperTest/0 (4242 ms total)
>>
>> [----------] Global test environment tear-down
>> [==========] 240 tests from 46 test cases ran. (91604 ms total)
>> [ PASSED ] 239 tests.
>> [ FAILED ] 1 test, listed below:
>> [ FAILED ] CgroupsIsolatorTest.ROOT_CGROUPS_BalloonFramework
>>
>> 1 FAILED TEST
>> make[3]: *** [check-local] Error 1
>> make[3]: Leaving directory `/usr/src/mesos/build/src'
>> make[2]: *** [check-am] Error 2
>> make[2]: Leaving directory `/usr/src/mesos/build/src'
>> make[1]: *** [check] Error 2
>> make[1]: Leaving directory `/usr/src/mesos/build/src'
>> make: *** [check-recursive] Error 1
>>
>> This is on an up-to-date Ubuntu 12.04.2. Ah. I realise I hadn't
>> installed lxc. OK, trying again.
>>
>> Nope. Now the errors are worse (built from a clean tree, same commit):
>>
>> [----------] Global test environment tear-down
>> [==========] 239 tests from 45 test cases ran. (91234 ms total)
>> [ PASSED ] 216 tests.
>> [ FAILED ] 23 tests, listed below:
>> [ FAILED ] SlaveRecoveryTest/1.RecoverSlaveState, where TypeParam =
>> mesos::internal::slave::CgroupsIsolator
>> [ FAILED ] SlaveRecoveryTest/1.RecoverStatusUpdateManager, where
>> TypeParam = mesos::internal::slave::CgroupsIsolator
>> [ FAILED ] SlaveRecoveryTest/1.ReconnectExecutor, where TypeParam =
>> mesos::internal::slave::CgroupsIsolator
>> [ FAILED ] SlaveRecoveryTest/1.RecoverUnregisteredExecutor, where
>> TypeParam = mesos::internal::slave::CgroupsIsolator
>> [ FAILED ] SlaveRecoveryTest/1.RecoverTerminatedExecutor, where
>> TypeParam = mesos::internal::slave::CgroupsIsolator
>> [ FAILED ] SlaveRecoveryTest/1.CleanupExecutor, where TypeParam =
>> mesos::internal::slave::CgroupsIsolator
>> [ FAILED ] SlaveRecoveryTest/1.RemoveNonCheckpointingFramework,
>> where TypeParam = mesos::internal::slave::CgroupsIsolator
>> [ FAILED ] SlaveRecoveryTest/1.NonCheckpointingFramework, where
>> TypeParam = mesos::internal::slave::CgroupsIsolator
>> [ FAILED ] SlaveRecoveryTest/1.NonCheckpointingSlave, where
>> TypeParam = mesos::internal::slave::CgroupsIsolator
>> [ FAILED ] SlaveRecoveryTest/1.KillTask, where TypeParam =
>> mesos::internal::slave::CgroupsIsolator
>> [ FAILED ] SlaveRecoveryTest/1.GCExecutor, where TypeParam =
>> mesos::internal::slave::CgroupsIsolator
>> [ FAILED ] SlaveRecoveryTest/1.ShutdownSlave, where TypeParam =
>> mesos::internal::slave::CgroupsIsolator
>> [ FAILED ] IsolatorTest/1.Usage, where TypeParam =
>> mesos::internal::slave::CgroupsIsolator
>> [ FAILED ] CgroupsIsolatorTest.ROOT_CGROUPS_BalloonFramework
>> [ FAILED ] CgroupsAnyHierarchyWithCpuMemoryTest.ROOT_CGROUPS_Busy
>> [ FAILED ] CgroupsAnyHierarchyWithCpuMemoryTest.ROOT_CGROUPS_SubsystemsHierarchy
>> [ FAILED ] CgroupsAnyHierarchyWithCpuMemoryTest.ROOT_CGROUPS_MountedSubsystems
>> [ FAILED ] CgroupsAnyHierarchyWithCpuMemoryTest.ROOT_CGROUPS_CreateRemove
>> [ FAILED ] CgroupsAnyHierarchyWithCpuMemoryTest.ROOT_CGROUPS_Listen
>> [ FAILED ] CgroupsAnyHierarchyWithCpuAcctMemoryTest.ROOT_CGROUPS_Stat
>> [ FAILED ] CgroupsAnyHierarchyWithCpuMemoryFreezerTest.ROOT_CGROUPS_Freeze
>> [ FAILED ] CgroupsAnyHierarchyWithCpuMemoryFreezerTest.ROOT_CGROUPS_Kill
>> [ FAILED ] CgroupsAnyHierarchyWithCpuMemoryFreezerTest.ROOT_CGROUPS_Destroy
>>
>> 23 FAILED TESTS
>> make[3]: *** [check-local] Error 1
>> make[3]: Leaving directory `/usr/src/mesos/build/src'
>> make[2]: *** [check-am] Error 2
>> make[2]: Leaving directory `/usr/src/mesos/build/src'
>> make[1]: *** [check] Error 2
>> make[1]: Leaving directory `/usr/src/mesos/build/src'
>> make: *** [check-recursive] Error 1
>>
>> Kevin
>>
>> --
>> Kevin Lyda
>> Galway, Ireland
>> US Citizen overseas? We can vote.
>> Register now: http://www.votefromabroad.org/
>
--
Kevin Lyda
Galway, Ireland
US Citizen overseas? We can vote.
Register now: http://www.votefromabroad.org/
Re: make check failure...
Posted by Ian Downes <id...@twitter.com>.
The documentation is out of date and lxc has been deprecated; cgroups are used directly instead. We'll be updating the documentation shortly to reflect this.
Installing lxc possibly mounted cgroups and led to the slew of busy errors. Could you please post the output of `mount` and, if you have libcgroup installed, `lscgroup`, ideally before and after lxc is installed? And also `uname -a`.
The failed BalloonFramework test may be related to a bug that we're investigating. Are you running this inside a VM by any chance?
Ian
On Aug 1, 2013, at 8:59 AM, Kevin Lyda <ke...@ie.suberic.net> wrote:
> Running make check on master from github (bfdfdce) and I'm getting a failure:
>
> [ OK ] AllocatorZooKeeperTest/0.SlaveReregistersFirst (2098 ms)
> [----------] 2 tests from AllocatorZooKeeperTest/0 (4242 ms total)
>
> [----------] Global test environment tear-down
> [==========] 240 tests from 46 test cases ran. (91604 ms total)
> [ PASSED ] 239 tests.
> [ FAILED ] 1 test, listed below:
> [ FAILED ] CgroupsIsolatorTest.ROOT_CGROUPS_BalloonFramework
>
> 1 FAILED TEST
> make[3]: *** [check-local] Error 1
> make[3]: Leaving directory `/usr/src/mesos/build/src'
> make[2]: *** [check-am] Error 2
> make[2]: Leaving directory `/usr/src/mesos/build/src'
> make[1]: *** [check] Error 2
> make[1]: Leaving directory `/usr/src/mesos/build/src'
> make: *** [check-recursive] Error 1
>
> This is on an up-to-date Ubuntu 12.04.2. Ah. I realise I hadn't
> installed lxc. OK, trying again.
>
> Nope. Now the errors are worse (built from a clean tree, same commit):
>
> [----------] Global test environment tear-down
> [==========] 239 tests from 45 test cases ran. (91234 ms total)
> [ PASSED ] 216 tests.
> [ FAILED ] 23 tests, listed below:
> [ FAILED ] SlaveRecoveryTest/1.RecoverSlaveState, where TypeParam =
> mesos::internal::slave::CgroupsIsolator
> [ FAILED ] SlaveRecoveryTest/1.RecoverStatusUpdateManager, where
> TypeParam = mesos::internal::slave::CgroupsIsolator
> [ FAILED ] SlaveRecoveryTest/1.ReconnectExecutor, where TypeParam =
> mesos::internal::slave::CgroupsIsolator
> [ FAILED ] SlaveRecoveryTest/1.RecoverUnregisteredExecutor, where
> TypeParam = mesos::internal::slave::CgroupsIsolator
> [ FAILED ] SlaveRecoveryTest/1.RecoverTerminatedExecutor, where
> TypeParam = mesos::internal::slave::CgroupsIsolator
> [ FAILED ] SlaveRecoveryTest/1.CleanupExecutor, where TypeParam =
> mesos::internal::slave::CgroupsIsolator
> [ FAILED ] SlaveRecoveryTest/1.RemoveNonCheckpointingFramework,
> where TypeParam = mesos::internal::slave::CgroupsIsolator
> [ FAILED ] SlaveRecoveryTest/1.NonCheckpointingFramework, where
> TypeParam = mesos::internal::slave::CgroupsIsolator
> [ FAILED ] SlaveRecoveryTest/1.NonCheckpointingSlave, where
> TypeParam = mesos::internal::slave::CgroupsIsolator
> [ FAILED ] SlaveRecoveryTest/1.KillTask, where TypeParam =
> mesos::internal::slave::CgroupsIsolator
> [ FAILED ] SlaveRecoveryTest/1.GCExecutor, where TypeParam =
> mesos::internal::slave::CgroupsIsolator
> [ FAILED ] SlaveRecoveryTest/1.ShutdownSlave, where TypeParam =
> mesos::internal::slave::CgroupsIsolator
> [ FAILED ] IsolatorTest/1.Usage, where TypeParam =
> mesos::internal::slave::CgroupsIsolator
> [ FAILED ] CgroupsIsolatorTest.ROOT_CGROUPS_BalloonFramework
> [ FAILED ] CgroupsAnyHierarchyWithCpuMemoryTest.ROOT_CGROUPS_Busy
> [ FAILED ] CgroupsAnyHierarchyWithCpuMemoryTest.ROOT_CGROUPS_SubsystemsHierarchy
> [ FAILED ] CgroupsAnyHierarchyWithCpuMemoryTest.ROOT_CGROUPS_MountedSubsystems
> [ FAILED ] CgroupsAnyHierarchyWithCpuMemoryTest.ROOT_CGROUPS_CreateRemove
> [ FAILED ] CgroupsAnyHierarchyWithCpuMemoryTest.ROOT_CGROUPS_Listen
> [ FAILED ] CgroupsAnyHierarchyWithCpuAcctMemoryTest.ROOT_CGROUPS_Stat
> [ FAILED ] CgroupsAnyHierarchyWithCpuMemoryFreezerTest.ROOT_CGROUPS_Freeze
> [ FAILED ] CgroupsAnyHierarchyWithCpuMemoryFreezerTest.ROOT_CGROUPS_Kill
> [ FAILED ] CgroupsAnyHierarchyWithCpuMemoryFreezerTest.ROOT_CGROUPS_Destroy
>
> 23 FAILED TESTS
> make[3]: *** [check-local] Error 1
> make[3]: Leaving directory `/usr/src/mesos/build/src'
> make[2]: *** [check-am] Error 2
> make[2]: Leaving directory `/usr/src/mesos/build/src'
> make[1]: *** [check] Error 2
> make[1]: Leaving directory `/usr/src/mesos/build/src'
> make: *** [check-recursive] Error 1
>
> Kevin
>
> --
> Kevin Lyda
> Galway, Ireland
> US Citizen overseas? We can vote.
> Register now: http://www.votefromabroad.org/