You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@mesos.apache.org by Andrei Budnik <ab...@mesosphere.com> on 2018/04/04 12:17:58 UTC
Review Request 66449: Fixed flaky `ROOT_IsolatorFlags` test.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66449/
-----------------------------------------------------------
Review request for mesos, Alexander Rukletsov, Gilbert Song, and Jie Yu.
Bugs: MESOS-8489
https://issues.apache.org/jira/browse/MESOS-8489
Repository: mesos
Description
-------
Starting more than one agent simultaneously in tests leads to a race
condition between a linux launcher which calls `cgroups::prepare()` for
the first slave and `LinuxLauncherProcess::recover()` which iterates
over cgroups hierarchy for the second slave. Therefore, `mesos/test`
cgroup that is created to check if the kernel supports nested cgroups
can be detected by a recovery process as they use same cgroup hierarchy
path by default. That leads to orphan containers and causes flakiness
of `ROOT_IsolatorFlags` test. To fix the issue, this patch adds
termination of an agent before starting a new one.
Diffs
-----
src/tests/containerizer/linux_capabilities_isolator_tests.cpp 147f2cc09307cf8c9cf6f71d0175f8a3593c0256
Diff: https://reviews.apache.org/r/66449/diff/1/
Testing
-------
internal CI
Thanks,
Andrei Budnik
Re: Review Request 66449: Fixed flaky `ROOT_IsolatorFlags` test.
Posted by Mesos Reviewbot Windows <re...@mesos.apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66449/#review200449
-----------------------------------------------------------
FAIL: Some of the unit tests failed. Please check the relevant logs.
Reviews applied: `['66449']`
Failed command: `Start-MesosCITesting`
All the build artifacts available at: http://dcos-win.westus.cloudapp.azure.com/mesos-build/review/66449
Relevant logs:
- [mesos-tests-stdout.log](http://dcos-win.westus.cloudapp.azure.com/mesos-build/review/66449/logs/mesos-tests-stdout.log):
```
[ OK ] Endpoint/SlaveEndpointTest.NoAuthorizer/2 (116 ms)
[----------] 9 tests from Endpoint/SlaveEndpointTest (1067 ms total)
[----------] 2 tests from ContainerizerType/DefaultContainerDNSFlagTest
[ RUN ] ContainerizerType/DefaultContainerDNSFlagTest.ValidateFlag/0
[ OK ] ContainerizerType/DefaultContainerDNSFlagTest.ValidateFlag/0 (38 ms)
[ RUN ] ContainerizerType/DefaultContainerDNSFlagTest.ValidateFlag/1
[ OK ] ContainerizerType/DefaultContainerDNSFlagTest.ValidateFlag/1 (41 ms)
[----------] 2 tests from ContainerizerType/DefaultContainerDNSFlagTest (81 ms total)
[----------] 1 test from IsolationFlag/CpuIsolatorTest
[ RUN ] IsolationFlag/CpuIsolatorTest.ROOT_UserCpuUsage/0
[ OK ] IsolationFlag/CpuIsolatorTest.ROOT_UserCpuUsage/0 (807 ms)
[----------] 1 test from IsolationFlag/CpuIsolatorTest (835 ms total)
[----------] 1 test from IsolationFlag/MemoryIsolatorTest
[ RUN ] IsolationFlag/MemoryIsolatorTest.ROOT_MemUsage/0
[ OK ] IsolationFlag/MemoryIsolatorTest.ROOT_MemUsage/0 (836 ms)
[----------] 1 test from IsolationFlag/MemoryIsolatorTest (860 ms total)
[----------] Global test environment tear-down
[==========] 949 tests from 94 test cases ran. (448338 ms total)
[ PASSED ] 948 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] CommandExecutorCheckTest.CommandCheckTimeout
1 FAILED TEST
YOU HAVE 214 DISABLED TESTS
```
- [mesos-tests-stderr.log](http://dcos-win.westus.cloudapp.azure.com/mesos-build/review/66449/logs/mesos-tests-stderr.log):
```
I0404 13:15:01.126520 6260 master.cpp:10449] Updating the state of task eb30f9c5-e691-4ac4-89de-a00a5f2cea69 of framework 970932f1-3c0e-4e32-af5d-101b00483f56-0000 (latest state: TASK_KILLED, status update state: TASK_KILLED)
I0404 13:15:01.126520 12600 slave.cpp:3877] Shutting down fraI0404 13:15:00.947546 13736 exec.cpp:162] Version: 1.6.0
I0404 13:15:00.973520 6356 exec.cpp:236] Executor registered on agent 970932f1-3c0e-4e32-af5d-101b00483f56-S0
I0404 13:15:00.977519 12940 executor.cpp:176] Received SUBSCRIBED event
I0404 13:15:00.982553 12940 executor.cpp:180] Subscribed executor on winbldsrv-01.zq4gs31qjdiunm1ryi1452nvnh.dx.internal.cloudapp.net
I0404 13:15:00.982553 12940 executor.cpp:176] Received LAUNCH event
I0404 13:15:00.988550 12940 executor.cpp:648] Starting task eb30f9c5-e691-4ac4-89de-a00a5f2cea69
I0404 13:15:01.069747 12940 executor.cpp:483] Running 'D:\DCOS\mesos\src\mesos-containerizer.exe launch <POSSIBLY-SENSITIVE-DATA>'
I0404 13:15:01.098561 12940 executor.cpp:661] Forked command at 5616
I0404 13:15:01.129519 5664 exec.cpp:445] Executor asked to shutdown
I0404 13:15:01.129519 12972 executor.cpp:176] Received SHUTDOWN event
I0404 13:15:01.129519 12972 executor.cpp:758] Shutting down
I0404 13:15:01.129519 12972 executor.cpp:868] Sending SIGTERM to process tree at pid 5mework 970932f1-3c0e-4e32-af5d-101b00483f56-0000
I0404 13:15:01.127542 12600 slave.cpp:6574] Shutting down executor 'eb30f9c5-e691-4ac4-89de-a00a5f2cea69' of framework 970932f1-3c0e-4e32-af5d-101b00483f56-0000 at executor(1)@10.3.1.8:59570
I0404 13:15:01.128708 13916 slave.cpp:923] Agent terminating
W0404 13:15:01.128708 13916 slave.cpp:3873] Ignoring shutdown framework 970932f1-3c0e-4e32-af5d-101b00483f56-0000 because it is terminating
I0404 13:15:01.129519 6260 master.cpp:10548] Removing task eb30f9c5-e691-4ac4-89de-a00a5f2cea69 with resources cpus(allocated: *):4; mem(allocated: *):2048; disk(allocated: *):1024; ports(allocated: *):[31000-32000] of framework 970932f1-3c0e-4e32-af5d-101b00483f56-0000 on agent 970932f1-3c0e-4e32-af5d-101b00483f56-S0 at slave(418)@10.3.1.8:59549 (winbldsrv-01.zq4gs31qjdiunm1ryi1452nvnh.dx.internal.cloudapp.net)
I0404 13:15:01.131566 8520 containerizer.cpp:2338] Destroying container 16a2a12b-8ed1-4d4b-9f8e-66983e83a42c in RUNNING state
I0404 13:15:01.132565 8520 containerizer.cpp:2952] Transitioning the state of container 16a2a12b-8ed1-4d4b-9f8e-66983e83a42c from RUNNING to DESTROYING
I0404 13:15:01.133525 6260 master.cpp:1295] Agent 970932f1-3c0e-4e32-af5d-101b00483f56-S0 at slave(418)@10.3.1.8:59549 (winbldsrv-01.zq4gs31qjdiunm1ryi1452nvnh.dx.internal.cloudapp.net) disconnected
I0404 13:15:01.133525 6260 master.cpp:3286] Disconnecting agent 970932f1-3c0e-4e32-af5d-101b00483f56-S0 at slave(418)@10.3.1.8:59549 (winbldsrv-01.zq4gs31qjdiunm1ryi1452nvnh.dx.internal.cloudapp.net)
I0404 13:15:01.133525 6260 master.cpp:3305] Deactivating agent 970932f1-3c0e-4e32-af5d-101b00483f56-S0 at slave(418)@10.3.1.8:59549 (winbldsrv-01.zq4gs31qjdiunm1ryi1452nvnh.dx.internal.cloudapp.net)
I0404 13:15:01.133525 8520 launcher.cpp:156] Asked to destroy container 16a2a12b-8ed1-4d4b-9f8e-66983e83a42c
I0404 13:15:01.133525 12400 hierarchical.cpp:344] Removed framework 970932f1-3c0e-4e32-af5d-101b00483f56-0000
I0404 13:15:01.134529 12400 hierarchical.cpp:766] Agent 970932f1-3c0e-4e32-af5d-101b00483f56-S0 deactivated
I0404 13:15:01.243808 1696 containerizer.cpp:2791] Container 16a2a12b-8ed1-4d4b-9f8e-66983e83a42c has exited
I0404 13:15:01.276810 13916 master.cpp:1137] Master terminating
I0404 13:15:01.278810 8520 hierarchical.cpp:609] Removed agent 970932f1-3c0e-4e32-af5d-101b00483f56-S0
I0404 13:15:01.664340 9736 process.cpp:929] Stopped the socket accept loop
```
- Mesos Reviewbot Windows
On April 4, 2018, 5:17 a.m., Andrei Budnik wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66449/
> -----------------------------------------------------------
>
> (Updated April 4, 2018, 5:17 a.m.)
>
>
> Review request for mesos, Alexander Rukletsov, Gilbert Song, and Jie Yu.
>
>
> Bugs: MESOS-8489
> https://issues.apache.org/jira/browse/MESOS-8489
>
>
> Repository: mesos
>
>
> Description
> -------
>
> Starting more than one agent simultaneously in tests leads to a race
> condition between a linux launcher which calls `cgroups::prepare()` for
> the first slave and `LinuxLauncherProcess::recover()` which iterates
> over cgroups hierarchy for the second slave. Therefore, `mesos/test`
> cgroup that is created to check if the kernel supports nested cgroups
> can be detected by a recovery process as they use same cgroup hierarchy
> path by default. That leads to orphan containers and causes flakiness
> of `ROOT_IsolatorFlags` test. To fix the issue, this patch adds
> termination of an agent before starting a new one.
>
>
> Diffs
> -----
>
> src/tests/containerizer/linux_capabilities_isolator_tests.cpp 147f2cc09307cf8c9cf6f71d0175f8a3593c0256
>
>
> Diff: https://reviews.apache.org/r/66449/diff/1/
>
>
> Testing
> -------
>
> internal CI
>
>
> Thanks,
>
> Andrei Budnik
>
>
Re: Review Request 66449: Fixed flaky `ROOT_IsolatorFlags` test.
Posted by Gilbert Song <so...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66449/#review200483
-----------------------------------------------------------
Ship it!
Ship It!
- Gilbert Song
On April 4, 2018, 5:17 a.m., Andrei Budnik wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66449/
> -----------------------------------------------------------
>
> (Updated April 4, 2018, 5:17 a.m.)
>
>
> Review request for mesos, Alexander Rukletsov, Gilbert Song, and Jie Yu.
>
>
> Bugs: MESOS-8489
> https://issues.apache.org/jira/browse/MESOS-8489
>
>
> Repository: mesos
>
>
> Description
> -------
>
> Starting more than one agent simultaneously in tests leads to a race
> condition between a linux launcher which calls `cgroups::prepare()` for
> the first slave and `LinuxLauncherProcess::recover()` which iterates
> over cgroups hierarchy for the second slave. Therefore, `mesos/test`
> cgroup that is created to check if the kernel supports nested cgroups
> can be detected by a recovery process as they use same cgroup hierarchy
> path by default. That leads to orphan containers and causes flakiness
> of `ROOT_IsolatorFlags` test. To fix the issue, this patch adds
> termination of an agent before starting a new one.
>
>
> Diffs
> -----
>
> src/tests/containerizer/linux_capabilities_isolator_tests.cpp 147f2cc09307cf8c9cf6f71d0175f8a3593c0256
>
>
> Diff: https://reviews.apache.org/r/66449/diff/1/
>
>
> Testing
> -------
>
> internal CI
>
>
> Thanks,
>
> Andrei Budnik
>
>
Re: Review Request 66449: Fixed flaky `ROOT_IsolatorFlags` test.
Posted by Andrei Budnik <ab...@mesosphere.com>.
> On April 4, 2018, 6:23 p.m., Gilbert Song wrote:
> > src/tests/containerizer/linux_capabilities_isolator_tests.cpp
> > Lines 747 (patched)
> > <https://reviews.apache.org/r/66449/diff/1/?file=1992360#file1992360line747>
> >
> > Should we call `slave.get()->terminate()` before `reset()`? See SlaveRecoveryTest.
I think it's not necessary since we call `terminate()` in `~Slave()`: https://github.com/apache/mesos/blob/594ee20c2453dad836313769aef9f8655cd75cd5/src/tests/cluster.cpp#L630
- Andrei
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66449/#review200472
-----------------------------------------------------------
On April 4, 2018, 12:17 p.m., Andrei Budnik wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66449/
> -----------------------------------------------------------
>
> (Updated April 4, 2018, 12:17 p.m.)
>
>
> Review request for mesos, Alexander Rukletsov, Gilbert Song, and Jie Yu.
>
>
> Bugs: MESOS-8489
> https://issues.apache.org/jira/browse/MESOS-8489
>
>
> Repository: mesos
>
>
> Description
> -------
>
> Starting more than one agent simultaneously in tests leads to a race
> condition between a linux launcher which calls `cgroups::prepare()` for
> the first slave and `LinuxLauncherProcess::recover()` which iterates
> over cgroups hierarchy for the second slave. Therefore, `mesos/test`
> cgroup that is created to check if the kernel supports nested cgroups
> can be detected by a recovery process as they use same cgroup hierarchy
> path by default. That leads to orphan containers and causes flakiness
> of `ROOT_IsolatorFlags` test. To fix the issue, this patch adds
> termination of an agent before starting a new one.
>
>
> Diffs
> -----
>
> src/tests/containerizer/linux_capabilities_isolator_tests.cpp 147f2cc09307cf8c9cf6f71d0175f8a3593c0256
>
>
> Diff: https://reviews.apache.org/r/66449/diff/1/
>
>
> Testing
> -------
>
> internal CI
>
>
> Thanks,
>
> Andrei Budnik
>
>
Re: Review Request 66449: Fixed flaky `ROOT_IsolatorFlags` test.
Posted by Gilbert Song <so...@gmail.com>.
> On April 4, 2018, 11:23 a.m., Gilbert Song wrote:
> > src/tests/containerizer/linux_capabilities_isolator_tests.cpp
> > Lines 747 (patched)
> > <https://reviews.apache.org/r/66449/diff/1/?file=1992360#file1992360line747>
> >
> > Should we call `slave.get()->terminate()` before `reset()`? See SlaveRecoveryTest.
>
> Andrei Budnik wrote:
> I think it's not necessary since we call `terminate()` in `~Slave()`: https://github.com/apache/mesos/blob/594ee20c2453dad836313769aef9f8655cd75cd5/src/tests/cluster.cpp#L630
gotcha.
- Gilbert
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66449/#review200472
-----------------------------------------------------------
On April 4, 2018, 5:17 a.m., Andrei Budnik wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66449/
> -----------------------------------------------------------
>
> (Updated April 4, 2018, 5:17 a.m.)
>
>
> Review request for mesos, Alexander Rukletsov, Gilbert Song, and Jie Yu.
>
>
> Bugs: MESOS-8489
> https://issues.apache.org/jira/browse/MESOS-8489
>
>
> Repository: mesos
>
>
> Description
> -------
>
> Starting more than one agent simultaneously in tests leads to a race
> condition between a linux launcher which calls `cgroups::prepare()` for
> the first slave and `LinuxLauncherProcess::recover()` which iterates
> over cgroups hierarchy for the second slave. Therefore, `mesos/test`
> cgroup that is created to check if the kernel supports nested cgroups
> can be detected by a recovery process as they use same cgroup hierarchy
> path by default. That leads to orphan containers and causes flakiness
> of `ROOT_IsolatorFlags` test. To fix the issue, this patch adds
> termination of an agent before starting a new one.
>
>
> Diffs
> -----
>
> src/tests/containerizer/linux_capabilities_isolator_tests.cpp 147f2cc09307cf8c9cf6f71d0175f8a3593c0256
>
>
> Diff: https://reviews.apache.org/r/66449/diff/1/
>
>
> Testing
> -------
>
> internal CI
>
>
> Thanks,
>
> Andrei Budnik
>
>
Re: Review Request 66449: Fixed flaky `ROOT_IsolatorFlags` test.
Posted by Gilbert Song <so...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66449/#review200472
-----------------------------------------------------------
src/tests/containerizer/linux_capabilities_isolator_tests.cpp
Lines 747 (patched)
<https://reviews.apache.org/r/66449/#comment281217>
Should we call `slave.get()->terminate()` before `reset()`? See SlaveRecoveryTest.
- Gilbert Song
On April 4, 2018, 5:17 a.m., Andrei Budnik wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66449/
> -----------------------------------------------------------
>
> (Updated April 4, 2018, 5:17 a.m.)
>
>
> Review request for mesos, Alexander Rukletsov, Gilbert Song, and Jie Yu.
>
>
> Bugs: MESOS-8489
> https://issues.apache.org/jira/browse/MESOS-8489
>
>
> Repository: mesos
>
>
> Description
> -------
>
> Starting more than one agent simultaneously in tests leads to a race
> condition between a linux launcher which calls `cgroups::prepare()` for
> the first slave and `LinuxLauncherProcess::recover()` which iterates
> over cgroups hierarchy for the second slave. Therefore, `mesos/test`
> cgroup that is created to check if the kernel supports nested cgroups
> can be detected by a recovery process as they use same cgroup hierarchy
> path by default. That leads to orphan containers and causes flakiness
> of `ROOT_IsolatorFlags` test. To fix the issue, this patch adds
> termination of an agent before starting a new one.
>
>
> Diffs
> -----
>
> src/tests/containerizer/linux_capabilities_isolator_tests.cpp 147f2cc09307cf8c9cf6f71d0175f8a3593c0256
>
>
> Diff: https://reviews.apache.org/r/66449/diff/1/
>
>
> Testing
> -------
>
> internal CI
>
>
> Thanks,
>
> Andrei Budnik
>
>
Re: Review Request 66449: Fixed flaky `ROOT_IsolatorFlags` test.
Posted by Mesos Reviewbot <re...@mesos.apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66449/#review200453
-----------------------------------------------------------
Patch looks great!
Reviews applied: [66449]
Passed command: export OS='ubuntu:14.04' BUILDTOOL='autotools' COMPILER='gcc' CONFIGURATION='--verbose --disable-libtool-wrappers' ENVIRONMENT='GLOG_v=1 MESOS_VERBOSE=1'; ./support/docker-build.sh
- Mesos Reviewbot
On April 4, 2018, 3:17 p.m., Andrei Budnik wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66449/
> -----------------------------------------------------------
>
> (Updated April 4, 2018, 3:17 p.m.)
>
>
> Review request for mesos, Alexander Rukletsov, Gilbert Song, and Jie Yu.
>
>
> Bugs: MESOS-8489
> https://issues.apache.org/jira/browse/MESOS-8489
>
>
> Repository: mesos
>
>
> Description
> -------
>
> Starting more than one agent simultaneously in tests leads to a race
> condition between a linux launcher which calls `cgroups::prepare()` for
> the first slave and `LinuxLauncherProcess::recover()` which iterates
> over cgroups hierarchy for the second slave. Therefore, `mesos/test`
> cgroup that is created to check if the kernel supports nested cgroups
> can be detected by a recovery process as they use same cgroup hierarchy
> path by default. That leads to orphan containers and causes flakiness
> of `ROOT_IsolatorFlags` test. To fix the issue, this patch adds
> termination of an agent before starting a new one.
>
>
> Diffs
> -----
>
> src/tests/containerizer/linux_capabilities_isolator_tests.cpp 147f2cc09307cf8c9cf6f71d0175f8a3593c0256
>
>
> Diff: https://reviews.apache.org/r/66449/diff/1/
>
>
> Testing
> -------
>
> internal CI
>
>
> Thanks,
>
> Andrei Budnik
>
>