You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@mesos.apache.org by "Alexander Rukletsov (JIRA)" <ji...@apache.org> on 2016/11/03 20:01:59 UTC
[jira] [Commented] (MESOS-6357)
`NestedMesosContainerizerTest.ROOT_CGROUPS_ParentExit` is flaky in Debian
8.
[ https://issues.apache.org/jira/browse/MESOS-6357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15634059#comment-15634059 ]
Alexander Rukletsov commented on MESOS-6357:
--------------------------------------------
Just saw that in the internal CI:
{noformat}
[17:41:36] : [Step 10/10] [ RUN ] NestedMesosContainerizerTest.ROOT_CGROUPS_ParentExit
[17:41:36] : [Step 10/10] I1103 17:41:36.569166 23098 containerizer.cpp:201] Using isolation: cgroups/cpu,filesystem/linux,namespaces/pid,network/cni,volume/image
[17:41:36] : [Step 10/10] I1103 17:41:36.572278 23098 linux_launcher.cpp:150] Using /sys/fs/cgroup/freezer as the freezer hierarchy for the Linux launcher
[17:41:36] : [Step 10/10] I1103 17:41:36.577702 23118 containerizer.cpp:557] Recovering containerizer
[17:41:36] : [Step 10/10] I1103 17:41:36.578866 23118 provisioner.cpp:253] Provisioner recovery complete
[17:41:36] : [Step 10/10] I1103 17:41:36.579076 23114 containerizer.cpp:940] Starting container 0aedeff2-9704-42a1-a245-dff493c1ae73 for executor 'executor' of framework
[17:41:36] : [Step 10/10] I1103 17:41:36.579404 23115 cgroups.cpp:405] Creating cgroup at '/sys/fs/cgroup/cpu,cpuacct/mesos_test_c7d3570c-5d28-43a2-88cc-6a8877d9d527/0aedeff2-9704-42a1-a245-dff493c1ae73' for container 0aedeff2-9704-42a1-a245-dff493c1ae73
[17:41:36] : [Step 10/10] I1103 17:41:36.580543 23112 cpu.cpp:101] Updated 'cpu.shares' to 1024 (cpus 1) for container 0aedeff2-9704-42a1-a245-dff493c1ae73
[17:41:36] : [Step 10/10] I1103 17:41:36.581063 23115 containerizer.cpp:1469] Launching 'mesos-containerizer' with flags '--command="{"arguments":["bash","-c","read key <&29"],"shell":false,"value":"\/bin\/bash"}" --environment="{"MESOS_SANDBOX":"\/mnt\/teamcity\/temp\/buildTmp\/NestedMesosContainerizerTest_ROOT_CGROUPS_ParentExit_0X7GV8"}" --help="false" --pipe_read="29" --pipe_write="32" --pre_exec_commands="[{"arguments":["mesos-containerizer","mount","--help=false","--operation=make-rslave","--path=\/"],"shell":false,"value":"\/mnt\/teamcity\/work\/4240ba9ddd0997c3\/build\/src\/mesos-containerizer"},{"shell":true,"value":"mount -n -t proc proc \/proc -o nosuid,noexec,nodev"}]" --runtime_directory="/mnt/teamcity/temp/buildTmp/NestedMesosContainerizerTest_ROOT_CGROUPS_ParentExit_kML9EY/containers/0aedeff2-9704-42a1-a245-dff493c1ae73" --unshare_namespace_mnt="false" --working_directory="/mnt/teamcity/temp/buildTmp/NestedMesosContainerizerTest_ROOT_CGROUPS_ParentExit_0X7GV8"'
[17:41:36] : [Step 10/10] I1103 17:41:36.581171 23112 linux_launcher.cpp:421] Launching container 0aedeff2-9704-42a1-a245-dff493c1ae73 and cloning with namespaces CLONE_NEWNS | CLONE_NEWPID
[17:41:36] : [Step 10/10] I1103 17:41:36.584450 23115 containerizer.cpp:1506] Checkpointing container's forked pid 17644 to '/mnt/teamcity/temp/buildTmp/NestedMesosContainerizerTest_ROOT_CGROUPS_ParentExit_bajwqO/meta/slaves/frameworks/executors/executor/runs/0aedeff2-9704-42a1-a245-dff493c1ae73/pids/forked.pid'
[17:41:36] : [Step 10/10] I1103 17:41:36.585805 23115 fetcher.cpp:345] Starting to fetch URIs for container: 0aedeff2-9704-42a1-a245-dff493c1ae73, directory: /mnt/teamcity/temp/buildTmp/NestedMesosContainerizerTest_ROOT_CGROUPS_ParentExit_0X7GV8
[17:41:36] : [Step 10/10] I1103 17:41:36.586761 23118 containerizer.cpp:1674] Starting nested container 0aedeff2-9704-42a1-a245-dff493c1ae73.37cb3ef8-056a-438e-a3ed-f852e51e4197
[17:41:36] : [Step 10/10] I1103 17:41:36.587481 23119 containerizer.cpp:1469] Launching 'mesos-containerizer' with flags '--command="{"shell":true,"value":"sleep 1000"}" --environment="{"MESOS_SANDBOX":"\/mnt\/teamcity\/temp\/buildTmp\/NestedMesosContainerizerTest_ROOT_CGROUPS_ParentExit_0X7GV8\/containers\/37cb3ef8-056a-438e-a3ed-f852e51e4197"}" --help="false" --pipe_read="29" --pipe_write="32" --pre_exec_commands="[{"arguments":["mesos-containerizer","mount","--help=false","--operation=make-rslave","--path=\/"],"shell":false,"value":"\/mnt\/teamcity\/work\/4240ba9ddd0997c3\/build\/src\/mesos-containerizer"},{"shell":true,"value":"mount -n -t proc proc \/proc -o nosuid,noexec,nodev"}]" --runtime_directory="/mnt/teamcity/temp/buildTmp/NestedMesosContainerizerTest_ROOT_CGROUPS_ParentExit_kML9EY/containers/0aedeff2-9704-42a1-a245-dff493c1ae73/containers/37cb3ef8-056a-438e-a3ed-f852e51e4197" --unshare_namespace_mnt="false" --working_directory="/mnt/teamcity/temp/buildTmp/NestedMesosContainerizerTest_ROOT_CGROUPS_ParentExit_0X7GV8/containers/37cb3ef8-056a-438e-a3ed-f852e51e4197"'
[17:41:36] : [Step 10/10] I1103 17:41:36.587601 23113 linux_launcher.cpp:421] Launching nested container 0aedeff2-9704-42a1-a245-dff493c1ae73.37cb3ef8-056a-438e-a3ed-f852e51e4197 and cloning with namespaces CLONE_NEWNS | CLONE_NEWPID
[17:41:36] : [Step 10/10] Executing pre-exec command '{"arguments":["mesos-containerizer","mount","--help=false","--operation=make-rslave","--path=\/"],"shell":false,"value":"\/mnt\/teamcity\/work\/4240ba9ddd0997c3\/build\/src\/mesos-containerizer"}'
[17:41:36] : [Step 10/10] Executing pre-exec command '{"shell":true,"value":"mount -n -t proc proc \/proc -o nosuid,noexec,nodev"}'
[17:41:36] : [Step 10/10] bash: 29: Bad file descriptor
[17:41:36] : [Step 10/10] ../../src/tests/containerizer/nested_mesos_containerizer_tests.cpp:464: Failure
[17:41:36] : [Step 10/10] (launch).failure(): Failed to fork: Failed to clone child process: Failed to synchronize child process
[17:41:36] : [Step 10/10] I1103 17:41:36.696262 23113 cgroups.cpp:2726] Freezing cgroup /sys/fs/cgroup/freezer/mesos_test_c7d3570c-5d28-43a2-88cc-6a8877d9d527/0aedeff2-9704-42a1-a245-dff493c1ae73/mesos/37cb3ef8-056a-438e-a3ed-f852e51e4197
[17:41:36] : [Step 10/10] I1103 17:41:36.697134 23116 cgroups.cpp:1439] Successfully froze cgroup /sys/fs/cgroup/freezer/mesos_test_c7d3570c-5d28-43a2-88cc-6a8877d9d527/0aedeff2-9704-42a1-a245-dff493c1ae73/mesos/37cb3ef8-056a-438e-a3ed-f852e51e4197 after 838912ns
[17:41:36] : [Step 10/10] I1103 17:41:36.697980 23113 cgroups.cpp:2744] Thawing cgroup /sys/fs/cgroup/freezer/mesos_test_c7d3570c-5d28-43a2-88cc-6a8877d9d527/0aedeff2-9704-42a1-a245-dff493c1ae73/mesos/37cb3ef8-056a-438e-a3ed-f852e51e4197
[17:41:36] : [Step 10/10] I1103 17:41:36.698784 23113 cgroups.cpp:1468] Successfully thawed cgroup /sys/fs/cgroup/freezer/mesos_test_c7d3570c-5d28-43a2-88cc-6a8877d9d527/0aedeff2-9704-42a1-a245-dff493c1ae73/mesos/37cb3ef8-056a-438e-a3ed-f852e51e4197 after 786944ns
[17:41:36] : [Step 10/10] I1103 17:41:36.700161 23119 cgroups.cpp:2726] Freezing cgroup /sys/fs/cgroup/freezer/mesos_test_c7d3570c-5d28-43a2-88cc-6a8877d9d527/0aedeff2-9704-42a1-a245-dff493c1ae73/mesos
[17:41:36] : [Step 10/10] I1103 17:41:36.802855 23114 cgroups.cpp:1439] Successfully froze cgroup /sys/fs/cgroup/freezer/mesos_test_c7d3570c-5d28-43a2-88cc-6a8877d9d527/0aedeff2-9704-42a1-a245-dff493c1ae73/mesos after 102.659072ms
[17:41:36] : [Step 10/10] I1103 17:41:36.803766 23112 cgroups.cpp:2744] Thawing cgroup /sys/fs/cgroup/freezer/mesos_test_c7d3570c-5d28-43a2-88cc-6a8877d9d527/0aedeff2-9704-42a1-a245-dff493c1ae73/mesos
[17:41:36] : [Step 10/10] I1103 17:41:36.804561 23112 cgroups.cpp:1468] Successfully thawed cgroup /sys/fs/cgroup/freezer/mesos_test_c7d3570c-5d28-43a2-88cc-6a8877d9d527/0aedeff2-9704-42a1-a245-dff493c1ae73/mesos after 776960ns
[17:41:36] : [Step 10/10] I1103 17:41:36.805766 23118 cgroups.cpp:2726] Freezing cgroup /sys/fs/cgroup/freezer/mesos_test_c7d3570c-5d28-43a2-88cc-6a8877d9d527/0aedeff2-9704-42a1-a245-dff493c1ae73
[17:41:36] : [Step 10/10] I1103 17:41:36.907245 23117 cgroups.cpp:1439] Successfully froze cgroup /sys/fs/cgroup/freezer/mesos_test_c7d3570c-5d28-43a2-88cc-6a8877d9d527/0aedeff2-9704-42a1-a245-dff493c1ae73 after 101.461248ms
[17:41:36] : [Step 10/10] I1103 17:41:36.908140 23113 cgroups.cpp:2744] Thawing cgroup /sys/fs/cgroup/freezer/mesos_test_c7d3570c-5d28-43a2-88cc-6a8877d9d527/0aedeff2-9704-42a1-a245-dff493c1ae73
[17:41:36] : [Step 10/10] I1103 17:41:36.908953 23113 cgroups.cpp:1468] Successfully thawed cgroup /sys/fs/cgroup/freezer/mesos_test_c7d3570c-5d28-43a2-88cc-6a8877d9d527/0aedeff2-9704-42a1-a245-dff493c1ae73 after 795904ns
[17:41:36] : [Step 10/10] I1103 17:41:36.910051 23115 cgroups.cpp:2726] Freezing cgroup /sys/fs/cgroup/freezer/mesos_test_c7d3570c-5d28-43a2-88cc-6a8877d9d527
[17:41:37] : [Step 10/10] I1103 17:41:37.011844 23114 cgroups.cpp:1439] Successfully froze cgroup /sys/fs/cgroup/freezer/mesos_test_c7d3570c-5d28-43a2-88cc-6a8877d9d527 after 101.77024ms
[17:41:37] : [Step 10/10] I1103 17:41:37.012712 23118 cgroups.cpp:2744] Thawing cgroup /sys/fs/cgroup/freezer/mesos_test_c7d3570c-5d28-43a2-88cc-6a8877d9d527
[17:41:37] : [Step 10/10] I1103 17:41:37.013509 23118 cgroups.cpp:1468] Successfully thawed cgroup /sys/fs/cgroup/freezer/mesos_test_c7d3570c-5d28-43a2-88cc-6a8877d9d527 after 779008ns
[17:41:37] : [Step 10/10] [ FAILED ] NestedMesosContainerizerTest.ROOT_CGROUPS_ParentExit (457 ms)
{noformat}
> `NestedMesosContainerizerTest.ROOT_CGROUPS_ParentExit` is flaky in Debian 8.
> ----------------------------------------------------------------------------
>
> Key: MESOS-6357
> URL: https://issues.apache.org/jira/browse/MESOS-6357
> Project: Mesos
> Issue Type: Bug
> Components: tests
> Affects Versions: 1.1.0
> Environment: Debian 8 with SSL enabled
> Reporter: Gilbert Song
> Assignee: Gilbert Song
> Labels: flaky-test
>
> {noformat}
> [00:21:51] : [Step 10/10] [ RUN ] NestedMesosContainerizerTest.ROOT_CGROUPS_ParentExit
> [00:21:51]W: [Step 10/10] I1008 00:21:51.357839 23530 containerizer.cpp:202] Using isolation: cgroups/cpu,filesystem/linux,namespaces/pid,network/cni,volume/image
> [00:21:51]W: [Step 10/10] I1008 00:21:51.361143 23530 linux_launcher.cpp:150] Using /sys/fs/cgroup/freezer as the freezer hierarchy for the Linux launcher
> [00:21:51]W: [Step 10/10] I1008 00:21:51.366930 23547 containerizer.cpp:557] Recovering containerizer
> [00:21:51]W: [Step 10/10] I1008 00:21:51.367962 23551 provisioner.cpp:253] Provisioner recovery complete
> [00:21:51]W: [Step 10/10] I1008 00:21:51.368253 23549 containerizer.cpp:954] Starting container 42589936-56b2-4e41-86d8-447bfaba4666 for executor 'executor' of framework
> [00:21:51]W: [Step 10/10] I1008 00:21:51.368577 23548 cgroups.cpp:404] Creating cgroup at '/sys/fs/cgroup/cpu,cpuacct/mesos_test_458f8018-67e7-4cc6-8126-a535974db35d/42589936-56b2-4e41-86d8-447bfaba4666' for container 42589936-56b2-4e41-86d8-447bfaba4666
> [00:21:51]W: [Step 10/10] I1008 00:21:51.369863 23544 cpu.cpp:103] Updated 'cpu.shares' to 1024 (cpus 1) for container 42589936-56b2-4e41-86d8-447bfaba4666
> [00:21:51]W: [Step 10/10] I1008 00:21:51.370384 23545 containerizer.cpp:1443] Launching 'mesos-containerizer' with flags '--command="{"shell":true,"value":"read key <&30"}" --help="false" --pipe_read="30" --pipe_write="34" --pre_exec_commands="[{"arguments":["mesos-containerizer","mount","--help=false","--operation=make-rslave","--path=\/"],"shell":false,"value":"\/mnt\/teamcity\/work\/4240ba9ddd0997c3\/build\/src\/mesos-containerizer"},{"shell":true,"value":"mount -n -t proc proc \/proc -o nosuid,noexec,nodev"}]" --runtime_directory="/mnt/teamcity/temp/buildTmp/NestedMesosContainerizerTest_ROOT_CGROUPS_ParentExit_sEbtvQ/containers/42589936-56b2-4e41-86d8-447bfaba4666" --unshare_namespace_mnt="false" --working_directory="/mnt/teamcity/temp/buildTmp/NestedMesosContainerizerTest_ROOT_CGROUPS_ParentExit_MqjHi0"'
> [00:21:51]W: [Step 10/10] I1008 00:21:51.370483 23544 linux_launcher.cpp:421] Launching container 42589936-56b2-4e41-86d8-447bfaba4666 and cloning with namespaces CLONE_NEWNS | CLONE_NEWPID
> [00:21:51]W: [Step 10/10] I1008 00:21:51.374867 23545 containerizer.cpp:1480] Checkpointing container's forked pid 14139 to '/mnt/teamcity/temp/buildTmp/NestedMesosContainerizerTest_ROOT_CGROUPS_ParentExit_gzjeKG/meta/slaves/frameworks/executors/executor/runs/42589936-56b2-4e41-86d8-447bfaba4666/pids/forked.pid'
> [00:21:51]W: [Step 10/10] I1008 00:21:51.376519 23551 containerizer.cpp:1648] Starting nested container 42589936-56b2-4e41-86d8-447bfaba4666.a5bc9913-c32c-40c6-ab78-2b08910847f8
> [00:21:51]W: [Step 10/10] I1008 00:21:51.377296 23549 containerizer.cpp:1443] Launching 'mesos-containerizer' with flags '--command="{"shell":true,"value":"sleep 1000"}" --help="false" --pipe_read="30" --pipe_write="34" --pre_exec_commands="[{"arguments":["mesos-containerizer","mount","--help=false","--operation=make-rslave","--path=\/"],"shell":false,"value":"\/mnt\/teamcity\/work\/4240ba9ddd0997c3\/build\/src\/mesos-containerizer"},{"shell":true,"value":"mount -n -t proc proc \/proc -o nosuid,noexec,nodev"}]" --runtime_directory="/mnt/teamcity/temp/buildTmp/NestedMesosContainerizerTest_ROOT_CGROUPS_ParentExit_sEbtvQ/containers/42589936-56b2-4e41-86d8-447bfaba4666/containers/a5bc9913-c32c-40c6-ab78-2b08910847f8" --unshare_namespace_mnt="false" --working_directory="/mnt/teamcity/temp/buildTmp/NestedMesosContainerizerTest_ROOT_CGROUPS_ParentExit_MqjHi0/containers/a5bc9913-c32c-40c6-ab78-2b08910847f8"'
> [00:21:51]W: [Step 10/10] I1008 00:21:51.377424 23548 linux_launcher.cpp:421] Launching nested container 42589936-56b2-4e41-86d8-447bfaba4666.a5bc9913-c32c-40c6-ab78-2b08910847f8 and cloning with namespaces CLONE_NEWNS | CLONE_NEWPID
> [00:21:51] : [Step 10/10] Executing pre-exec command '{"arguments":["mesos-containerizer","mount","--help=false","--operation=make-rslave","--path=\/"],"shell":false,"value":"\/mnt\/teamcity\/work\/4240ba9ddd0997c3\/build\/src\/mesos-containerizer"}'
> [00:21:51] : [Step 10/10] Executing pre-exec command '{"shell":true,"value":"mount -n -t proc proc \/proc -o nosuid,noexec,nodev"}'
> [00:21:51]W: [Step 10/10] sh: 1: Syntax error: Bad fd number
> [00:21:51] : [Step 10/10] ../../src/tests/containerizer/nested_mesos_containerizer_tests.cpp:451: Failure
> [00:21:51] : [Step 10/10] (launch).failure(): Failed to fork: Failed to clone child process: Failed to synchronize child process
> [00:21:51]W: [Step 10/10] I1008 00:21:51.506002 23544 cgroups.cpp:2705] Freezing cgroup /sys/fs/cgroup/freezer/mesos_test_458f8018-67e7-4cc6-8126-a535974db35d/42589936-56b2-4e41-86d8-447bfaba4666/mesos/a5bc9913-c32c-40c6-ab78-2b08910847f8
> [00:21:51]W: [Step 10/10] I1008 00:21:51.507055 23546 cgroups.cpp:1439] Successfully froze cgroup /sys/fs/cgroup/freezer/mesos_test_458f8018-67e7-4cc6-8126-a535974db35d/42589936-56b2-4e41-86d8-447bfaba4666/mesos/a5bc9913-c32c-40c6-ab78-2b08910847f8 after 1.019904ms
> [00:21:51]W: [Step 10/10] I1008 00:21:51.507967 23549 cgroups.cpp:2723] Thawing cgroup /sys/fs/cgroup/freezer/mesos_test_458f8018-67e7-4cc6-8126-a535974db35d/42589936-56b2-4e41-86d8-447bfaba4666/mesos/a5bc9913-c32c-40c6-ab78-2b08910847f8
> [00:21:51]W: [Step 10/10] I1008 00:21:51.508783 23549 cgroups.cpp:1468] Successfully thawed cgroup /sys/fs/cgroup/freezer/mesos_test_458f8018-67e7-4cc6-8126-a535974db35d/42589936-56b2-4e41-86d8-447bfaba4666/mesos/a5bc9913-c32c-40c6-ab78-2b08910847f8 after 797952ns
> [00:21:51]W: [Step 10/10] I1008 00:21:51.510030 23551 cgroups.cpp:2705] Freezing cgroup /sys/fs/cgroup/freezer/mesos_test_458f8018-67e7-4cc6-8126-a535974db35d/42589936-56b2-4e41-86d8-447bfaba4666/mesos
> [00:21:51]W: [Step 10/10] I1008 00:21:51.612181 23551 cgroups.cpp:1439] Successfully froze cgroup /sys/fs/cgroup/freezer/mesos_test_458f8018-67e7-4cc6-8126-a535974db35d/42589936-56b2-4e41-86d8-447bfaba4666/mesos after 102.128896ms
> [00:21:51]W: [Step 10/10] I1008 00:21:51.613134 23548 cgroups.cpp:2723] Thawing cgroup /sys/fs/cgroup/freezer/mesos_test_458f8018-67e7-4cc6-8126-a535974db35d/42589936-56b2-4e41-86d8-447bfaba4666/mesos
> [00:21:51]W: [Step 10/10] I1008 00:21:51.614034 23547 cgroups.cpp:1468] Successfully thawed cgroup /sys/fs/cgroup/freezer/mesos_test_458f8018-67e7-4cc6-8126-a535974db35d/42589936-56b2-4e41-86d8-447bfaba4666/mesos after 878080ns
> [00:21:51]W: [Step 10/10] I1008 00:21:51.615409 23550 cgroups.cpp:2705] Freezing cgroup /sys/fs/cgroup/freezer/mesos_test_458f8018-67e7-4cc6-8126-a535974db35d/42589936-56b2-4e41-86d8-447bfaba4666
> [00:21:51]W: [Step 10/10] I1008 00:21:51.717521 23550 cgroups.cpp:1439] Successfully froze cgroup /sys/fs/cgroup/freezer/mesos_test_458f8018-67e7-4cc6-8126-a535974db35d/42589936-56b2-4e41-86d8-447bfaba4666 after 102.08512ms
> [00:21:51]W: [Step 10/10] I1008 00:21:51.718426 23545 cgroups.cpp:2723] Thawing cgroup /sys/fs/cgroup/freezer/mesos_test_458f8018-67e7-4cc6-8126-a535974db35d/42589936-56b2-4e41-86d8-447bfaba4666
> [00:21:51]W: [Step 10/10] I1008 00:21:51.719254 23547 cgroups.cpp:1468] Successfully thawed cgroup /sys/fs/cgroup/freezer/mesos_test_458f8018-67e7-4cc6-8126-a535974db35d/42589936-56b2-4e41-86d8-447bfaba4666 after 806912ns
> [00:21:51]W: [Step 10/10] I1008 00:21:51.720582 23551 cgroups.cpp:2705] Freezing cgroup /sys/fs/cgroup/freezer/mesos_test_458f8018-67e7-4cc6-8126-a535974db35d
> [00:21:51]W: [Step 10/10] I1008 00:21:51.822980 23546 cgroups.cpp:1439] Successfully froze cgroup /sys/fs/cgroup/freezer/mesos_test_458f8018-67e7-4cc6-8126-a535974db35d after 102.373888ms
> [00:21:51]W: [Step 10/10] I1008 00:21:51.824023 23544 cgroups.cpp:2723] Thawing cgroup /sys/fs/cgroup/freezer/mesos_test_458f8018-67e7-4cc6-8126-a535974db35d
> [00:21:51]W: [Step 10/10] I1008 00:21:51.824843 23546 cgroups.cpp:1468] Successfully thawed cgroup /sys/fs/cgroup/freezer/mesos_test_458f8018-67e7-4cc6-8126-a535974db35d after 801024ns
> [00:21:51] : [Step 10/10] [ FAILED ] NestedMesosContainerizerTest.ROOT_CGROUPS_ParentExit (480 ms)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)