You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@mesos.apache.org by "Gilbert Song (JIRA)" <ji...@apache.org> on 2017/09/22 07:37:00 UTC
[jira] [Issue Comment Deleted] (MESOS-8004) Failed to kill all
processes in the container due to cgroup freeze failure
[ https://issues.apache.org/jira/browse/MESOS-8004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Gilbert Song updated MESOS-8004:
--------------------------------
Comment: was deleted
(was: [~highfly], what's your Mesos version?)
> Failed to kill all processes in the container due to cgroup freeze failure
> --------------------------------------------------------------------------
>
> Key: MESOS-8004
> URL: https://issues.apache.org/jira/browse/MESOS-8004
> Project: Mesos
> Issue Type: Bug
> Components: agent, containerization
> Affects Versions: 1.2.1
> Environment: CentOS Linux release 7.2.1511 (Core)
> 3.10.0-327.36.3.el7.x86_64
> Reporter: Haiwei Zhou
> Labels: launcher
>
> When using Mesos unified container, executor can not be destroyed because cgroup freeze operation failed. The logs from agent show that launcher tries to freeze cgroup several times, then timeout occurs. However, the content of /sys/fs/cgroup/freezer/mesos/e2778ccd-c7e5-4289-b382-e05f063200d8/freezer.state is "FROZEN".
> {quote}
> I0921 18:00:58.339440 3493 containerizer.cpp:2465] Container e2778ccd-c7e5-4289-b382-e05f063200d8 has exited
> I0921 18:00:58.339519 3493 containerizer.cpp:2102] Destroying container e2778ccd-c7e5-4289-b382-e05f063200d8 in RUNNING state
> I0921 18:00:58.339645 3484 linux_launcher.cpp:505] Asked to destroy container e2778ccd-c7e5-4289-b382-e05f063200d8
> I0921 18:00:58.340553 3484 linux_launcher.cpp:548] Using freezer to destroy cgroup mesos/e2778ccd-c7e5-4289-b382-e05f063200d8
> I0921 18:00:58.342226 3493 cgroups.cpp:2692] Freezing cgroup /sys/fs/cgroup/freezer/mesos/e2778ccd-c7e5-4289-b382-e05f063200d8
> I0921 18:01:00.042708 3475 slave.cpp:5155] Killing executor '47eb9350-9ab4-41f8-a5cd-39e855532b53' of framework 23aad131-26f7-44fd-9baa-dfb55e3e3926-0110 at executor(1)@172.29.0.18:40108
> I0921 18:01:02.009097 3483 process.cpp:3704] Handling HTTP event for process 'slave(1)' with path: '/slave(1)/containers'
> W0921 18:01:02.011672 3491 containerizer.cpp:2055] Skipping status for container e2778ccd-c7e5-4289-b382-e05f063200d8 because: Container does not exist
> I0921 18:01:04.269701 3487 slave.cpp:5732] Querying resource estimator for oversubscribable resources
> I0921 18:01:04.269775 3487 slave.cpp:5266] Current disk usage 0.11%. Max allowed age: 6.292478769607581days
> I0921 18:01:04.270349 3506 slave.cpp:5746] Received oversubscribable resources {} from the resource estimator
> I0921 18:01:08.300772 3474 slave.cpp:4346] Received ping from slave-observer(30)@10.16.85.66:5050
> I0921 18:01:08.345176 3517 cgroups.cpp:2710] Thawing cgroup /sys/fs/cgroup/freezer/mesos/e2778ccd-c7e5-4289-b382-e05f063200d8
> I0921 18:01:08.347452 3517 cgroups.cpp:1434] Successfully thawed cgroup /sys/fs/cgroup/freezer/mesos/e2778ccd-c7e5-4289-b382-e05f063200d8 after 2.183168ms
> I0921 18:01:08.347561 3517 cgroups.cpp:2692] Freezing cgroup /sys/fs/cgroup/freezer/mesos/e2778ccd-c7e5-4289-b382-e05f063200d8
> E0921 18:01:15.192441 3524 perf_event.cpp:176] Perf sample of 10secs failed to complete within 12secs; sampling will be halted
> E0921 18:01:15.192819 3489 perf_event.cpp:199] Failed to get the perf sample: timeout
> I0921 18:01:18.350342 3488 cgroups.cpp:2710] Thawing cgroup /sys/fs/cgroup/freezer/mesos/e2778ccd-c7e5-4289-b382-e05f063200d8
> I0921 18:01:18.352532 3488 cgroups.cpp:1434] Successfully thawed cgroup /sys/fs/cgroup/freezer/mesos/e2778ccd-c7e5-4289-b382-e05f063200d8 after 2.121984ms
> I0921 18:01:18.352646 3481 cgroups.cpp:2692] Freezing cgroup /sys/fs/cgroup/freezer/mesos/e2778ccd-c7e5-4289-b382-e05f063200d8
> I0921 18:01:19.301443 3520 slave.cpp:5732] Querying resource estimator for oversubscribable resources
> I0921 18:01:19.301566 3501 slave.cpp:5746] Received oversubscribable resources {} from the resource estimator
> I0921 18:01:23.307291 3518 slave.cpp:4346] Received ping from slave-observer(30)@10.16.85.66:5050
> I0921 18:01:28.121094 3491 process.cpp:3704] Handling HTTP event for process 'metrics' with path: '/metrics/snapshot'
> I0921 18:01:28.355551 3493 cgroups.cpp:2710] Thawing cgroup /sys/fs/cgroup/freezer/mesos/e2778ccd-c7e5-4289-b382-e05f063200d8
> I0921 18:01:28.357792 3493 cgroups.cpp:1434] Successfully thawed cgroup /sys/fs/cgroup/freezer/mesos/e2778ccd-c7e5-4289-b382-e05f063200d8 after 2.177024ms
> I0921 18:01:28.357890 3493 cgroups.cpp:2692] Freezing cgroup /sys/fs/cgroup/freezer/mesos/e2778ccd-c7e5-4289-b382-e05f063200d8
> I0921 18:01:34.302625 3503 slave.cpp:5732] Querying resource estimator for oversubscribable resources
> I0921 18:01:34.302738 3483 slave.cpp:5746] Received oversubscribable resources {} from the resource estimator
> I0921 18:01:38.315979 3505 slave.cpp:4346] Received ping from slave-observer(30)@10.16.85.66:5050
> I0921 18:01:38.360709 3511 cgroups.cpp:2710] Thawing cgroup /sys/fs/cgroup/freezer/mesos/e2778ccd-c7e5-4289-b382-e05f063200d8
> I0921 18:01:38.362891 3511 cgroups.cpp:1434] Successfully thawed cgroup /sys/fs/cgroup/freezer/mesos/e2778ccd-c7e5-4289-b382-e05f063200d8 after 2.12608ms
> I0921 18:01:38.362993 3475 cgroups.cpp:2692] Freezing cgroup /sys/fs/cgroup/freezer/mesos/e2778ccd-c7e5-4289-b382-e05f063200d8
> I0921 18:01:48.366251 3492 cgroups.cpp:2710] Thawing cgroup /sys/fs/cgroup/freezer/mesos/e2778ccd-c7e5-4289-b382-e05f063200d8
> I0921 18:01:48.368404 3496 cgroups.cpp:1434] Successfully thawed cgroup /sys/fs/cgroup/freezer/mesos/e2778ccd-c7e5-4289-b382-e05f063200d8 after 2.080256ms
> I0921 18:01:48.368501 3496 cgroups.cpp:2692] Freezing cgroup /sys/fs/cgroup/freezer/mesos/e2778ccd-c7e5-4289-b382-e05f063200d8
> E0921 18:01:58.342779 3478 slave.cpp:4746] Termination of executor '47eb9350-9ab4-41f8-a5cd-39e855532b53' of framework 23aad131-26f7-44fd-9baa-dfb55e3e3926-0110 failed: Failed to kill all processes in the container: Timed out after 1mins
> I0921 18:01:58.342830 3478 slave.cpp:4868] Cleaning up executor '47eb9350-9ab4-41f8-a5cd-39e855532b53' of framework 23aad131-26f7-44fd-9baa-dfb55e3e3926-0110 at executor(1)@172.29.0.18:40108
> I0921 18:01:58.364516 3475 gc.cpp:55] Scheduling '/data/mesos/slaves/23aad131-26f7-44fd-9baa-dfb55e3e3926-S5/frameworks/23aad131-26f7-44fd-9baa-dfb55e3e3926-0110/executors/47eb9350-9ab4-41f8-a5cd-39e855532b53/runs/e2778ccd-c7e5-4289-b382-e05f063200d8' for gc 6.99999578195556days in the future
> I0921 18:01:58.364591 3475 gc.cpp:55] Scheduling '/data/mesos/slaves/23aad131-26f7-44fd-9baa-dfb55e3e3926-S5/frameworks/23aad131-26f7-44fd-9baa-dfb55e3e3926-0110/executors/47eb9350-9ab4-41f8-a5cd-39e855532b53' for gc 6.9999957811437days in the future
> I0921 18:01:58.364604 3478 slave.cpp:4956] Cleaning up framework 23aad131-26f7-44fd-9baa-dfb55e3e3926-0110
> I0921 18:01:58.364615 3475 gc.cpp:55] Scheduling '/data/mesos/meta/slaves/23aad131-26f7-44fd-9baa-dfb55e3e3926-S5/frameworks/23aad131-26f7-44fd-9baa-dfb55e3e3926-0110/executors/47eb9350-9ab4-41f8-a5cd-39e855532b53/runs/e2778ccd-c7e5-4289-b382-e05f063200d8' for gc 6.99999578062519days in the future
> I0921 18:01:58.364670 3475 gc.cpp:55] Scheduling '/data/mesos/meta/slaves/23aad131-26f7-44fd-9baa-dfb55e3e3926-S5/frameworks/23aad131-26f7-44fd-9baa-dfb55e3e3926-0110/executors/47eb9350-9ab4-41f8-a5cd-39e855532b53' for gc 6.99999578024296days in the future
> I0921 18:01:58.364683 3479 status_update_manager.cpp:285] Closing status update streams for framework 23aad131-26f7-44fd-9baa-dfb55e3e3926-0110
> I0921 18:01:58.364702 3475 gc.cpp:55] Scheduling '/data/mesos/slaves/23aad131-26f7-44fd-9baa-dfb55e3e3926-S5/frameworks/23aad131-26f7-44fd-9baa-dfb55e3e3926-0110' for gc 6.9999957791437days in the future
> I0921 18:01:58.364725 3479 status_update_manager.cpp:531] Cleaning up status update stream for task 47eb9350-9ab4-41f8-a5cd-39e855532b53 of framework 23aad131-26f7-44fd-9baa-dfb55e3e3926-0110
> I0921 18:01:58.364740 3475 gc.cpp:55] Scheduling '/data/mesos/meta/slaves/23aad131-26f7-44fd-9baa-dfb55e3e3926-S5/frameworks/23aad131-26f7-44fd-9baa-dfb55e3e3926-0110' for gc 6.99999577881778days in the future
> {quote}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)