You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@mesos.apache.org by "Charles Natali (Jira)" <ji...@apache.org> on 2020/04/18 09:55:00 UTC
[jira] [Created] (MESOS-10119) failure to destroy container can
cause the agent to "leak" a GPU
Charles Natali created MESOS-10119:
--------------------------------------
Summary: failure to destroy container can cause the agent to "leak" a GPU
Key: MESOS-10119
URL: https://issues.apache.org/jira/browse/MESOS-10119
Project: Mesos
Issue Type: Task
Components: agent, containerization
Reporter: Charles Natali
At work we hit the following problem:
# cgroup for a task using the GPU isolation failed to be destroy on OOM
# the agent continued advertising the GPU as available
# all subsequent attempts to start tasks using a GPU fails with "Requested 1 gpus but only 0 available"
Problem 1 looks like https://issues.apache.org/jira/browse/MESOS-9950) so can be tackled separately, however the fact that the agent basically leaks the GPU is pretty bad, because it basically turns into /dev/null, failing all subsequent tasks requesting a gpu
See the logs:
{noformat}
Apr 17 17:00:03 engpuc006 mesos-slave[2068]: E0417 17:00:03.874277 2138 memory.cpp:665] Failed to read 'memory.limit_in_bytes': No such file or directory
Apr 17 17:00:03 engpuc006 mesos-slave[2068]: E0417 17:00:03.874305 2138 memory.cpp:674] Failed to read 'memory.max_usage_in_bytes': No such file or directory
Apr 17 17:00:03 engpuc006 mesos-slave[2068]: E0417 17:00:03.874315 2138 memory.cpp:686] Failed to read 'memory.stat': No such file or directory
Apr 17 17:00:03 engpuc006 mesos-slave[2068]: E0417 17:00:03.874701 2136 memory.cpp:665] Failed to read 'memory.limit_in_bytes': No such file or directory
Apr 17 17:00:03 engpuc006 mesos-slave[2068]: E0417 17:00:03.874734 2136 memory.cpp:674] Failed to read 'memory.max_usage_in_bytes': No such file or directory
Apr 17 17:00:03 engpuc006 mesos-slave[2068]: E0417 17:00:03.874747 2136 memory.cpp:686] Failed to read 'memory.stat': No such file or directory
Apr 17 17:00:05 engpuc006 mesos-slave[2068]: E0417 17:00:05.062358 2152 slave.cpp:6994] Termination of executor 'task_0:067b0963-134f-a917-4503-89b6a2a630ac' of framework c0c4ce82-5cff-4116-aacb-c3fd6a93d61b-0000 failed: Failed to clean up an isolator when destroying container: Failed to destroy cgroups: Failed to get nested cgroups: Failed to determine canonical path of '/sys/fs/cgroup/memory/mesos/8ef00748-b640-4620-97dc-f719e9775e88': No such file or directory
Apr 17 17:00:05 engpuc006 mesos-slave[2068]: W0417 17:00:05.063295 2150 containerizer.cpp:2567] Skipping status for container 8ef00748-b640-4620-97dc-f719e9775e88 because: Container does not exist
Apr 17 17:00:05 engpuc006 mesos-slave[2068]: W0417 17:00:05.063429 2137 containerizer.cpp:2428] Ignoring update for currently being destroyed container 8ef00748-b640-4620-97dc-f719e9775e88
Apr 17 17:00:05 engpuc006 mesos-slave[2068]: E0417 17:00:05.079169 2150 slave.cpp:6994] Termination of executor 'task_1:a00165a1-123a-db09-6b1a-b6c4054b0acd' of framework c0c4ce82-5cff-4116-aacb-c3fd6a93d61b-0000 failed: Failed to kill all processes in the container: Failed to remove cgroup 'mesos/5c1418f0-1d4d-47cd-a188-0f4b87e394f2': Failed to remove cgroup '/sys/fs/cgroup/freezer/mesos/5c1418f0-1d4d-47cd-a188-0f4b87e394f2': Device or resource busy
Apr 17 17:00:05 engpuc006 mesos-slave[2068]: W0417 17:00:05.079537 2140 containerizer.cpp:2567] Skipping status for container 5c1418f0-1d4d-47cd-a188-0f4b87e394f2 because: Container does not exist
Apr 17 17:00:05 engpuc006 mesos-slave[2068]: W0417 17:00:05.079670 2136 containerizer.cpp:2428] Ignoring update for currently being destroyed container 5c1418f0-1d4d-47cd-a188-0f4b87e394f2
Apr 17 17:00:07 engpuc006 mesos-slave[2068]: E0417 17:00:07.956969 2136 slave.cpp:6889] Container '87253521-8d39-47ea-b4d1-febe527d230c' for executor 'task_2:8b129d24-70d2-2cab-b2df-c73911954ec3' of framework c0c4ce82-5cff-4116-aacb-c3fd6a93d61b-0000 failed to start: Requested 1 gpus but only 0 available
Apr 17 17:00:07 engpuc006 mesos-slave[2068]: E0417 17:00:07.957670 2149 memory.cpp:637] Listening on OOM events failed for container 87253521-8d39-47ea-b4d1-febe527d230c: Event listener is terminating
Apr 17 17:00:07 engpuc006 mesos-slave[2068]: W0417 17:00:07.966552 2150 containerizer.cpp:2421] Ignoring update for unknown container 87253521-8d39-47ea-b4d1-febe527d230c
Apr 17 17:00:08 engpuc006 mesos-slave[2068]: W0417 17:00:08.109067 2154 process.cpp:1480] Failed to link to '172.16.22.201:34059', connect: Failed connect: connection closed
Apr 17 17:00:10 engpuc006 mesos-slave[2068]: E0417 17:00:10.310817 2141 slave.cpp:6889] Container '257b45f1-8582-4cb5-8138-454e9697bfe4' for executor 'task_3:6bdd99ca-7a2b-f19c-bbb3-d9478fe8f81e' of framework c0c4ce82-5cff-4116-aacb-c3fd6a93d61b-0000 failed to start: Requested 1 gpus but only 0 available
Apr 17 17:00:10 engpuc006 mesos-slave[2068]: E0417 17:00:10.311614 2141 memory.cpp:637] Listening on OOM events failed for container 257b45f1-8582-4cb5-8138-454e9697bfe4: Event listener is terminating
Apr 17 17:00:10 engpuc006 mesos-slave[2068]: W0417 17:00:10.316352 2149 containerizer.cpp:2421] Ignoring update for unknown container 257b45f1-8582-4cb5-8138-454e9697bfe4
Apr 17 17:00:12 engpuc006 mesos-slave[2068]: E0417 17:00:12.675910 2146 slave.cpp:6889] Container '47f61e7b-8022-4731-93fe-36a356920a4e' for executor 'task_2:9de8db83-39fa-b24a-b5b6-2ef6f1e1713c' of framework c0c4ce82-5cff-4116-aacb-c3fd6a93d61b-0000 failed to start: Requested 1 gpus but only 0 available
Apr 17 17:00:12 engpuc006 mesos-slave[2068]: E0417 17:00:12.676764 2139 memory.cpp:637] Listening on OOM events failed for container 47f61e7b-8022-4731-93fe-36a356920a4e: Event listener is terminating
Apr 17 17:00:12 engpuc006 mesos-slave[2068]: W0417 17:00:12.681100 2150 containerizer.cpp:2421] Ignoring update for unknown container 47f61e7b-8022-4731-93fe-36a356920a4e
{noformat}
I'll see if I can reproduce and debug the cgroup destruction problem and report my findings to https://issues.apache.org/jira/browse/MESOS-9950), however I think that the bigger issue is the fact that the agent leaks the GPU.
I'm wondering which how the following makes sense:
# Consider the failure of destruction a container a fatal error: this turns this byzantine fault into a clean crash failure, which forces the operator to detect it and deal with it.
# Fix the leak by at least not advertising the GPU anymore.
[~bmahler] [~abudnik]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)