You are viewing a plain text version of this content. The canonical link for it is here.
Posted to yarn-issues@hadoop.apache.org by "Miklos Szegedi (JIRA)" <ji...@apache.org> on 2018/04/30 17:47:00 UTC
[jira] [Commented] (YARN-4599) Set OOM control for memory cgroups
[ https://issues.apache.org/jira/browse/YARN-4599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16458806#comment-16458806 ]
Miklos Szegedi commented on YARN-4599:
--------------------------------------
I will provide the patch shortly. Here are the design suggestions of the patch:
* The basic idea is what was discussed above. It disables the OOM killer on the hadoop-yarn cgroup. This will trigger a pause on all containers when all of them exceeded the node limit.
* YARN will be notified with an executable listening to the cgroups OOM Linux event. This should be very fast. The executable is oom-listener, not container-executor. This is because it does not need to run as root. I avoided JNI to be more defensive on security and it also helps to test the executable easier.
* When YARN receives the notification, it runs a pluggable OOM handler to resolve the situation. YARN is outside the hadoop-yarn group, so it can run freely, all containers are frozen at this point. Different users may have different preferences, thus the handler is pluggable.
* The default OOM handler picks the latest container that ran out of it's request. This ensures that it kills a container that did not cost much so far but it keeps guaranteed containers that play by the rules and use memory within their limits. It repeats the process until the OOM is resolved. Based on my experiments the kernel updates the flag almost instantaneously, so it will just kill as many containers as necessary.
* If the default OOM handler cannot pick a container with the logic above it kills the latest container until the OOM is resolved.
* If we are still in OOM without any containers, an exception is thrown and the node is brought down. This can be the case, if containers leaked processes, had processes running with another user and cannot be killed the container user or someone put a process into the root hadoop-yarn cgroup.
* The killer is not the normal container cleanup code. The standard behaviour is to send a SIGTERM to the container PGID process and if it does not respond in 250 milliseconds, it sends a SIGKILL. However, in our case all the processes are frozen by cgroups, so that they cannot respond to a SIGTERM. Because of this it uses the standard container executor code to send a SIGKILL to the PGID right away with the container user. The kernel OOM killer would do the same. This works pretty fast. It walks through all the thread/process IDs in the tasks file, so that all active PGIDs are found in the container. The current code does not delete standalone processes that are not a process group leader. If they are not part of one of the container local process groups they may be leaked. It also cannot handle processes that are running as different users in than the container user in the cgroup of the container. This should be rare.
* The code adds a watchdog to measure the time to resolve an OOM situation. The time to resolve an OOM situation takes 10-160 milliseconds based on my experiments.
* The patch contains documentation to set up and troubleshoot the feature.
* I was able to test it manually but I did not do large scale and longhaul tests, yet.
> Set OOM control for memory cgroups
> ----------------------------------
>
> Key: YARN-4599
> URL: https://issues.apache.org/jira/browse/YARN-4599
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager
> Affects Versions: 2.9.0
> Reporter: Karthik Kambatla
> Assignee: Miklos Szegedi
> Priority: Major
> Labels: oct16-medium
> Attachments: YARN-4599.sandflee.patch, yarn-4599-not-so-useful.patch
>
>
> YARN-1856 adds memory cgroups enforcing support. We should also explicitly set OOM control so that containers are not killed as soon as they go over their usage. Today, one could set the swappiness to control this, but clusters with swap turned off exist.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org