[jira] [Commented] (YARN-4599) Set OOM control for memory cgroups

["Karthik Kambatla (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/YARN-4599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15254335#comment-15254335 ] 

Karthik Kambatla commented on YARN-4599:
----------------------------------------

Looked more into this, specifically on how to resume paused tasks. Per [this article|https://lwn.net/Articles/529927]
{noformat}
This operation is only allowed to the top cgroup of a sub-hierarchy.
If OOM-killer is disabled, tasks under cgroup will hang/sleep
in memory cgroup's OOM-waitqueue when they request accountable memory.

For running them, you have to relax the memory cgroup's OOM status by
	* enlarge limit or reduce usage.
To reduce usage,
	* kill some tasks.
	* move some tasks to other group with account migration.
	* remove some files (on tmpfs?)

Then, stopped tasks will work again.

At reading, current status of OOM is shown.
	oom_kill_disable 0 or 1 (if 1, oom-killer is disabled)
	under_oom	 0 or 1 (if 1, the memory cgroup is under OOM, tasks may
				 be stopped.)
{noformat}

Looks like setting OOM control per each task is not particularly useful. We are likely better off setting hard limit for all yarn containers so they don't interfere anything else on the machine. We could disable OOM control on the cgroup corresponding to all yarn containers (not including NM) and if all containers are paused, the NM can decide what tasks to kill. This is particularly useful if we are oversubscribing the node. 

[~aw], [~vvasudev], [~vinodkv] - what do you think? 

> Set OOM control for memory cgroups
> ----------------------------------
>
>                 Key: YARN-4599
>                 URL: https://issues.apache.org/jira/browse/YARN-4599
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: nodemanager
>    Affects Versions: 2.9.0
>            Reporter: Karthik Kambatla
>            Assignee: Karthik Kambatla
>
> YARN-1856 adds memory cgroups enforcing support. We should also explicitly set OOM control so that containers are not killed as soon as they go over their usage. Today, one could set the swappiness to control this, but clusters with swap turned off exist.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)