You are viewing a plain text version of this content. The canonical link for it is here.
Posted to yarn-issues@hadoop.apache.org by "ASF GitHub Bot (Jira)" <ji...@apache.org> on 2026/07/09 09:37:00 UTC
[jira] [Commented] (YARN-11967) Strict memory enforcement should not skip polling-based memory check when CGroups memory is disabled
[ https://issues.apache.org/jira/browse/YARN-11967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18094946#comment-18094946 ]
ASF GitHub Bot commented on YARN-11967:
---------------------------------------
pan3793 commented on PR #8573:
URL: https://github.com/apache/hadoop/pull/8573#issuecomment-4923671888
@singer-bin, some tests under `Apache Hadoop YARN DistributedShell` break after this PR, could you please take a look? You can refer to the trunk commit history to find the GHA report and logs.
> Strict memory enforcement should not skip polling-based memory check when CGroups memory is disabled
> ----------------------------------------------------------------------------------------------------
>
> Key: YARN-11967
> URL: https://issues.apache.org/jira/browse/YARN-11967
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
> Reporter: yanbin.zhang
> Assignee: yanbin.zhang
> Priority: Major
> Labels: pull-request-available
> Fix For: 3.6.0
>
>
> h3. Problem
> When the CGroups memory controller is *not* enabled
> (\{{yarn.nodemanager.resource.memory.enabled=false}}, the default) and the
> operator relies on the legacy polling-based memory check
> (\{{yarn.nodemanager.pmem-check-enabled=true}}), containers that exceed their
> requested memory are *not* killed. A container that requested e.g. 4 GB (a
> Spark executor) can grow to 20 GB+ without being terminated, eventually
> causing NodeManager / host memory pressure.
> h3. Root cause
> {\{ContainersMonitorImpl#checkLimit()}} short-circuits the polling-based
> pmem/vmem check whenever strict memory enforcement is on and elastic memory
> control is off:
> {code:java}
> if (strictMemoryEnforcement && !elasticMemoryEnforcement) {
> // the oom-kill would take care of it.
> return;
> }
> {code}
> This skip logic was introduced by YARN-8930. The intent is correct: when
> CGroups-based strict memory enforcement is active, the kernel OOM killer
> enforces the hard limit, so the polling check is redundant.
> The problem is that \{{strictMemoryEnforcement}} is derived *only* from
> {\{yarn.nodemanager.resource.memory.enforced}}, whose default is \{{true}}:
> {code:java}
> strictMemoryEnforcement = conf.getBoolean(
> YarnConfiguration.NM_MEMORY_RESOURCE_ENFORCED, // default: true
> YarnConfiguration.DEFAULT_NM_MEMORY_RESOURCE_ENFORCED);
> {code}
> However, the CGroups memory hard limit is *only written* when
> {\{yarn.nodemanager.resource.memory.enabled}} is \{{true}} (default \{{false}}),
> as seen in \{{ResourceHandlerModule#initMemoryResourceHandler()}}:
> {code:java}
> if (conf.getBoolean(NM_MEMORY_RESOURCE_ENABLED,
> DEFAULT_NM_MEMORY_RESOURCE_ENABLED)) {
> return getCgroupsMemoryResourceHandler(conf); // writes the hard limit
> }
> return null;
> {code}
> So with the default configuration (\{{memory.enabled=false}},
> {\{memory.enforced=true}}):
> ||Mechanism||State||Result||
> |CGroups hard limit + OOM killer|not written (memory controller disabled)|not enforced|
> |Polling pmem/vmem check|short-circuited because strictMemoryEnforcement==true|skipped|
> => *No mechanism limits container memory.* This is inconsistent: the CGroups
> memory handler itself is correctly gated by \{{memory.enabled}}, but the
> monitor's skip logic is not.
> h3. Reproduce
> # \{{yarn.nodemanager.container-executor.class}} = \{{DefaultContainerExecutor}}
> (or LCE without enabling CGroups memory).
> # Do *not* set \{{yarn.nodemanager.resource.memory.enabled}} (defaults to \{{false}}).
> # \{{yarn.nodemanager.pmem-check-enabled=true}}.
> # Leave \{{yarn.nodemanager.resource.memory.enforced}} at its default (\{{true}}).
> # Launch a container (e.g. a Spark executor with 4 GB) that allocates far more
> memory than requested.
> # Expected: container killed with \{{KILLED_EXCEEDED_PMEM}}. Actual: the
> container keeps growing and is never killed.
> h3. Fix
> Gate \{{strictMemoryEnforcement}} on *both* \{{memory.enabled}} and
> {\{memory.enforced}}, so the polling check is skipped only when CGroups memory
> enforcement is actually in effect.
> h3. Impact / Compatibility
> * Clusters that already enabled CGroups memory (\{{memory.enabled=true}},
> {\{memory.enforced=true}}) are *unaffected* -- behaviour is identical (skip
> polling, rely on the OOM killer).
> * Clusters that did *not* enable CGroups memory but rely on pmem/vmem polling
> now get the protection they expect.
> * No configuration change required; the default value of \{{memory.enforced}}
> remains \{{true}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org