[jira] [Commented] (YARN-11991) CGroupsV2ResourceCalculator reports wrong virtual memory size (swap only instead of anon + swap)

["ASF GitHub Bot (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/YARN-11991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18121898#comment-18121898 ] 

ASF GitHub Bot commented on YARN-11991:
---------------------------------------

Abhi0505-kinagi commented on PR #8764:
URL: https://github.com/apache/hadoop/pull/8764#issuecomment-5949053571

   Thanks for the review @joseluisll . I've pushed an update, squashed into one commit titled `YARN-11991. Fix counting virtual memory in cgroup v2`.
   
   ### Blocking items
   - [x] Removed both `.save` files.
   - [x] Fixed the indentation on `getVirtualMemorySize`, the `calculateVirtualMemory` overrides and the abstract declaration, and added a blank line after the constructor. Checkstyle shows no new violations in the touched files.
   - [x] Squashed the commits and restored the title `YARN-11991. Fix counting virtual memory in cgroup v2`.
   
   ### Suggestions
   - [x] Added javadoc on `calculateVirtualMemory()`.
   - [ ] Missing-swap test: not in this push. I'll follow up on it separately.
   - [ ] Release note: not in this push. Suggested text, if a committer could add it to the Release Note field on YARN-11991: "Clusters that configure `CGroupsV2ResourceCalculator` directly with `yarn.nodemanager.vmem-check-enabled=true` now compare anon + swap against `pmem * vmem-pmem-ratio`. Before, only swap was counted, so the check almost never fired. The default `CombinedResourceCalculator` is unaffected."
   
   ### Testing
   Ran all tests in the `linux/resources` package locally:
   
   | Result | Count |
   |---|---|
   | Tests run | 57 |
   | Failures | 0 |
   | Errors | 0 |
   | Skipped | 1 (`TestCompareResourceCalculators`, no cgroup environment here) |




>  CGroupsV2ResourceCalculator reports wrong virtual memory size (swap only instead of anon + swap)
> -------------------------------------------------------------------------------------------------
>
>                 Key: YARN-11991
>                 URL: https://issues.apache.org/jira/browse/YARN-11991
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: yarn
>    Affects Versions: 3.5.0
>            Reporter: Milan Vymazal
>            Priority: Minor
>              Labels: pull-request-available
>
> On hosts using cgroup v2, CGroupsV2ResourceCalculator#getVirtualMemorySize()
> returns only the swap usage (memory.swap.current) instead of the real
> virtual memory of the container process tree.
> AbstractCGroupsResourceCalculator takes a single "virtualMemoryKey" and
> reads it directly:
> {code:java}
> getVirtualMemorySize() -> getStat(virtualMemoryKey)
> {code}
> This is correct for cgroup v1, where memory.memsw.usage_in_bytes already
> contains memory + swap in one counter.
> cgroup v2 has no combined memory+swap counter. The v2 calculator maps
> MEMSW_STAT to memory.swap.current, which is swap usage only. The resident
> part (memory.stat#anon) is never added, so the reported virtual memory is
> much lower than reality.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org