[jira] [Resolved] (HDDS-14106) Set -XX:NewRatio=3 explicitly when using ConcurrentMarkSweep GC

["Attila Doroszlai (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HDDS-14106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Attila Doroszlai resolved HDDS-14106.
-------------------------------------
    Fix Version/s: 2.2.0
       Resolution: Implemented

> Set -XX:NewRatio=3 explicitly when using ConcurrentMarkSweep GC
> ---------------------------------------------------------------
>
>                 Key: HDDS-14106
>                 URL: https://issues.apache.org/jira/browse/HDDS-14106
>             Project: Apache Ozone
>          Issue Type: Bug
>    Affects Versions: 1.4.0, 2.0.0, 1.4.1, 2.1.0
>            Reporter: Ryan Blough
>            Assignee: Ryan Blough
>            Priority: Minor
>              Labels: pull-request-available
>             Fix For: 2.2.0
>
>
> The Young Generation heap size is chronically too low when using the ConcurrentMarkSweep garbage collector.
> This is actually expected behavior, being present in openJDK since JDK6 (earliest I found was this openJDK jira: [https://bugs.openjdk.org/browse/JDK-6872335], so predating Ozone). The root cause is that instead of honoring the default NewRatio size of 2, an ergonomic value is calculated based on the number of ParallelGC threads and the value of the CMSYoungGenPerWorker flag, plus some other adjustments. The original goal was to ensure that the ParallelGC run never went long (more than a second, say); but the logic was written when 4 was a high number of cores to have available on a server.
> In light of modern hardware, this logic is entirely obsolete. The known solution is to explicitly set the NewRatio value in the Java options, overriding the ergonomic calculation.
> I recommend a value of XX:NewRatio=3, resulting in a young generation heap size of 25% of the total heap size. I choose NewRatio=3 because Ozone operations do not appear to require a full 33% of the heap for Young Gen activity, and because we don't want to reduce the size of the reminder of the heap unduly during an upgrade or similar.
> Importantly we should only set this when -XX:+UseConcMarkSweep is _also_ set; this problem does not impact other GC algorithms, and some of them (including G1GC, as I understand it) rely on dynamically adjusting the Young Generation as part of the algorithm.
> Using NewRatio for our defaults is preferred to NewSize and MaxNewSize because it dynamically adjusts with the total heap size changing, instead of requiring manual Java option changes every time.
> For context this problem was observed on a cluster approaching 900 nodes and ~70 petabytes, running on Azul Zulu JDK11. Flight Recorder data revealed that a maximum heap size of 92GB had a maximum Young Generation size of 3.5GB set for the Ozone Manager which was experiencing long GC pauses. The problem seemed to be Young Generation heap thrashing, prematurely tenuring objects until the old generation was full and driving spurious full GC pauses. Adding -XX:NewRatio=3 resolved the problem.
> To reproduce the behavior, increase the heap size on any Ozone cluster running on a JDK version where ConcurrentMarkSweep is available (which I believe is any version prior to JDK14). Looking at the Young Generation size compared to the total heap size will show the discrepancy, such as via the jmap command.
> As a historical note, this was also a common GC-related problem in large HDFS clusters; as far as I can tell the root cause never became common knowledge there.
> This is not a very high priority because ConcurrentMarkSweep is not even available in JDK17 or 21, but I believe most working clusters still employ it, and adding a java option is a simple change.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org