You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "slfan1989 (via GitHub)" <gi...@apache.org> on 2024/07/06 01:45:56 UTC
[PR] HDDS-11100. OM/SCM Metrics support displaying Netty off-heap memory. [ozone]
slfan1989 opened a new pull request, #6900:
URL: https://github.com/apache/ozone/pull/6900
## What changes were proposed in this pull request?
During a recent period, our cluster's DataNode (DN) exhibited high off-heap memory utilization, triggering frequent system.gc() calls. Upon reading relevant GC logs, we observed normal heap memory usage and monitored heap metrics. It became apparent that off-heap memory usage might be causing the system.gc() calls. We identified that DataNode can display off-heap memory usage, which indirectly helped us infer potential off-heap memory overflow. In [HDDS-9070](https://issues.apache.org/jira/browse/HDDS-9070), we now support displaying Netty off-heap memory usage on DataNode, facilitating off-heap memory monitoring. Similarly, we can display Netty off-heap memory usage on OM and SCM.
Examples of well-written pull requests:
* [#3980](https://github.com/apache/ozone/pull/3980)
* [#5265](https://github.com/apache/ozone/pull/5265)
* [#4701](https://github.com/apache/ozone/pull/4701)
* [#5283](https://github.com/apache/ozone/pull/5283)
* [#5300](https://github.com/apache/ozone/pull/5300)
## What is the link to the Apache JIRA
Please create an issue in ASF JIRA before opening a pull request, and you need to set the title of the pull
request which starts with the corresponding JIRA issue number. (e.g. HDDS-XXXX. Fix a typo in YYY.)
(Please replace this section with the link to the Apache JIRA)
## How was this patch tested?
(Please explain how this patch was tested. Ex: unit tests, manual tests, workflow run on the fork git repo.)
(If this patch involves UI changes, please attach a screenshot; otherwise, remove this.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11100. OM/SCM Metrics support displaying Netty off-heap memory. [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on PR #6900:
URL: https://github.com/apache/ozone/pull/6900#issuecomment-2225903109
Thanks @slfan1989 for the patch, @duongkame, @kerneltime for the review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11100. OM/SCM Metrics support displaying Netty off-heap memory. [ozone]
Posted by "slfan1989 (via GitHub)" <gi...@apache.org>.
slfan1989 commented on PR #6900:
URL: https://github.com/apache/ozone/pull/6900#issuecomment-2226541415
@adoroszlai @duongkame @kerneltime Thank you for reviewing the code and helping to merge it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11100. OM/SCM Metrics support displaying Netty off-heap memory. [ozone]
Posted by "kerneltime (via GitHub)" <gi...@apache.org>.
kerneltime commented on PR #6900:
URL: https://github.com/apache/ozone/pull/6900#issuecomment-2214799434
@slfan1989 can you also update the details for how was this patch tested?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11100. OM/SCM Metrics support displaying Netty off-heap memory. [ozone]
Posted by "slfan1989 (via GitHub)" <gi...@apache.org>.
slfan1989 commented on code in PR #6900:
URL: https://github.com/apache/ozone/pull/6900#discussion_r1669454949
##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/StorageContainerManager.java:
##########
@@ -1710,6 +1713,10 @@ public void stop() {
metrics.unRegister();
}
+ if (this.nettyMetrics != null) {
+ this.nettyMetrics.unregister();
+ }
+
Review Comment:
Thank you very much for reviewing this pr! I will improve the code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11100. OM/SCM Metrics support displaying Netty off-heap memory. [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai merged PR #6900:
URL: https://github.com/apache/ozone/pull/6900
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11100. OM/SCM Metrics support displaying Netty off-heap memory. [ozone]
Posted by "slfan1989 (via GitHub)" <gi...@apache.org>.
slfan1989 commented on PR #6900:
URL: https://github.com/apache/ozone/pull/6900#issuecomment-2221665880
In our Ozone cluster, we have been continuously troubled by DN memory issues. Below is our DN memory configuration:
```
-Xms64g -Xmx64g -Xmn16g -XX:MaxDirectMemorySize=32g
```
We may wonder why we configured such large memory settings. This is because our Ozone cluster is very large, with the largest cluster currently having more than 1K nodes, and the DN stores a large amount of data. Historically, we have encountered OOM (Out of Memory) issues, so we increased the memory allocation.
However, during usage, we encounter situations where users experience data access timeouts. For example, downloading a few megabytes of data can take 20-30 seconds or even longer. While analyzing this issue, we focused on the GC (Garbage Collection) logs.
In the GC logs, we can observe a significant pattern where a large number of system.gc() calls occur within a short period. Initially, we thought that these calls were triggered by certain conditions within a third-party package. It wasn't until we captured the relevant stack traces for system.gc() .
```
groupManagement;id=6d;is_daemon=false;priority=5;TCCL=sun.misc.Launcher$AppClassLoader@214c265e
@java.lang.System.gc()
at java.nio.Bits.reserveMemory(Bits.java:666)
at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123)
at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311)
at org.apache.ratis.server.raftlog.segmented.SegmentedRaftLogWorker.<init>(SegmentedRaftLogWorker.java:208)
at org.apache.ratis.server.raftlog.segmented.SegmentedRaftLog.<init>(SegmentedRaftLog.java:217)
at org.apache.ratis.server.raftlog.segmented.SegmentedRaftLog.<init>(SegmentedRaftLog.java:86)
at org.apache.ratis.server.raftlog.segmented.SegmentedRaftLog$Builder.build(SegmentedRaftLog.java:608)
at org.apache.ratis.server.impl.ServerState.initRaftLog(ServerState.java:189)
at org.apache.ratis.server.impl.ServerState.initRaftLog(ServerState.java:166)
at org.apache.ratis.server.impl.ServerState.lambda$new$6(ServerState.java:130)
at org.apache.ratis.util.MemoizedSupplier.get(MemoizedSupplier.java:62)
at org.apache.ratis.server.impl.ServerState.initialize(ServerState.java:147)
at org.apache.ratis.server.impl.RaftServerImpl.start(RaftServerImpl.java:392)
at org.apache.ratis.server.impl.RaftServerProxy.lambda$groupAddAsync$13(RaftServerProxy.java:509)
at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)
at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
at java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:442)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
```
The stack trace java.nio.Bits.reserveMemory(Bits.java:666) indicates that we are experiencing an out-of-heap memory issue while attempting to allocate off-heap memory, which triggers system.gc() calls to try and free up memory. This results in a cycle of repeated system.gc() calls, explaining why a large number of these calls are being triggered in a short period. After identifying this issue, we dumped the DN heap and indeed found a large number of DirectByteBuffer instances.
Our team members also analyzed the off-heap memory using GDB and found a large number of Netty objects. At this point, we wondered if there were any metrics that could quickly show the usage of off-heap memory. I researched JMX and found that [HDDS-9070](https://issues.apache.org/jira/browse/HDDS-9070) can display Netty's off-heap memory usage. We configured this metric in Grafana.
This metric effectively showcases the usage of off-heap memory. Both OM and SCM use Ratis and thus also utilize off-heap memory. I believe that displaying this metric in both OM and SCM would provide more tools for diagnosing memory issues. Therefore, I attempted to add this off-heap memory monitoring metric to OM and SCM.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11100. OM/SCM Metrics support displaying Netty off-heap memory. [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on code in PR #6900:
URL: https://github.com/apache/ozone/pull/6900#discussion_r1667347706
##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/StorageContainerManager.java:
##########
@@ -1710,6 +1713,10 @@ public void stop() {
metrics.unRegister();
}
+ if (this.nettyMetrics != null) {
+ this.nettyMetrics.unregister();
+ }
+
Review Comment:
It seems these lines make `stop()` too long for checkstyle's limits:
```
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/StorageContainerManager.java
1634: Method stop length is 151 lines (max allowed is 150).
```
https://github.com/apache/ozone/actions/runs/9815585695/job/27104790904?pr=6900#step:8:11
Since `nettyMetrics` is `final` and created in the constructor, I think we can omit the null check to satisfy checkstyle.
```suggestion
nettyMetrics.unregister();
```
Alternatively, if you have more time for this, please extract all metrics-related cleanup into a separate method to reduce length of `stop()`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org