You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Wei-Chiu Chuang (Jira)" <ji...@apache.org> on 2026/06/03 07:05:00 UTC
[jira] [Updated] (HDDS-11234) Manage Netty native memory consumption
[ https://issues.apache.org/jira/browse/HDDS-11234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Wei-Chiu Chuang updated HDDS-11234:
-----------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Manage Netty native memory consumption
> --------------------------------------
>
> Key: HDDS-11234
> URL: https://issues.apache.org/jira/browse/HDDS-11234
> Project: Apache Ozone
> Issue Type: Sub-task
> Components: common, documentation
> Reporter: Wei-Chiu Chuang
> Assignee: Siyao Meng
> Priority: Major
> Labels: pull-request-available
> Fix For: 2.2.0
>
> Attachments: Screenshot 2024-07-25 at 10.32.40 AM.png, ozone_dn_leak.pdf
>
>
> We observed an Ozone DataNode that used way too much memory with HBase LoadTest workload.
> Most of its memory allocation was non-heap, so we thought there might be a native memory leak.
> Used jemalloc and jeprof tools, I managed to produce the native memory allocation map, and it shows most of it comes from Unsafe_AllocateMemory. It turns out it's not a memory leak (we were on the default leak detection level 'simple') After some digging, it led me to this post [https://github.com/netty/netty/issues/11835] where it suggests Netty's internal memory management is to blame. A workaround is to disable it (Java property -Dio.netty.allocator.type=unpooled) and another is to reduce the native memory size using -Dio.netty.maxDirectMemory=<size> Disabling it has negative performance impact so I think controlling maximum memory size used by Netty makes more sense.
>
> By default, the size is the same as JDK's maximum direct memory size (-XX:MaxDirectMemorySize), which is usally the same as max heap size (-Xmx). We should provide a best practice for users. In addition, we have Ratis shaded Netty and gRPC. They use different Netty properties to configure memory size (-Dio.netty.maxDirectMemory, -Dorg.apache.ratis.thirdparty.io.netty.maxDirectMemory) So in theory the memory consumption can go up to 3x of maximum heap size.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org