You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Sean Chow (Jira)" <ji...@apache.org> on 2020/08/17 03:27:00 UTC
[jira] [Comment Edited] (HADOOP-17209) ErasureCode native library
memory leak
[ https://issues.apache.org/jira/browse/HADOOP-17209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17178227#comment-17178227 ]
Sean Chow edited comment on HADOOP-17209 at 8/17/20, 3:26 AM:
--------------------------------------------------------------
Patch attached!
After running for a week, the memory is steady.
!image-2020-08-17-11-26-04-276.png!
was (Author: seanlook):
Patch attached!
After running for a week, the memory is steady.
> ErasureCode native library memory leak
> --------------------------------------
>
> Key: HADOOP-17209
> URL: https://issues.apache.org/jira/browse/HADOOP-17209
> Project: Hadoop Common
> Issue Type: Bug
> Components: native
> Affects Versions: 3.3.0, 3.2.1, 3.1.3
> Reporter: Sean Chow
> Assignee: Sean Chow
> Priority: Major
> Attachments: HADOOP-17209.001.patch, datanode.202137.detail_diff.5.txt, image-2020-08-15-18-26-44-744.png, image-2020-08-17-11-26-04-276.png
>
>
> We use both {{apache-hadoop-3.1.3}} and {{CDH-6.1.1-1.cdh6.1.1.p0.875250}} HDFS in production, and both of them have the memory increasing over {{-Xmx}} value.
> !image-2020-08-15-18-26-44-744.png!
>
> We use EC strategy to to save storage costs.
> This's the jvm options:
> {code:java}
> -Dproc_datanode -Dhdfs.audit.logger=INFO,RFAAUDIT -Dsecurity.audit.logger=INFO,RFAS -Djava.net.preferIPv4Stack=true -Xms8589934592 -Xmx8589934592 -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 -XX:+CMSParallelRemarkEnabled -XX:+HeapDumpOnOutOfMemoryError ...{code}
> The max jvm heapsize is 8GB, but we can see the datanode RSS memory is 48g. All the other datanodes in this hdfs cluster has the same issue.
> {code:java}
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> 226044 hdfs 20 0 50.6g 48g 4780 S 90.5 77.0 14728:27 /usr/java/jdk1.8.0_162/bin/java -Dproc_datanode{code}
>
> This too much memory used leads to my machine unresponsive(if enable swap), or oom-killer happens.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org