You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by GitBox <gi...@apache.org> on 2020/04/30 06:30:56 UTC
[GitHub] [hadoop-ozone] runzhiwang opened a new pull request #885: HDDS-3514. Fix memory leak of RaftServerImpl
runzhiwang opened a new pull request #885:
URL: https://github.com/apache/hadoop-ozone/pull/885
## What changes were proposed in this pull request?
This PR depends on [RATIS-845](https://github.com/apache/incubator-ratis/pull/76), we should wait
RATIS-845 to merge.
As the image shows, there are 1885 instances of RaftServerImpl, most of them are Closed, and should be GC, but actually not. You can find from the image
1513 RaftServerImpl were held by ManagermentFactory->jxmMBeanServer->HashMap, 372 RaftServerImpl were held by Datanode ReportManager Thread -> prometheus -> HashMap. So 1513 RaftServerImpl leak in ratis, and 372 leak in ozone. If RaftServerImpl can not GC, there are a lot of related resource can not be GC, such as the [DirectByteBuffer](https://github.com/apache/incubator-ratis/blob/master/ratis-server/src/main/java/org/apache/ratis/server/raftlog/segmented/SegmentedRaftLogWorker.java#L150) in SegmentRaftLogWorker, which result 1GB memory leak out of heap.
**1. 1885 instances of RaftServerImpl**

**2. 1513 RaftServerImpl were held by ManagermentFactory->jxmMBeanServer->HashMap, 372 RaftServerImpl were held by Datanode ReportManager Thread -> prometheus -> HashMap**

**3. 1513 RaftServerImpl were held by ManagermentFactory->jxmMBeanServer->HashMap**

**4. 372 RaftServerImpl were held by Datanode ReportManager Thread -> prometheus -> HashMap**

5. 2038 DirectByteBuffer, and 1885 held by RaftServerImpl

6. 1033 DirectByteBuffer were held by ManagermentFactory, 802 DirectByteBuffer were held by Datanode ReportManager Thread, total 1885.

7. The reason RaftServerImpl held by ManagermentFactory->jxmMBeanServer->HashMap is ratis start [JmxReporter](https://github.com/apache/incubator-ratis/blob/master/ratis-metrics/src/main/java/org/apache/ratis/metrics/MetricsReporting.java#L47), but does not stop it.
8. The reason RaftServerImpl held by Datanode ReportManager Thread -> prometheus -> HashMap is ozone call the ratis function to [register](https://github.com/apache/hadoop-ozone/blob/master/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/HddsDatanodeService.java#L189) metric in prometheus, but does not unregister it.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-3514
## How was this patch tested?
ungister prometheus and stopJmxReporter when removed in ratis.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] runzhiwang closed pull request #885: HDDS-3514. Fix memory leak of RaftServerImpl
Posted by GitBox <gi...@apache.org>.
runzhiwang closed pull request #885:
URL: https://github.com/apache/hadoop-ozone/pull/885
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] adoroszlai commented on pull request #885: HDDS-3514. Fix memory leak of RaftServerImpl
Posted by GitBox <gi...@apache.org>.
adoroszlai commented on pull request #885:
URL: https://github.com/apache/hadoop-ozone/pull/885#issuecomment-634574304
/pending needs new Ratis snapshot (`b3cfec0e` or later) in Ozone
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] runzhiwang commented on pull request #885: HDDS-3514. Fix memory leak of RaftServerImpl
Posted by GitBox <gi...@apache.org>.
runzhiwang commented on pull request #885:
URL: https://github.com/apache/hadoop-ozone/pull/885#issuecomment-631328275
/ready
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] arp7 commented on pull request #885: HDDS-3514. Fix memory leak of RaftServerImpl
Posted by GitBox <gi...@apache.org>.
arp7 commented on pull request #885:
URL: https://github.com/apache/hadoop-ozone/pull/885#issuecomment-641468682
@bshashikant can you rebase the patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] avijayanhwx commented on pull request #885: HDDS-3514. Fix memory leak of RaftServerImpl
Posted by GitBox <gi...@apache.org>.
avijayanhwx commented on pull request #885:
URL: https://github.com/apache/hadoop-ozone/pull/885#issuecomment-629542686
/pending
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] runzhiwang commented on pull request #885: HDDS-3514. Fix memory leak of RaftServerImpl
Posted by GitBox <gi...@apache.org>.
runzhiwang commented on pull request #885:
URL: https://github.com/apache/hadoop-ozone/pull/885#issuecomment-631368992
@avijayanhwx Could you help review it ? Thank you very much.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] avijayanhwx commented on pull request #885: HDDS-3514. Fix memory leak of RaftServerImpl
Posted by GitBox <gi...@apache.org>.
avijayanhwx commented on pull request #885:
URL: https://github.com/apache/hadoop-ozone/pull/885#issuecomment-624295814
I am working with @runzhiwang on reviewing the ratis changes in RATIS-845. Once that is done, I will review this as well.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] runzhiwang commented on pull request #885: HDDS-3514. Fix memory leak of RaftServerImpl
Posted by GitBox <gi...@apache.org>.
runzhiwang commented on pull request #885:
URL: https://github.com/apache/hadoop-ozone/pull/885#issuecomment-641648535
duplicated with https://issues.apache.org/jira/browse/HDDS-3564. No need to merge.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org