You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by GitBox <gi...@apache.org> on 2020/03/04 09:53:57 UTC
[GitHub] [hadoop-ozone] runzhiwang opened a new pull request #611:
HDDS-3041. Fix the memory leak of s3g by releasing the connection resource
runzhiwang opened a new pull request #611: HDDS-3041. Fix the memory leak of s3g by releasing the connection resource
URL: https://github.com/apache/hadoop-ozone/pull/611
## What changes were proposed in this pull request?
1. s3g creates client for each request at https://github.com/apache/hadoop-ozone/blob/master/hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/OzoneClientProducer.java#L66, but does not release the connection resource when destroy the client. So memory leak happens. The details of memory leak are as follows.
2. When Physical memory of s3g is 10GB, jmap shows memory leak happens on jvm heap.

3. Restart s3g, dump the heap when physical memory cost 3GB, the image shows there are 262144 `InternalSubchannel` in `subchannels`, each `InternalSubchannel` in grpc-java represents a connection, so many connections means the connection resource was not released when destroy client.

4. However if with this PR, s3g will occupy all the cpu, as the image shows the cpu cost of s3g increases to 2381% in the machine with 24 cores.

5. When use jstack, I find a lot of threads which cost 100% cpu are in the state: RUNNABLE at ScheduledThreadPoolExecutor.java:809, and this is a bug of java8: https://bugs.openjdk.java.net/browse/JDK-8129861. Then I compile and run ozone with java9, the cpu cost is normal, so I'm almost sure it's the bug.

6. Then I find the related code with the bug JDK-8129861 is in ratis: https://github.com/apache/incubator-ratis/blob/master/ratis-common/src/main/java/org/apache/ratis/util/TimeoutScheduler.java#L88. The code init a thread pool with size zero, then` client.close` in ozone trigger the bug. The trigger stack is as the image shows.

7. If this PR work together with the ratis PR: https://github.com/apache/incubator-ratis/pull/56, the memory and cpu of s3g all will be normal.
8. So if want to fix memory leak of s3g, there are 3 steps: 1. Merge the ratis PR: https://github.com/apache/incubator-ratis/pull/56 and release a new ratis version. Because there is a duplicated jira RATIS-814 with a same fix was merged at Mar 2, 2020, so I close my ratis jira; 2. Upgrade the ratis version in ozone 3. merge this PR.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-3041
The jira was created by JieWang, JieWang is also my username. I forget the password of username runzhiwang in jira and has not find it back, sorry for that.
## How was this patch tested?
1. Compile ratis-0.5.0-rc0 with PR: https://github.com/apache/incubator-ratis/pull/56.
2. Compile ozone with this PR.
3. Replace ozone-0.5.0-SNAPSHOT/share/ozone/lib/ratis-common-0.5.0.jar with ratis-common/target/ratis-common-0.5.0.jar.
4. Make a stress test on s3g: send total 4 million requests by 40 threads to s3g. The physical memory of s3g is stable at 1.5 G.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org