You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@activemq.apache.org by morefuntang <gi...@git.apache.org> on 2018/07/30 09:02:38 UTC
[GitHub] activemq-artemis pull request #2199: ARTEMIS-1996 MappedSequentialFileFactor...
GitHub user morefuntang opened a pull request:
https://github.com/apache/activemq-artemis/pull/2199
ARTEMIS-1996 MappedSequentialFileFactory may cause DirectByteBuffer off-heap memory leaks
Because of some customization requirements, the readJournalFile method of JournalImpl needs to be calledmultiple times. During the stress test, it was found that almost every 5 hours, the Broker appeared a Full GC.
This is the information about the Full GC in the GC log:
**2018-07-25T12:14:07.323+0800: 10089.523: [Full GC (System.gc()) 6767M->253M(16G), 8.7138691 secs]
[Eden: 632.0M(712.0M)->0.0B(816.0M) Survivors: 104.0M->0.0B Heap: 6767.6M(16.0G)->253.9M(16.0G)], [Metaspace: 36323K->35961K(1083392K)]
[Times: user=2.56 sys=0.42, real=8.71 secs]**
When the Full GC appears, the thread stack is as follows:
**java.lang.System.gc(System.java:993)
java.nio.Bits.reserveMemory(Bits.java:666)
java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123)
java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311)
org.apache.activemq.artemis.core.io.mapped.MappedSequentialFileFactory.newBuffer(MappedSequentialFileFactory.java:109)
org.apache.activemq.artemis.core.journal.impl.JournalImpl.readJournalFile(JournalImpl.java:463)**
By analyzing the stack, it should be that the JVM's off-heap memory cannot be allocated, causing the JVM to call the System.gc() method.
In the Broker, MappedSequentialFileFactory caches off-heap memory through ThreadLocal. Once the thread is evicted by the CompactExecutor(keepalive is 60s) in the Journal, the off-heap memory is "**leaked**". NIOSequentialFileFactory also has the same problem.
My advice is to use netty.ThreadDeathWatcher to monitor the threads of the CompactorExcutor. Once the CompactorExcutor thread finishes, can release the heap memory by executing the callback method in ThreadDeathWatcher.
jira link: _https://issues.apache.org/jira/browse/ARTEMIS-1996_
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/morefuntang/activemq-artemis Branch20180730
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/activemq-artemis/pull/2199.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2199
----
commit 45ad52fd63be35b7aee783c0d4ba94964df0a167
Author: tangpu <mo...@...>
Date: 2018-07-30T08:41:45Z
ARTEMIS-1996 MappedSequentialFileFactory may cause DirectByteBuffer memory leaks
----
---
[GitHub] activemq-artemis issue #2199: ARTEMIS-1996 MappedSequentialFileFactory may c...
Posted by michaelandrepearce <gi...@git.apache.org>.
Github user michaelandrepearce commented on the issue:
https://github.com/apache/activemq-artemis/pull/2199
@franz1981 how long do you need to do 1 and 2?
---
[GitHub] activemq-artemis issue #2199: ARTEMIS-1996 MappedSequentialFileFactory may c...
Posted by franz1981 <gi...@git.apache.org>.
Github user franz1981 commented on the issue:
https://github.com/apache/activemq-artemis/pull/2199
@morefuntang Please check if master has fixed your issue with GC and close this one: it has been addressed on https://github.com/apache/activemq-artemis/pull/2250
With it there isn't any pooled direct buffers anymore, but each compaction will create just one direct ByteBuffer reusing it for the whole process and releasing it deterministically when finished :+1:
---
[GitHub] activemq-artemis issue #2199: ARTEMIS-1996 MappedSequentialFileFactory may c...
Posted by franz1981 <gi...@git.apache.org>.
Github user franz1981 commented on the issue:
https://github.com/apache/activemq-artemis/pull/2199
@michaelandrepearce I have near finished :)
The thing is that I have introduced a small API change on SequentialFile that I'm not very happy with
---
[GitHub] activemq-artemis issue #2199: ARTEMIS-1996 MappedSequentialFileFactory may c...
Posted by clebertsuconic <gi...@git.apache.org>.
Github user clebertsuconic commented on the issue:
https://github.com/apache/activemq-artemis/pull/2199
@morefuntang / @franz1981 what's the status here?
---
[GitHub] activemq-artemis pull request #2199: ARTEMIS-1996 MappedSequentialFileFactor...
Posted by franz1981 <gi...@git.apache.org>.
Github user franz1981 commented on a diff in the pull request:
https://github.com/apache/activemq-artemis/pull/2199#discussion_r206064916
--- Diff: artemis-journal/src/main/java/org/apache/activemq/artemis/core/io/mapped/MappedSequentialFileFactory.java ---
@@ -20,10 +20,13 @@
import java.nio.ByteBuffer;
import java.util.Arrays;
+import io.netty.util.ThreadDeathWatcher;
--- End diff --
Quoting the [Netty 4 javadoc](https://netty.io/4.0/api/io/netty/util/ThreadDeathWatcher.html):
> Deprecated.
> will be removed in the next major release
I think the issue could be solved differently, given that this util class has been deprecated...
---
[GitHub] activemq-artemis issue #2199: ARTEMIS-1996 MappedSequentialFileFactory may c...
Posted by jbertram <gi...@git.apache.org>.
Github user jbertram commented on the issue:
https://github.com/apache/activemq-artemis/pull/2199
IMO this PR should be closed based on the commits from #2250. @morefuntang, can you close it?
---
[GitHub] activemq-artemis issue #2199: ARTEMIS-1996 MappedSequentialFileFactory may c...
Posted by franz1981 <gi...@git.apache.org>.
Github user franz1981 commented on the issue:
https://github.com/apache/activemq-artemis/pull/2199
@michaelandrepearce good point! I will need to modify it because my solution do not create any ByteBuffer at all (or will dispose it immediately with Asyncio) and the test proposed won't work as it is
---
[GitHub] activemq-artemis pull request #2199: ARTEMIS-1996 MappedSequentialFileFactor...
Posted by franz1981 <gi...@git.apache.org>.
Github user franz1981 commented on a diff in the pull request:
https://github.com/apache/activemq-artemis/pull/2199#discussion_r207702423
--- Diff: artemis-journal/src/main/java/org/apache/activemq/artemis/core/io/mapped/MappedSequentialFileFactory.java ---
@@ -20,10 +20,13 @@
import java.nio.ByteBuffer;
import java.util.Arrays;
+import io.netty.util.ThreadDeathWatcher;
--- End diff --
@morefuntang Please take a look at the JIRA, where I have propose an alternative solution that would avoid any garbage at best
---
[GitHub] activemq-artemis issue #2199: ARTEMIS-1996 MappedSequentialFileFactory may c...
Posted by michaelandrepearce <gi...@git.apache.org>.
Github user michaelandrepearce commented on the issue:
https://github.com/apache/activemq-artemis/pull/2199
@franz1981 are you providing an alternative fix? If not why not lets merge this and later you can provide an alternative to replace this solution?
---
[GitHub] activemq-artemis issue #2199: ARTEMIS-1996 MappedSequentialFileFactory may c...
Posted by michaelandrepearce <gi...@git.apache.org>.
Github user michaelandrepearce commented on the issue:
https://github.com/apache/activemq-artemis/pull/2199
@franz1981 , @morefuntang provided a testcase to reproduce the issue, i note on your branch you havent copied the test, to prove your alternative change also would fix the issue. Could i suggest you add the same test case and raise a PR
---
[GitHub] activemq-artemis issue #2199: ARTEMIS-1996 MappedSequentialFileFactory may c...
Posted by franz1981 <gi...@git.apache.org>.
Github user franz1981 commented on the issue:
https://github.com/apache/activemq-artemis/pull/2199
@clebertsuconic I think that using a near-to-deprecation feature to solve the issue can be avoided: https://github.com/franz1981/activemq-artemis/tree/mmap_ro_2_read_journal
It is a POC that I need @morefuntang feedback to see if it solves the issue (and it could be applied to ASYNCIO probably) by saving entirely any `ByteBuffer` allocation while reading journal files.
---
[GitHub] activemq-artemis issue #2199: ARTEMIS-1996 MappedSequentialFileFactory may c...
Posted by franz1981 <gi...@git.apache.org>.
Github user franz1981 commented on the issue:
https://github.com/apache/activemq-artemis/pull/2199
@michaelandrepearce @morefuntang
I have just verified that the fix I have done is not complete yet;
there are indeed 3 points where native allocations are performed:
1. AbstractJournalUpdateTask::openFile - whole file
2. AbstractJournalUpdateTask::writeControlFile - tiny allocation
3. JournalImpl::readJournalFile - whole file
I have fixed just the last one, so It is ok to merge this PR in the meantime a complete fix will arrive:
just cleaning the thread locals won't save from the high peak of native resources utilisation when compaction is happening very often IMO.
---