[jira] [Commented] (ARTEMIS-1996) MappedSequentialFileFactory may cause DirectByteBuffer memory leaks

["Francesco Nigro (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ARTEMIS-1996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16561658#comment-16561658 ] 

Francesco Nigro commented on ARTEMIS-1996:
------------------------------------------

> Because of some customization requirements

Which kind of customization? This will help to understand the issue...

 

> Once the thread is evicted by the CompactExecutor(keepalive is 60s) in the Journal, the heap memory is "leaked".

AFAIK ThreadLocal entries are being collected if the Thread owner got collected (with a minor GC as well) and each direct ByteBuffer is being collected like any other

on-heap instance, but freed through a Cleaner that will be triggered when the buffer owner instance got collected.

If this isn't happening that means that there are many owner Threads that are spawned (holding each one a direct ByteBuffer instance) exceeding the 

configured -XX:MaxDirectMemorySize configuration.

-XX:MaxDirectMemorySize is being used by java.nio.Bits.reserveMemory to choose if run a Full GC to force cleaning up of the current allocated native memory.

Quoting some docs: "By default, the JVM does not set a limit on how much memory is reserved for Direct Byte Buffers. A soft limit of 64 MB is set, which the JVM automatically expands in 32 MB chunks, as required."

There are other docs instead stating that if not configured it will be defaulted to the -Xmx value (that means that a 5GB heap will allow additional 5GB of off heap memory): TBH I'm not feeling confident about it.

If you are experiencing FullGC that just means that you have too many live threads concurrently pooling and no minor GCs that free up memory in time.

If that's the case I see 2 solutions:
 * changing the pooling mechanism using a AtomicReferenceArray with size == available processors or a concurrent queue/stack (ie with fixed size and not depending by Threads or minorGCs)
 * not pooling when such huge direct ByteBuffers are needed (for NIO and ASYNCIO, while MAPPED could directly use its internal MappedByteBuffer to read the content)

 

 

 

 

 

 

 

 

 

> MappedSequentialFileFactory may cause DirectByteBuffer memory leaks
> -------------------------------------------------------------------
>
>                 Key: ARTEMIS-1996
>                 URL: https://issues.apache.org/jira/browse/ARTEMIS-1996
>             Project: ActiveMQ Artemis
>          Issue Type: Bug
>          Components: Broker
>            Reporter: tang pu
>            Priority: Minor
>
> Because of some customization requirements, the readJournalFile method of JournalImpl needs to be calledmultiple times. 
> During the stress test, it was found that almost every 5 hours, the Broker appeared a Full GC.
> This is the information about the Full GC in the GC log.
> {color:#FF0000}2018-07-25T12:14:07.323+0800: 10089.523: [Full GC (System.gc()) 6767M->253M(16G), 8.7138691 secs]{color}
> {color:#FF0000} [Eden: 632.0M(712.0M)->0.0B(816.0M) Survivors: 104.0M->0.0B Heap: 6767.6M(16.0G)->253.9M(16.0G)], [Metaspace: 36323K->35961K(1083392K)]{color}
> {color:#FF0000} [Times: user=2.56 sys=0.42, real=8.71 secs] {color}
> When the Full GC appears, the thread stack is as follows:
> {color:#FF0000}java.lang.System.gc(System.java:993){color}
> {color:#FF0000}java.nio.Bits.reserveMemory(Bits.java:666){color}
> {color:#FF0000}java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123){color}
> {color:#FF0000}java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311){color}
> {color:#FF0000}org.apache.activemq.artemis.core.io.mapped.MappedSequentialFileFactory.newBuffer(MappedSequentialFileFactory.java:109){color}
> {color:#FF0000}org.apache.activemq.artemis.core.journal.impl.JournalImpl.readJournalFile(JournalImpl.java:463){color}
> By analyzing the stack, it should be that the JVM's heap memory cannot be allocated, causing the JVM to call the System.gc() method.
> In the Broker, MappedSequentialFileFactory caches off-heap memory through ThreadLocal. Once the thread is evicted by the CompactExecutor(keepalive is 60s) in the Journal, the heap memory is "leaked".
> {color:#FF0000}NIOSequentialFileFactory{color} also has the same problem
>  
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)