nio buffer memory problem

[RuralHunter <ru...@gmail.com>]

Hi,

We've been run activemq 5.10.2 with thousands of clients in our production
for quite a long time. Several days ago, we upgraded to the newer version
5.13.3. After running for several days, we saw the OutOfMemoryError and
activemq hung up in busy GC:
2016-06-06 02:01:54,238 | ERROR | Error in thread 'ActiveMQ
BrokerService[localhost] Task-10229' |
org.apache.activemq.thread.TaskRunnerFactory | ActiveMQ
BrokerService[localhost] Task-10229
  java.lang.OutOfMemoryError: Direct buffer memory
	at java.nio.Bits.reserveMemory(Bits.java:658)[:1.7.0_60]
	at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123)[:1.7.0_60]
	at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:306)[:1.7.0_60]
	at sun.nio.ch.Util.getTemporaryDirectBuffer(Util.java:174)[:1.7.0_60]
	at sun.nio.ch.IOUtil.write(IOUtil.java:58)[:1.7.0_60]
	at
sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:487)[:1.7.0_60]
	at
org.apache.activemq.transport.nio.NIOOutputStream.write(NIOOutputStream.java:206)[activemq-client-5.13.3.jar:5.13.3]
	at
org.apache.activemq.transport.nio.NIOOutputStream.write(NIOOutputStream.java:116)[activemq-client-5.13.3.jar:5.13.3]
	at java.io.DataOutputStream.write(DataOutputStream.java:107)[:1.7.0_60]
	at
org.apache.activemq.openwire.v10.BaseDataStreamMarshaller.tightMarshalByteSequence2(BaseDataStreamMarshaller.java:431)[activemq-client-5.13.3.jar:5.13.3]
	at
org.apache.activemq.openwire.v10.MessageMarshaller.tightMarshal2(MessageMarshaller.java:182)[activemq-client-5.13.3.jar:5.13.3]
	at
org.apache.activemq.openwire.v10.ActiveMQMessageMarshaller.tightMarshal2(ActiveMQMessageMarshaller.java:89)[activemq-client-5.13.3.jar:5.13.3]
	at
org.apache.activemq.openwire.v10.ActiveMQTextMessageMarshaller.tightMarshal2(ActiveMQTextMessageMarshaller.java:89)[activemq-client-5.13.3.jar:5.13.3]
	at
org.apache.activemq.openwire.OpenWireFormat.tightMarshalNestedObject2(OpenWireFormat.java:426)[activemq-client-5.13.3.jar:5.13.3]
	at
org.apache.activemq.openwire.v10.BaseDataStreamMarshaller.tightMarshalNestedObject2(BaseDataStreamMarshaller.java:135)[activemq-client-5.13.3.jar:5.13.3]
	at
org.apache.activemq.openwire.v10.MessageDispatchMarshaller.tightMarshal2(MessageDispatchMarshaller.java:105)[activemq-client-5.13.3.jar:5.13.3]
	at
org.apache.activemq.openwire.OpenWireFormat.marshal(OpenWireFormat.java:234)[activemq-client-5.13.3.jar:5.13.3]
	at
org.apache.activemq.transport.tcp.TcpTransport.oneway(TcpTransport.java:193)[activemq-client-5.13.3.jar:5.13.3]

I searched the error on web and found it's nio direct buffer limit problem
and we can increase the jvm setting -XX:MaxDirectMemorySize. The default
value of this setting is pretty small, only 64m . We never set this value
with 5.10.2. So I guessed the new version required more memory than the old
version so I set this value to 4g. I thought it should be more than enough.
We also started to monitor the direct buffer memory usage which we hadn't
done before. 

Unfortunately, after one day we saw the direct buffer usage reached 4g and
activemq again was busy GC'ing. After a while we saw the same
OutOfMemoryError again. I then thought there might be some memory leak in
the new version. So I reverted back to 5.10.2, without the
MaxDirectMemorySize setting. Interesting thing happened. The old version
runs well but the actual direct buffer it uses is far more than 4g. It's
around 10g all the time! 

So I have 2 questions here:
1. How much direct buffer activemq actually needs, or based what should we
set this memory setting?
2. How can activemq 5.10.2 ignores the -XX:MaxDirectMemorySize while the new
version can not?



--
View this message in context: http://activemq.2283324.n4.nabble.com/nio-buffer-memory-problem-tp4712706.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: nio buffer memory problem

[RuralHunter <ru...@gmail.com>]

All right. I will try with that. Thanks for the info.



--
View this message in context: http://activemq.2283324.n4.nabble.com/nio-buffer-memory-problem-tp4712706p4712910.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: nio buffer memory problem

[Christopher Shannon <ch...@gmail.com>]

Yes, the buffer was changed to a direct byte buffer for performance
reasons. This was done in https://issues.apache.org/jira/browse/AMQ-6184

 I think the best thing you can do for now is to keep increasing the size
and see if it stabilizes.  You mentioned that you could see it using 10g
under version 5.10 so maybe you can set it to that and see if it will run
without running out of memory.  My guess is your app just needs that much
memory to run and there isn't an actual memory leak going on but hard to
say until you increase the available memory to the same amount 5.10.2 uses.

On Tue, Jun 7, 2016 at 10:11 PM, RuralHunter <ru...@gmail.com> wrote:

> I found this article: http://www.evanjones.ca/java-bytebuffer-leak.html
> I guess 5.10.2 uses 'Heap ByteBuffers' which is backed with unlimited
> 'temporary Direct ByteBuffer', while 5.13.3 uses 'Direct ByteBuffer'
> directly?
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/nio-buffer-memory-problem-tp4712706p4712766.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>

Re: nio buffer memory problem

[RuralHunter <ru...@gmail.com>]

I found this article: http://www.evanjones.ca/java-bytebuffer-leak.html
I guess 5.10.2 uses 'Heap ByteBuffers' which is backed with unlimited
'temporary Direct ByteBuffer', while 5.13.3 uses 'Direct ByteBuffer'
directly?



--
View this message in context: http://activemq.2283324.n4.nabble.com/nio-buffer-memory-problem-tp4712706p4712766.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.