You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@qpid.apache.org by Adel Boutros <ad...@live.com> on 2016/07/28 09:58:56 UTC

[Java Broker - 6.0.1] OutOfMemory in the directy memory buffer

Hello,

While sending messages to the Java broker which is backed by a Berkley DB message store, I am getting the below exception. However, the heap memory is stable so I assume the problem is coming from elsewhere.
I am sending 5 000 000 messages which contain a byte array of 100KB of data. I have the error at around 4 000 000 messages. Please note that the messages are all being consumed by consumers which are not congested.

I have also monitored the heap and it is never above 1 GB.

So I have the following questions:
What parts of the broker uses direct memory?Does the fact using BDB means that each message which is referenced by BDB has memory allocated on the direct memory even if the message is consumed?
How can I analyse this OutOfMemory exception? (There is no heap dump So I assume the error is not related to GC and I personally suspect the BDB to be the culprit)


Broker Console output
Info: QPID_JAVA_GC not set. Defaulting to JAVA_GC -XX:+UseConcMarkSweepGC -XX:+HeapDumpOnOutOfMemoryError
Using QPID_JAVA_MEM setting -Xmx16g -Xms2g
[Broker] BRK-1006 : Using configuration : /dell445srv/apps/messaging_server/brokers/work1/config.json
[Broker] BRK-1001 : Startup : Version: 6.0.1 Build: 1731621
[Broker] BRK-1010 : Platform : JVM : Oracle Corporation version: 1.7.0_80-b15 OS : Linux version: 2.6.32-358.el6.x86_64 arch: amd64 cores: 20
[Broker] BRK-1011 : Maximum Memory : Heap : 17,049,059,328 bytes Direct : 17,049,059,328 bytes
[Broker] BRK-1017 : Process : PID : 26179
[Broker] BRK-1002 : Starting : Listening on TCP port 10255
[Broker] MNG-1001 : Web Management Startup
[Broker] MNG-1002 : Starting : HTTP : Listening on TCP port 10256
[Broker] MNG-1004 : Web Management Ready
[Broker] BRK-1004 : Qpid Broker Ready
########################################################################
#
# Unhandled Exception java.lang.OutOfMemoryError: Direct buffer memory in Thread IO-/10.25.6.48:54546
#
# Exiting
#
########################################################################
java.lang.OutOfMemoryError: Direct buffer memory
...



Qpid Log
2016-07-28 11:41:50,143 ERROR [IO-/10.25.6.48:54546] (o.a.q.s.Main) - Uncaught exception, shutting down.
java.lang.OutOfMemoryError: Direct buffer memory
        at java.nio.Bits.reserveMemory(Bits.java:658) ~[na:1.7.0_80]
        at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123) ~[na:1.7.0_80]
        at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:306) ~[na:1.7.0_80]
        at org.apache.qpid.bytebuffer.QpidByteBuffer.allocateDirect(QpidByteBuffer.java:469) ~[qpid-common-6.0.1.jar:6.0.1]
        at org.apache.qpid.bytebuffer.QpidByteBuffer.allocateDirect(QpidByteBuffer.java:482) ~[qpid-common-6.0.1.jar:6.0.1]
        at org.apache.qpid.amqp_1_0.framing.FrameHandler.parse(FrameHandler.java:149) ~[qpid-broker-plugins-amqp-1-0-protocol-6.0.1.jar:6.0.1]
        at org.apache.qpid.server.protocol.v1_0.AMQPConnection_1_0$5.run(AMQPConnection_1_0.java:440) ~[qpid-broker-plugins-amqp-1-0-protocol-6.0.1.jar:6.0.1]
        at org.apache.qpid.server.protocol.v1_0.AMQPConnection_1_0$5.run(AMQPConnection_1_0.java:436) ~[qpid-broker-plugins-amqp-1-0-protocol-6.0.1.jar:6.0.1]
        at java.security.AccessController.doPrivileged(Native Method) ~[na:1.7.0_80]
        at org.apache.qpid.server.protocol.v1_0.AMQPConnection_1_0.received(AMQPConnection_1_0.java:435) ~[qpid-broker-plugins-amqp-1-0-protocol-6.0.1.jar:6.0.1]
        at org.apache.qpid.server.transport.MultiVersionProtocolEngine.received(MultiVersionProtocolEngine.java:142) ~[qpid-broker-core-6.0.1.jar:6.0.1]
        at org.apache.qpid.server.transport.NonBlockingConnection.processAmqpData(NonBlockingConnection.java:564) ~[qpid-broker-core-6.0.1.jar:6.0.1]
        at org.apache.qpid.server.transport.NonBlockingConnectionPlainDelegate.processData(NonBlockingConnectionPlainDelegate.java:58) ~[qpid-broker-core-6.0.1.jar:6.0.1]
        at org.apache.qpid.server.transport.NonBlockingConnection.doRead(NonBlockingConnection.java:463) ~[qpid-broker-core-6.0.1.jar:6.0.1]
        at org.apache.qpid.server.transport.NonBlockingConnection.doWork(NonBlockingConnection.java:259) ~[qpid-broker-core-6.0.1.jar:6.0.1]
        at org.apache.qpid.server.transport.NetworkConnectionScheduler.processConnection(NetworkConnectionScheduler.java:108) ~[qpid-broker-core-6.0.1.jar:6.0.1]
        at org.apache.qpid.server.transport.SelectorThread$ConnectionProcessor.processConnection(SelectorThread.java:505) ~[qpid-broker-core-6.0.1.jar:6.0.1]
        at org.apache.qpid.server.transport.SelectorThread$SelectionTask.performSelect(SelectorThread.java:338) ~[qpid-broker-core-6.0.1.jar:6.0.1]
        at org.apache.qpid.server.transport.SelectorThread$SelectionTask.run(SelectorThread.java:87) ~[qpid-broker-core-6.0.1.jar:6.0.1]
        at org.apache.qpid.server.transport.SelectorThread.run(SelectorThread.java:463) ~[qpid-broker-core-6.0.1.jar:6.0.1]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) ~[na:1.7.0_80]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) ~[na:1.7.0_80]
        at java.lang.Thread.run(Thread.java:745) ~[na:1.7.0_80]






Regards,
Adel

 		 	   		  

RE: [Java Broker - 6.0.1] OutOfMemory in the directy memory buffer

Posted by Adel Boutros <ad...@live.com>.
Hello again,

I re-launched the tests with 2 configurations and they both failed very quick:
Xmx1g ---> Failed after sending just 12 000 messages
Xmx2g --->  Failed after sending just 25 000 messages

Regards,
Adel

> From: adelboutros@live.com
> To: users@qpid.apache.org
> Subject: RE: [Java Broker - 6.0.1] OutOfMemory in the directy memory buffer
> Date: Fri, 29 Jul 2016 15:15:24 +0200
> 
> Hello Keith,
> 
> As stated in this thread[1], we are actually doing some benchmarks to test the performance of the broker and the dispatch router. In the failing test case, I try to send the messages using 4 JMS producers connected directly to the broker each having its own connection (1 JVM per producer). The messages are sent in synchronous mode with AUTO_ACKNOWLEDGE. We try to send them as fast as possible but the test actually takes around 2 hours.
> 
> There are no transactions and no batching. Each producer sends a message continuously in the main thread.
> 
> In order for the test to pass, I had to set the maximum heap to 32 GB which resulted in augmenting the direct memory to 32 GB also.
> 
> Because the test takes a long time, I have to run it at night. So I cannot experiment different heap sizes now.
> 
> [1]: http://qpid.2158936.n2.nabble.com/Performance-Benchmarking-Qpid-dispatch-router-0-6-0-with-Qpid-Java-Broker-6-0-0-td7648076.html
> 
> Regards,
> Adel
> 
> > From: keith.wall@gmail.com
> > Date: Fri, 29 Jul 2016 11:44:57 +0100
> > Subject: Re: [Java Broker - 6.0.1] OutOfMemory in the directy memory buffer
> > To: users@qpid.apache.org
> > 
> > Hello Adel
> > 
> > Since v6, the Java Broker keeps messages in direct memory rather than
> > heap.  There is some explanation here:
> > 
> > https://qpid.apache.org/releases/qpid-java-6.0.4/java-broker/book/Java-Broker-Runtime-Memory.html
> > 
> > Internally the Broker for Java uses a direct-memory caching strategy
> > that aims to avoid the underlying JVM having to release/free chunks of
> > direct memory frequently.  In the AMQP 1.0 implementation within the
> > Broker there are currently some compromises around this caching
> > strategy that mean it sometimes relies on a JVM GC to free direct
> > memory.   Now, the JVM normally does not trigger a GC until there is
> > pressure on the _heap_.  I speculate that as your heap is so large,
> > that the GC events are not occurring sufficiently frequently and
> > meaning direct memory is not released and allowing direct memory to
> > became exhausted.  Try experimenting by reducing the heap.
> > 
> > If you can tell us a little more about your test case, we may be able
> > to make other suggestions.  How exactly are the 5,000,000 messages
> > arriving at the Broker? Are they arriving on a short period of time?
> > How connections are used to publish the messages? Are transactions in
> > use? Batching?
> > 
> > 
> > Kind regards, Keith.
> > 
> > On 28 July 2016 at 10:58, Adel Boutros <ad...@live.com> wrote:
> > > Hello,
> > >
> > > While sending messages to the Java broker which is backed by a Berkley DB message store, I am getting the below exception. However, the heap memory is stable so I assume the problem is coming from elsewhere.
> > > I am sending 5 000 000 messages which contain a byte array of 100KB of data. I have the error at around 4 000 000 messages. Please note that the messages are all being consumed by consumers which are not congested.
> > >
> > > I have also monitored the heap and it is never above 1 GB.
> > >
> > > So I have the following questions:
> > > What parts of the broker uses direct memory?Does the fact using BDB means that each message which is referenced by BDB has memory allocated on the direct memory even if the message is consumed?
> > > How can I analyse this OutOfMemory exception? (There is no heap dump So I assume the error is not related to GC and I personally suspect the BDB to be the culprit)
> > >
> > >
> > > Broker Console output
> > > Info: QPID_JAVA_GC not set. Defaulting to JAVA_GC -XX:+UseConcMarkSweepGC -XX:+HeapDumpOnOutOfMemoryError
> > > Using QPID_JAVA_MEM setting -Xmx16g -Xms2g
> > > [Broker] BRK-1006 : Using configuration : /dell445srv/apps/messaging_server/brokers/work1/config.json
> > > [Broker] BRK-1001 : Startup : Version: 6.0.1 Build: 1731621
> > > [Broker] BRK-1010 : Platform : JVM : Oracle Corporation version: 1.7.0_80-b15 OS : Linux version: 2.6.32-358.el6.x86_64 arch: amd64 cores: 20
> > > [Broker] BRK-1011 : Maximum Memory : Heap : 17,049,059,328 bytes Direct : 17,049,059,328 bytes
> > > [Broker] BRK-1017 : Process : PID : 26179
> > > [Broker] BRK-1002 : Starting : Listening on TCP port 10255
> > > [Broker] MNG-1001 : Web Management Startup
> > > [Broker] MNG-1002 : Starting : HTTP : Listening on TCP port 10256
> > > [Broker] MNG-1004 : Web Management Ready
> > > [Broker] BRK-1004 : Qpid Broker Ready
> > > ########################################################################
> > > #
> > > # Unhandled Exception java.lang.OutOfMemoryError: Direct buffer memory in Thread IO-/10.25.6.48:54546
> > > #
> > > # Exiting
> > > #
> > > ########################################################################
> > > java.lang.OutOfMemoryError: Direct buffer memory
> > > ...
> > >
> > >
> > >
> > > Qpid Log
> > > 2016-07-28 11:41:50,143 ERROR [IO-/10.25.6.48:54546] (o.a.q.s.Main) - Uncaught exception, shutting down.
> > > java.lang.OutOfMemoryError: Direct buffer memory
> > >         at java.nio.Bits.reserveMemory(Bits.java:658) ~[na:1.7.0_80]
> > >         at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123) ~[na:1.7.0_80]
> > >         at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:306) ~[na:1.7.0_80]
> > >         at org.apache.qpid.bytebuffer.QpidByteBuffer.allocateDirect(QpidByteBuffer.java:469) ~[qpid-common-6.0.1.jar:6.0.1]
> > >         at org.apache.qpid.bytebuffer.QpidByteBuffer.allocateDirect(QpidByteBuffer.java:482) ~[qpid-common-6.0.1.jar:6.0.1]
> > >         at org.apache.qpid.amqp_1_0.framing.FrameHandler.parse(FrameHandler.java:149) ~[qpid-broker-plugins-amqp-1-0-protocol-6.0.1.jar:6.0.1]
> > >         at org.apache.qpid.server.protocol.v1_0.AMQPConnection_1_0$5.run(AMQPConnection_1_0.java:440) ~[qpid-broker-plugins-amqp-1-0-protocol-6.0.1.jar:6.0.1]
> > >         at org.apache.qpid.server.protocol.v1_0.AMQPConnection_1_0$5.run(AMQPConnection_1_0.java:436) ~[qpid-broker-plugins-amqp-1-0-protocol-6.0.1.jar:6.0.1]
> > >         at java.security.AccessController.doPrivileged(Native Method) ~[na:1.7.0_80]
> > >         at org.apache.qpid.server.protocol.v1_0.AMQPConnection_1_0.received(AMQPConnection_1_0.java:435) ~[qpid-broker-plugins-amqp-1-0-protocol-6.0.1.jar:6.0.1]
> > >         at org.apache.qpid.server.transport.MultiVersionProtocolEngine.received(MultiVersionProtocolEngine.java:142) ~[qpid-broker-core-6.0.1.jar:6.0.1]
> > >         at org.apache.qpid.server.transport.NonBlockingConnection.processAmqpData(NonBlockingConnection.java:564) ~[qpid-broker-core-6.0.1.jar:6.0.1]
> > >         at org.apache.qpid.server.transport.NonBlockingConnectionPlainDelegate.processData(NonBlockingConnectionPlainDelegate.java:58) ~[qpid-broker-core-6.0.1.jar:6.0.1]
> > >         at org.apache.qpid.server.transport.NonBlockingConnection.doRead(NonBlockingConnection.java:463) ~[qpid-broker-core-6.0.1.jar:6.0.1]
> > >         at org.apache.qpid.server.transport.NonBlockingConnection.doWork(NonBlockingConnection.java:259) ~[qpid-broker-core-6.0.1.jar:6.0.1]
> > >         at org.apache.qpid.server.transport.NetworkConnectionScheduler.processConnection(NetworkConnectionScheduler.java:108) ~[qpid-broker-core-6.0.1.jar:6.0.1]
> > >         at org.apache.qpid.server.transport.SelectorThread$ConnectionProcessor.processConnection(SelectorThread.java:505) ~[qpid-broker-core-6.0.1.jar:6.0.1]
> > >         at org.apache.qpid.server.transport.SelectorThread$SelectionTask.performSelect(SelectorThread.java:338) ~[qpid-broker-core-6.0.1.jar:6.0.1]
> > >         at org.apache.qpid.server.transport.SelectorThread$SelectionTask.run(SelectorThread.java:87) ~[qpid-broker-core-6.0.1.jar:6.0.1]
> > >         at org.apache.qpid.server.transport.SelectorThread.run(SelectorThread.java:463) ~[qpid-broker-core-6.0.1.jar:6.0.1]
> > >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) ~[na:1.7.0_80]
> > >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) ~[na:1.7.0_80]
> > >         at java.lang.Thread.run(Thread.java:745) ~[na:1.7.0_80]
> > >
> > >
> > >
> > >
> > >
> > >
> > > Regards,
> > > Adel
> > >
> > >
> > 
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> > For additional commands, e-mail: users-help@qpid.apache.org
> > 
>  		 	   		  
 		 	   		  

RE: [Java Broker - 6.0.1] OutOfMemory in the directy memory buffer

Posted by Adel Boutros <ad...@live.com>.
Hello Keith,

As stated in this thread[1], we are actually doing some benchmarks to test the performance of the broker and the dispatch router. In the failing test case, I try to send the messages using 4 JMS producers connected directly to the broker each having its own connection (1 JVM per producer). The messages are sent in synchronous mode with AUTO_ACKNOWLEDGE. We try to send them as fast as possible but the test actually takes around 2 hours.

There are no transactions and no batching. Each producer sends a message continuously in the main thread.

In order for the test to pass, I had to set the maximum heap to 32 GB which resulted in augmenting the direct memory to 32 GB also.

Because the test takes a long time, I have to run it at night. So I cannot experiment different heap sizes now.

[1]: http://qpid.2158936.n2.nabble.com/Performance-Benchmarking-Qpid-dispatch-router-0-6-0-with-Qpid-Java-Broker-6-0-0-td7648076.html

Regards,
Adel

> From: keith.wall@gmail.com
> Date: Fri, 29 Jul 2016 11:44:57 +0100
> Subject: Re: [Java Broker - 6.0.1] OutOfMemory in the directy memory buffer
> To: users@qpid.apache.org
> 
> Hello Adel
> 
> Since v6, the Java Broker keeps messages in direct memory rather than
> heap.  There is some explanation here:
> 
> https://qpid.apache.org/releases/qpid-java-6.0.4/java-broker/book/Java-Broker-Runtime-Memory.html
> 
> Internally the Broker for Java uses a direct-memory caching strategy
> that aims to avoid the underlying JVM having to release/free chunks of
> direct memory frequently.  In the AMQP 1.0 implementation within the
> Broker there are currently some compromises around this caching
> strategy that mean it sometimes relies on a JVM GC to free direct
> memory.   Now, the JVM normally does not trigger a GC until there is
> pressure on the _heap_.  I speculate that as your heap is so large,
> that the GC events are not occurring sufficiently frequently and
> meaning direct memory is not released and allowing direct memory to
> became exhausted.  Try experimenting by reducing the heap.
> 
> If you can tell us a little more about your test case, we may be able
> to make other suggestions.  How exactly are the 5,000,000 messages
> arriving at the Broker? Are they arriving on a short period of time?
> How connections are used to publish the messages? Are transactions in
> use? Batching?
> 
> 
> Kind regards, Keith.
> 
> On 28 July 2016 at 10:58, Adel Boutros <ad...@live.com> wrote:
> > Hello,
> >
> > While sending messages to the Java broker which is backed by a Berkley DB message store, I am getting the below exception. However, the heap memory is stable so I assume the problem is coming from elsewhere.
> > I am sending 5 000 000 messages which contain a byte array of 100KB of data. I have the error at around 4 000 000 messages. Please note that the messages are all being consumed by consumers which are not congested.
> >
> > I have also monitored the heap and it is never above 1 GB.
> >
> > So I have the following questions:
> > What parts of the broker uses direct memory?Does the fact using BDB means that each message which is referenced by BDB has memory allocated on the direct memory even if the message is consumed?
> > How can I analyse this OutOfMemory exception? (There is no heap dump So I assume the error is not related to GC and I personally suspect the BDB to be the culprit)
> >
> >
> > Broker Console output
> > Info: QPID_JAVA_GC not set. Defaulting to JAVA_GC -XX:+UseConcMarkSweepGC -XX:+HeapDumpOnOutOfMemoryError
> > Using QPID_JAVA_MEM setting -Xmx16g -Xms2g
> > [Broker] BRK-1006 : Using configuration : /dell445srv/apps/messaging_server/brokers/work1/config.json
> > [Broker] BRK-1001 : Startup : Version: 6.0.1 Build: 1731621
> > [Broker] BRK-1010 : Platform : JVM : Oracle Corporation version: 1.7.0_80-b15 OS : Linux version: 2.6.32-358.el6.x86_64 arch: amd64 cores: 20
> > [Broker] BRK-1011 : Maximum Memory : Heap : 17,049,059,328 bytes Direct : 17,049,059,328 bytes
> > [Broker] BRK-1017 : Process : PID : 26179
> > [Broker] BRK-1002 : Starting : Listening on TCP port 10255
> > [Broker] MNG-1001 : Web Management Startup
> > [Broker] MNG-1002 : Starting : HTTP : Listening on TCP port 10256
> > [Broker] MNG-1004 : Web Management Ready
> > [Broker] BRK-1004 : Qpid Broker Ready
> > ########################################################################
> > #
> > # Unhandled Exception java.lang.OutOfMemoryError: Direct buffer memory in Thread IO-/10.25.6.48:54546
> > #
> > # Exiting
> > #
> > ########################################################################
> > java.lang.OutOfMemoryError: Direct buffer memory
> > ...
> >
> >
> >
> > Qpid Log
> > 2016-07-28 11:41:50,143 ERROR [IO-/10.25.6.48:54546] (o.a.q.s.Main) - Uncaught exception, shutting down.
> > java.lang.OutOfMemoryError: Direct buffer memory
> >         at java.nio.Bits.reserveMemory(Bits.java:658) ~[na:1.7.0_80]
> >         at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123) ~[na:1.7.0_80]
> >         at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:306) ~[na:1.7.0_80]
> >         at org.apache.qpid.bytebuffer.QpidByteBuffer.allocateDirect(QpidByteBuffer.java:469) ~[qpid-common-6.0.1.jar:6.0.1]
> >         at org.apache.qpid.bytebuffer.QpidByteBuffer.allocateDirect(QpidByteBuffer.java:482) ~[qpid-common-6.0.1.jar:6.0.1]
> >         at org.apache.qpid.amqp_1_0.framing.FrameHandler.parse(FrameHandler.java:149) ~[qpid-broker-plugins-amqp-1-0-protocol-6.0.1.jar:6.0.1]
> >         at org.apache.qpid.server.protocol.v1_0.AMQPConnection_1_0$5.run(AMQPConnection_1_0.java:440) ~[qpid-broker-plugins-amqp-1-0-protocol-6.0.1.jar:6.0.1]
> >         at org.apache.qpid.server.protocol.v1_0.AMQPConnection_1_0$5.run(AMQPConnection_1_0.java:436) ~[qpid-broker-plugins-amqp-1-0-protocol-6.0.1.jar:6.0.1]
> >         at java.security.AccessController.doPrivileged(Native Method) ~[na:1.7.0_80]
> >         at org.apache.qpid.server.protocol.v1_0.AMQPConnection_1_0.received(AMQPConnection_1_0.java:435) ~[qpid-broker-plugins-amqp-1-0-protocol-6.0.1.jar:6.0.1]
> >         at org.apache.qpid.server.transport.MultiVersionProtocolEngine.received(MultiVersionProtocolEngine.java:142) ~[qpid-broker-core-6.0.1.jar:6.0.1]
> >         at org.apache.qpid.server.transport.NonBlockingConnection.processAmqpData(NonBlockingConnection.java:564) ~[qpid-broker-core-6.0.1.jar:6.0.1]
> >         at org.apache.qpid.server.transport.NonBlockingConnectionPlainDelegate.processData(NonBlockingConnectionPlainDelegate.java:58) ~[qpid-broker-core-6.0.1.jar:6.0.1]
> >         at org.apache.qpid.server.transport.NonBlockingConnection.doRead(NonBlockingConnection.java:463) ~[qpid-broker-core-6.0.1.jar:6.0.1]
> >         at org.apache.qpid.server.transport.NonBlockingConnection.doWork(NonBlockingConnection.java:259) ~[qpid-broker-core-6.0.1.jar:6.0.1]
> >         at org.apache.qpid.server.transport.NetworkConnectionScheduler.processConnection(NetworkConnectionScheduler.java:108) ~[qpid-broker-core-6.0.1.jar:6.0.1]
> >         at org.apache.qpid.server.transport.SelectorThread$ConnectionProcessor.processConnection(SelectorThread.java:505) ~[qpid-broker-core-6.0.1.jar:6.0.1]
> >         at org.apache.qpid.server.transport.SelectorThread$SelectionTask.performSelect(SelectorThread.java:338) ~[qpid-broker-core-6.0.1.jar:6.0.1]
> >         at org.apache.qpid.server.transport.SelectorThread$SelectionTask.run(SelectorThread.java:87) ~[qpid-broker-core-6.0.1.jar:6.0.1]
> >         at org.apache.qpid.server.transport.SelectorThread.run(SelectorThread.java:463) ~[qpid-broker-core-6.0.1.jar:6.0.1]
> >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) ~[na:1.7.0_80]
> >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) ~[na:1.7.0_80]
> >         at java.lang.Thread.run(Thread.java:745) ~[na:1.7.0_80]
> >
> >
> >
> >
> >
> >
> > Regards,
> > Adel
> >
> >
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
> 
 		 	   		  

Re: [Java Broker - 6.0.1] OutOfMemory in the directy memory buffer

Posted by Keith W <ke...@gmail.com>.
Hello Adel

Since v6, the Java Broker keeps messages in direct memory rather than
heap.  There is some explanation here:

https://qpid.apache.org/releases/qpid-java-6.0.4/java-broker/book/Java-Broker-Runtime-Memory.html

Internally the Broker for Java uses a direct-memory caching strategy
that aims to avoid the underlying JVM having to release/free chunks of
direct memory frequently.  In the AMQP 1.0 implementation within the
Broker there are currently some compromises around this caching
strategy that mean it sometimes relies on a JVM GC to free direct
memory.   Now, the JVM normally does not trigger a GC until there is
pressure on the _heap_.  I speculate that as your heap is so large,
that the GC events are not occurring sufficiently frequently and
meaning direct memory is not released and allowing direct memory to
became exhausted.  Try experimenting by reducing the heap.

If you can tell us a little more about your test case, we may be able
to make other suggestions.  How exactly are the 5,000,000 messages
arriving at the Broker? Are they arriving on a short period of time?
How connections are used to publish the messages? Are transactions in
use? Batching?


Kind regards, Keith.

On 28 July 2016 at 10:58, Adel Boutros <ad...@live.com> wrote:
> Hello,
>
> While sending messages to the Java broker which is backed by a Berkley DB message store, I am getting the below exception. However, the heap memory is stable so I assume the problem is coming from elsewhere.
> I am sending 5 000 000 messages which contain a byte array of 100KB of data. I have the error at around 4 000 000 messages. Please note that the messages are all being consumed by consumers which are not congested.
>
> I have also monitored the heap and it is never above 1 GB.
>
> So I have the following questions:
> What parts of the broker uses direct memory?Does the fact using BDB means that each message which is referenced by BDB has memory allocated on the direct memory even if the message is consumed?
> How can I analyse this OutOfMemory exception? (There is no heap dump So I assume the error is not related to GC and I personally suspect the BDB to be the culprit)
>
>
> Broker Console output
> Info: QPID_JAVA_GC not set. Defaulting to JAVA_GC -XX:+UseConcMarkSweepGC -XX:+HeapDumpOnOutOfMemoryError
> Using QPID_JAVA_MEM setting -Xmx16g -Xms2g
> [Broker] BRK-1006 : Using configuration : /dell445srv/apps/messaging_server/brokers/work1/config.json
> [Broker] BRK-1001 : Startup : Version: 6.0.1 Build: 1731621
> [Broker] BRK-1010 : Platform : JVM : Oracle Corporation version: 1.7.0_80-b15 OS : Linux version: 2.6.32-358.el6.x86_64 arch: amd64 cores: 20
> [Broker] BRK-1011 : Maximum Memory : Heap : 17,049,059,328 bytes Direct : 17,049,059,328 bytes
> [Broker] BRK-1017 : Process : PID : 26179
> [Broker] BRK-1002 : Starting : Listening on TCP port 10255
> [Broker] MNG-1001 : Web Management Startup
> [Broker] MNG-1002 : Starting : HTTP : Listening on TCP port 10256
> [Broker] MNG-1004 : Web Management Ready
> [Broker] BRK-1004 : Qpid Broker Ready
> ########################################################################
> #
> # Unhandled Exception java.lang.OutOfMemoryError: Direct buffer memory in Thread IO-/10.25.6.48:54546
> #
> # Exiting
> #
> ########################################################################
> java.lang.OutOfMemoryError: Direct buffer memory
> ...
>
>
>
> Qpid Log
> 2016-07-28 11:41:50,143 ERROR [IO-/10.25.6.48:54546] (o.a.q.s.Main) - Uncaught exception, shutting down.
> java.lang.OutOfMemoryError: Direct buffer memory
>         at java.nio.Bits.reserveMemory(Bits.java:658) ~[na:1.7.0_80]
>         at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123) ~[na:1.7.0_80]
>         at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:306) ~[na:1.7.0_80]
>         at org.apache.qpid.bytebuffer.QpidByteBuffer.allocateDirect(QpidByteBuffer.java:469) ~[qpid-common-6.0.1.jar:6.0.1]
>         at org.apache.qpid.bytebuffer.QpidByteBuffer.allocateDirect(QpidByteBuffer.java:482) ~[qpid-common-6.0.1.jar:6.0.1]
>         at org.apache.qpid.amqp_1_0.framing.FrameHandler.parse(FrameHandler.java:149) ~[qpid-broker-plugins-amqp-1-0-protocol-6.0.1.jar:6.0.1]
>         at org.apache.qpid.server.protocol.v1_0.AMQPConnection_1_0$5.run(AMQPConnection_1_0.java:440) ~[qpid-broker-plugins-amqp-1-0-protocol-6.0.1.jar:6.0.1]
>         at org.apache.qpid.server.protocol.v1_0.AMQPConnection_1_0$5.run(AMQPConnection_1_0.java:436) ~[qpid-broker-plugins-amqp-1-0-protocol-6.0.1.jar:6.0.1]
>         at java.security.AccessController.doPrivileged(Native Method) ~[na:1.7.0_80]
>         at org.apache.qpid.server.protocol.v1_0.AMQPConnection_1_0.received(AMQPConnection_1_0.java:435) ~[qpid-broker-plugins-amqp-1-0-protocol-6.0.1.jar:6.0.1]
>         at org.apache.qpid.server.transport.MultiVersionProtocolEngine.received(MultiVersionProtocolEngine.java:142) ~[qpid-broker-core-6.0.1.jar:6.0.1]
>         at org.apache.qpid.server.transport.NonBlockingConnection.processAmqpData(NonBlockingConnection.java:564) ~[qpid-broker-core-6.0.1.jar:6.0.1]
>         at org.apache.qpid.server.transport.NonBlockingConnectionPlainDelegate.processData(NonBlockingConnectionPlainDelegate.java:58) ~[qpid-broker-core-6.0.1.jar:6.0.1]
>         at org.apache.qpid.server.transport.NonBlockingConnection.doRead(NonBlockingConnection.java:463) ~[qpid-broker-core-6.0.1.jar:6.0.1]
>         at org.apache.qpid.server.transport.NonBlockingConnection.doWork(NonBlockingConnection.java:259) ~[qpid-broker-core-6.0.1.jar:6.0.1]
>         at org.apache.qpid.server.transport.NetworkConnectionScheduler.processConnection(NetworkConnectionScheduler.java:108) ~[qpid-broker-core-6.0.1.jar:6.0.1]
>         at org.apache.qpid.server.transport.SelectorThread$ConnectionProcessor.processConnection(SelectorThread.java:505) ~[qpid-broker-core-6.0.1.jar:6.0.1]
>         at org.apache.qpid.server.transport.SelectorThread$SelectionTask.performSelect(SelectorThread.java:338) ~[qpid-broker-core-6.0.1.jar:6.0.1]
>         at org.apache.qpid.server.transport.SelectorThread$SelectionTask.run(SelectorThread.java:87) ~[qpid-broker-core-6.0.1.jar:6.0.1]
>         at org.apache.qpid.server.transport.SelectorThread.run(SelectorThread.java:463) ~[qpid-broker-core-6.0.1.jar:6.0.1]
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) ~[na:1.7.0_80]
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) ~[na:1.7.0_80]
>         at java.lang.Thread.run(Thread.java:745) ~[na:1.7.0_80]
>
>
>
>
>
>
> Regards,
> Adel
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org