You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by ws07 <ws...@ecs.soton.ac.uk> on 2010/02/21 23:49:57 UTC
Problem: Failed to page in more queue messages
Hi,
I try to do a performance test of my application. I submit a huge number of
messages
to my application that consist of several message queues and measure a
system throughput.
I use difference size of messages 10 bytes, 100 bytes and 1 Kbytes in this
test.
The problem is that when I run 1 Kbytes - 1,000,000 messages. I found
"java.lang.OutOfMemoryError: Java heap space". Because I set memory limit =
100 MB and JVM option -Xmx2600m (~2.5GB), I think
the OOM problem should not be occured. Below is the exception message:
Feb 21, 2010 11:05:44 PM org.apache.activemq.broker.region.Queue iterate
SEVERE: Failed to page in more queue messages
java.lang.OutOfMemoryError: Java heap space
at java.util.ArrayList.ensureCapacity(ArrayList.java:169)
at java.util.ArrayList.add(ArrayList.java:351)
at
org.apache.activemq.broker.region.Queue.doDispatch(Queue.java:1497)
at
org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1578)
at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1247)
at
org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:122)
at
org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:43)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
at java.lang.Thread.run(Thread.java:595)
Feb 21, 2010 11:05:46 PM org.apache.activemq.broker.TransportConnection
serviceException
WARNING: Async error occurred: java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
at
java.util.concurrent.CopyOnWriteArrayList.remove(CopyOnWriteArrayList.java:438)
at
org.apache.activemq.broker.region.PrefetchSubscription.acknowledge(PrefetchSubscription.java:270)
at
org.apache.activemq.broker.region.AbstractRegion.acknowledge(AbstractRegion.java:382)
at
org.apache.activemq.broker.region.RegionBroker.acknowledge(RegionBroker.java:490)
at
org.apache.activemq.broker.TransactionBroker.acknowledge(TransactionBroker.java:194)
at
org.apache.activemq.broker.BrokerFilter.acknowledge(BrokerFilter.java:75)
at
org.apache.activemq.broker.BrokerFilter.acknowledge(BrokerFilter.java:75)
at
org.apache.activemq.broker.BrokerFilter.acknowledge(BrokerFilter.java:75)
at
org.apache.activemq.broker.MutableBrokerFilter.acknowledge(MutableBrokerFilter.java:85)
at
org.apache.activemq.broker.TransportConnection.processMessageAck(TransportConnection.java:449)
at org.apache.activemq.command.MessageAck.visit(MessageAck.java:205)
at
org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:297)
at
org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:175)
at
org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:116)
at
org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:68)
at
org.apache.activemq.transport.vm.VMTransport.iterate(VMTransport.java:207)
at
org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:122)
at
org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:43)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
at java.lang.Thread.run(Thread.java:595)
I use the lastest snapshot - Activemq 5.4 -, Non-persistent deliverly and
asyncSend = true.
In addition, I try to reduce prefetch limit from the default value - 1000 -
to 10. It sometime can
solve this problem. I don't understand why. Can anyone has an idea to solve
this problem? (I also attach memory usage and threads information that I
capture from Jmx console. You can see the memory increase until it reach
maximum size and then the OOM error is occure)
http://old.nabble.com/file/p27677296/memory.png memory.png
http://old.nabble.com/file/p27677296/threads.png threads.png
--
View this message in context: http://old.nabble.com/Problem%3A-Failed-to-page-in-more-queue-messages-tp27677296p27677296.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Problem: Failed to page in more queue messages
Posted by ws07 <ws...@ecs.soton.ac.uk>.
Hi Rob,
Thanks for answering my question.
Yes, I am running producer, consumers and broker in the same JVM.
1 consumer for each queue. I'm trying to set the minimumMessageSize
on destinations (to 2048 bytes) as your suggestion, but the OOM error still
occurs.
It seem to me that the broker does not release memeory after messages are
consumed.
In addition, it's only happen when I increase size of messages to 1kb.
Do you have another suggestion that can help me to overcome this problem?
(I've attached memory usage information when I set minimumMessageSize =
2048) http://old.nabble.com/file/p27698165/memory1024_2048.jpg memory.jpg
Thanks,
Pete
--
View this message in context: http://old.nabble.com/Problem%3A-Failed-to-page-in-more-queue-messages-tp27677296p27698165.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Problem: Failed to page in more queue messages
Posted by Rob Davies <ra...@gmail.com>.
Looks you are running producers/consumers/broker all in the same JVM
using vm:// transport?
As messages won't be serialized going across the vm:// transport -
ActiveMQ will not be able to calculate how much memory each message
takes and apply floe control or offline the messages accordingly.
You can set the minimumMessageSize on a Destination (default is 1024
bytes) - and make it bigger - I'd suggest 2048 - as a message is
always going to take up more memory than just its payload.
To configure the Destination - you need to set a Policy for it in the
broker configuration - see http://activemq.apache.org/per-destination-policies.html
cheers,
Rob
On 21 Feb 2010, at 22:49, ws07 wrote:
>
> Hi,
>
> I try to do a performance test of my application. I submit a huge
> number of
> messages
> to my application that consist of several message queues and measure a
> system throughput.
> I use difference size of messages 10 bytes, 100 bytes and 1 Kbytes
> in this
> test.
>
> The problem is that when I run 1 Kbytes - 1,000,000 messages. I found
> "java.lang.OutOfMemoryError: Java heap space". Because I set memory
> limit =
> 100 MB and JVM option -Xmx2600m (~2.5GB), I think
> the OOM problem should not be occured. Below is the exception message:
>
> Feb 21, 2010 11:05:44 PM org.apache.activemq.broker.region.Queue
> iterate
> SEVERE: Failed to page in more queue messages
> java.lang.OutOfMemoryError: Java heap space
> at java.util.ArrayList.ensureCapacity(ArrayList.java:169)
> at java.util.ArrayList.add(ArrayList.java:351)
> at
> org.apache.activemq.broker.region.Queue.doDispatch(Queue.java:1497)
> at
> org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:
> 1578)
> at org.apache.activemq.broker.region.Queue.iterate(Queue.java:
> 1247)
> at
> org
> .apache
> .activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:122)
> at
> org.apache.activemq.thread.PooledTaskRunner
> $1.run(PooledTaskRunner.java:43)
> at
> java.util.concurrent.ThreadPoolExecutor
> $Worker.runTask(ThreadPoolExecutor.java:650)
> at
> java.util.concurrent.ThreadPoolExecutor
> $Worker.run(ThreadPoolExecutor.java:675)
> at java.lang.Thread.run(Thread.java:595)
> Feb 21, 2010 11:05:46 PM
> org.apache.activemq.broker.TransportConnection
> serviceException
> WARNING: Async error occurred: java.lang.OutOfMemoryError: Java heap
> space
> java.lang.OutOfMemoryError: Java heap space
> at
> java
> .util
> .concurrent.CopyOnWriteArrayList.remove(CopyOnWriteArrayList.java:438)
> at
> org
> .apache
> .activemq
> .broker
> .region.PrefetchSubscription.acknowledge(PrefetchSubscription.java:
> 270)
> at
> org
> .apache
> .activemq
> .broker.region.AbstractRegion.acknowledge(AbstractRegion.java:382)
> at
> org
> .apache
> .activemq.broker.region.RegionBroker.acknowledge(RegionBroker.java:
> 490)
> at
> org
> .apache
> .activemq
> .broker.TransactionBroker.acknowledge(TransactionBroker.java:194)
> at
> org
> .apache.activemq.broker.BrokerFilter.acknowledge(BrokerFilter.java:75)
> at
> org
> .apache.activemq.broker.BrokerFilter.acknowledge(BrokerFilter.java:75)
> at
> org
> .apache.activemq.broker.BrokerFilter.acknowledge(BrokerFilter.java:75)
> at
> org
> .apache
> .activemq
> .broker.MutableBrokerFilter.acknowledge(MutableBrokerFilter.java:85)
> at
> org
> .apache
> .activemq
> .broker
> .TransportConnection.processMessageAck(TransportConnection.java:449)
> at
> org.apache.activemq.command.MessageAck.visit(MessageAck.java:205)
> at
> org
> .apache
> .activemq
> .broker.TransportConnection.service(TransportConnection.java:297)
> at
> org.apache.activemq.broker.TransportConnection
> $1.onCommand(TransportConnection.java:175)
> at
> org
> .apache
> .activemq
> .transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:116)
> at
> org
> .apache
> .activemq.transport.TransportFilter.onCommand(TransportFilter.java:68)
> at
> org
> .apache.activemq.transport.vm.VMTransport.iterate(VMTransport.java:
> 207)
> at
> org
> .apache
> .activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:122)
> at
> org.apache.activemq.thread.PooledTaskRunner
> $1.run(PooledTaskRunner.java:43)
> at
> java.util.concurrent.ThreadPoolExecutor
> $Worker.runTask(ThreadPoolExecutor.java:650)
> at
> java.util.concurrent.ThreadPoolExecutor
> $Worker.run(ThreadPoolExecutor.java:675)
> at java.lang.Thread.run(Thread.java:595)
>
>
> I use the lastest snapshot - Activemq 5.4 -, Non-persistent
> deliverly and
> asyncSend = true.
> In addition, I try to reduce prefetch limit from the default value -
> 1000 -
> to 10. It sometime can
> solve this problem. I don't understand why. Can anyone has an idea
> to solve
> this problem? (I also attach memory usage and threads information
> that I
> capture from Jmx console. You can see the memory increase until it
> reach
> maximum size and then the OOM error is occure)
> http://old.nabble.com/file/p27677296/memory.png memory.png
> http://old.nabble.com/file/p27677296/threads.png threads.png
>
> --
> View this message in context: http://old.nabble.com/Problem%3A-Failed-to-page-in-more-queue-messages-tp27677296p27677296.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Rob Davies
http://twitter.com/rajdavies
I work here: http://fusesource.com
My Blog: http://rajdavies.blogspot.com/
I'm writing this: http://www.manning.com/snyder/