You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@activemq.apache.org by "Jouni Mäki-Panula (JIRA)" <ji...@apache.org> on 2015/10/01 08:33:05 UTC
[jira] [Created] (AMQ-5993) Purging messages leads to GC overhead
limit exceeded
Jouni Mäki-Panula created AMQ-5993:
--------------------------------------
Summary: Purging messages leads to GC overhead limit exceeded
Key: AMQ-5993
URL: https://issues.apache.org/jira/browse/AMQ-5993
Project: ActiveMQ
Issue Type: Bug
Affects Versions: 5.11.1
Reporter: Jouni Mäki-Panula
Priority: Minor
Purging a large number of messages from a queue in ActiveMQ 5.11.1 seems to lead to GC overhead limit exceeded error state.
I've witnessed this behavior twice during the last month. In the first case, 3 314 226 messages were purged. 12 minutes later, the broker basically halted and log started to fill with messages like these:
{noformat}
| WARN | Transport Connection to: tcp://172.20.50.174:56005 failed: java.io.IOException: GC overhead limit exceeded | org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ NIO Worker 27070
| ERROR | Could not accept connection : java.lang.Exception: java.lang.OutOfMemoryError: GC overhead limit exceeded | org.apache.activemq.broker.TransportConnector | ActiveMQ Transport Server Thread Handler: stomp+nio://0.0.0.0:61613?maximumConnections=1400&wireFormat.maxFrameSize=104857600&transport.keepAlive=true&transport.soLinger=30
| WARN | Async error occurred: | org.apache.activemq.broker.TransportConnection.Service | ActiveMQ BrokerService[localhost] Task-22496
java.lang.OutOfMemoryError: GC overhead limit exceeded
| WARN | Exception occurred processing:
null: java.lang.OutOfMemoryError: GC overhead limit exceeded | org.apache.activemq.transport.stomp.ProtocolConverter | ActiveMQ BrokerService[localhost] Task-22506
{noformat}
In the second case, 2 380 871 messages were purged. This time it took a bit longer for the garbage collection to kick in, but it did, causing similar error state 6 hours and 14 minutes later:
{noformat}
| WARN | Async error occurred: | org.apache.activemq.broker.TransportConnection.Service | ActiveMQ NIO Worker 16937
java.lang.OutOfMemoryError: GC overhead limit exceeded
at org.apache.activemq.store.kahadb.disk.journal.DataFileAppender.storeItem(DataFileAppender.java:142)[activemq-kahadb-store-5.11.1.jar:5.11.1]
at org.apache.activemq.store.kahadb.disk.journal.Journal.write(Journal.java:627)[activemq-kahadb-store-5.11.1.jar:5.11.1]
at org.apache.activemq.store.kahadb.plist.PListStoreImpl.write(PListStoreImpl.java:403)[activemq-kahadb-store-5.11.1.jar:5.11.1]
at org.apache.activemq.store.kahadb.plist.PListImpl.addLast(PListImpl.java:100)[activemq-kahadb-store-5.11.1.jar:5.11.1]
at org.apache.activemq.broker.region.cursors.FilePendingMessageCursor.flushToDisk(FilePendingMessageCursor.java:440)[activemq-broker-5.11.1.jar:5.11.1]
at org.apache.activemq.broker.region.cursors.FilePendingMessageCursor.tryAddMessageLast(FilePendingMessageCursor.java:231)[activemq-broker-5.11.1.jar:5.11.1]
at org.apache.activemq.broker.region.cursors.FilePendingMessageCursor.addMessageLast(FilePendingMessageCursor.java:207)[activemq-broker-5.11.1.jar:5.11.1]
at org.apache.activemq.broker.region.cursors.StoreQueueCursor.addMessageLast(StoreQueueCursor.java:97)[activemq-broker-5.11.1.jar:5.11.1]
at org.apache.activemq.broker.region.Queue.cursorAdd(Queue.java:1796)[activemq-broker-5.11.1.jar:5.11.1]
at org.apache.activemq.broker.region.Queue.orderedCursorAdd(Queue.java:878)[activemq-broker-5.11.1.jar:5.11.1]
at org.apache.activemq.broker.region.Queue.doMessageSend(Queue.java:854)[activemq-broker-5.11.1.jar:5.11.1]
at org.apache.activemq.broker.region.Queue.send(Queue.java:733)[activemq-broker-5.11.1.jar:5.11.1]
at org.apache.activemq.broker.region.AbstractRegion.send(AbstractRegion.java:419)[activemq-broker-5.11.1.jar:5.11.1]
at org.apache.activemq.broker.region.RegionBroker.send(RegionBroker.java:468)[activemq-broker-5.11.1.jar:5.11.1]
at org.apache.activemq.broker.jmx.ManagedRegionBroker.send(ManagedRegionBroker.java:297)[activemq-broker-5.11.1.jar:5.11.1]
at org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:152)[activemq-broker-5.11.1.jar:5.11.1]
at org.apache.activemq.broker.CompositeDestinationBroker.send(CompositeDestinationBroker.java:96)[activemq-broker-5.11.1.jar:5.11.1]
at org.apache.activemq.broker.TransactionBroker.send(TransactionBroker.java:307)[activemq-broker-5.11.1.jar:5.11.1]
at org.apache.activemq.broker.MutableBrokerFilter.send(MutableBrokerFilter.java:157)[activemq-broker-5.11.1.jar:5.11.1]
at org.apache.activemq.broker.util.LoggingBrokerPlugin.send(LoggingBrokerPlugin.java:275)[activemq-broker-5.11.1.jar:5.11.1]
at org.apache.activemq.broker.MutableBrokerFilter.send(MutableBrokerFilter.java:157)[activemq-broker-5.11.1.jar:5.11.1]
at org.apache.activemq.broker.TransportConnection.processMessage(TransportConnection.java:541)[activemq-broker-5.11.1.jar:5.11.1]
at org.apache.activemq.command.ActiveMQMessage.visit(ActiveMQMessage.java:768)[activemq-client-5.11.1.jar:5.11.1]
at org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:334)[activemq-broker-5.11.1.jar:5.11.1]
at org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:188)[activemq-broker-5.11.1.jar:5.11.1]
at org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:45)[activemq-client-5.11.1.jar:5.11.1]
at org.apache.activemq.transport.AbstractInactivityMonitor.onCommand(AbstractInactivityMonitor.java:270)[activemq-client-5.11.1.jar:5.11.1]
at org.apache.activemq.transport.stomp.StompTransportFilter.sendToActiveMQ(StompTransportFilter.java:87)[activemq-stomp-5.11.1.jar:5.11.1]
at org.apache.activemq.transport.stomp.ProtocolConverter.sendToActiveMQ(ProtocolConverter.java:199)[activemq-stomp-5.11.1.jar:5.11.1]
at org.apache.activemq.transport.stomp.ProtocolConverter.onStompSend(ProtocolConverter.java:332)[activemq-stomp-5.11.1.jar:5.11.1]
at org.apache.activemq.transport.stomp.ProtocolConverter.onStompCommand(ProtocolConverter.java:245)[activemq-stomp-5.11.1.jar:5.11.1]
at org.apache.activemq.transport.stomp.StompTransportFilter.onCommand(StompTransportFilter.java:75)[activemq-stomp-5.11.1.jar:5.11.1]
| WARN | Exception occurred processing:
null: java.lang.OutOfMemoryError: GC overhead limit exceeded | org.apache.activemq.transport.stomp.ProtocolConverter | ActiveMQ BrokerService[localhost] Task-17265
| WARN | Transport Connection to: tcp://172.20.50.179:50888 failed: java.io.IOException: GC overhead limit exceeded | org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ NIO Worker 17016
| ERROR | Could not accept connection : java.lang.Exception: java.lang.OutOfMemoryError: GC overhead limit exceeded | org.apache.activemq.broker.TransportConnector | ActiveMQ Transport Server Thread Handler: stomp+nio://0.0.0.0:61613?maximumConnections=1400&wireFormat.maxFrameSize=104857600&transport.keepAlive=true&transport.soLinger=30
{noformat}
The only way out seemed to be restarting the broker.
This ActiveMQ instance is running on a server with 4 GB system memory and 1 GB heap size for ActiveMQ. Heap wasn't a problem in neither of the cases. In normal usage, there is around 2.5 GB free memory when counting OS caches, but only around 150 MB actual free memory. OS is RHEL 6.6.
The most obvious solution may be to simply increase system memory. Another workaround would be to dequeue messages from the queue with a consumer instead of purging them.
But still it seems odd that ActiveMQ is able to handle millions of messages if they are dequeued, but not if they are purged. Should the purge operation be improved somehow? And is increasing system memory the best option here?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)