You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@activemq.apache.org by "Klaus Pittig (JIRA)" <ji...@apache.org> on 2016/01/11 10:14:39 UTC
[jira] [Commented] (AMQ-6115) No more browse/consume possible after
#checkpoint run
[ https://issues.apache.org/jira/browse/AMQ-6115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15091644#comment-15091644 ]
Klaus Pittig commented on AMQ-6115:
-----------------------------------
Corresponding mailing list thread: http://activemq.2283324.n4.nabble.com/How-to-avoid-blocking-of-queue-browsing-after-ActiveMQ-checkpoint-call-td4705696.html
Tim Bain requested some more information
{quote}
If you increase your JVM size (4GB, 8GB, etc., the biggest your OS and
hardware will support), does the behavior change? Does it truly take all
available memory, or just all the memory that you've made available to it
(which isn't tiny but really isn't all that big)?
Also, how do you know that the
MessageCursor seems to decide that there is not enough memory and stops
delivery of queue content to browsers/consumers? What symptom tells you
that?
{quote}
If I increase the JVM max heap size (4GB), the behavior does not change.
In my point of view, the configured memoryLimit (500 MB) works as
expected (heapdump shows same max. size for the TextMessage content,
i.e. 55002 byte[] instances containing 539 MB total).
However, trying to browse a queue shows no content, even if there is
enough heap memory available.
As far as i understand the sourcecode, this also due to the configured
memoryLimit, because - i hope this is the answer you expect - the
calculation for available causes hasSpace = false.
I found this here:
{code}
AbstractPendingMessageCursor {
public boolean hasSpace() {
return systemUsage != null ?
(!systemUsage.getMemoryUsage().isFull(memoryUsageHighWaterMark)) : true;
}
public boolean isFull() {
return systemUsage != null ? systemUsage.getMemoryUsage().isFull() :
false;
}
}
{code}
#hasSpace is in this case called during a click on a queue in the
Webconsole; see the 2 stacks during this workflow:
{noformat}
Daemon Thread [Queue:aaa114] (Suspended (breakpoint at line 107 in QueueStorePrefetch))
owns: QueueStorePrefetch (id=6036)
owns: StoreQueueCursor (id=6037)
owns: Object (id=6038)
QueueStorePrefetch.doFillBatch() line: 107
QueueStorePrefetch(AbstractStoreCursor).fillBatch() line: 381
QueueStorePrefetch(AbstractStoreCursor).reset() line: 142
StoreQueueCursor.reset() line: 159
Queue.doPageInForDispatch(boolean, boolean) line: 1897
Queue.pageInMessages(boolean) line: 2119
Queue.iterate() line: 1596
DedicatedTaskRunner.runTask() line: 112
DedicatedTaskRunner$1.run() line: 42
{noformat}
{noformat}
Daemon Thread [ActiveMQ VMTransport: vm://localhost#1] (Suspended (breakpoint at line 107 in QueueStorePrefetch))
owns: QueueStorePrefetch (id=5974)
owns: StoreQueueCursor (id=5975)
owns: Object (id=5976)
owns: Object (id=5977)
QueueStorePrefetch.doFillBatch() line: 107
QueueStorePrefetch(AbstractStoreCursor).fillBatch() line: 381
QueueStorePrefetch(AbstractStoreCursor).reset() line: 142
StoreQueueCursor.reset() line: 159
Queue.doPageInForDispatch(boolean, boolean) line: 1897
Queue.pageInMessages(boolean) line: 2119
Queue.iterate() line: 1596
Queue.wakeup() line: 1822
Queue.addSubscription(ConnectionContext, Subscription) line: 491
ManagedQueueRegion(AbstractRegion).addConsumer(ConnectionContext, ConsumerInfo) line: 399
ManagedRegionBroker(RegionBroker).addConsumer(ConnectionContext, ConsumerInfo) line: 427
ManagedRegionBroker.addConsumer(ConnectionContext, ConsumerInfo) line: 244
AdvisoryBroker(BrokerFilter).addConsumer(ConnectionContext, ConsumerInfo) line: 102
AdvisoryBroker.addConsumer(ConnectionContext, ConsumerInfo) line: 104
CompositeDestinationBroker(BrokerFilter).addConsumer(ConnectionContext, ConsumerInfo) line: 102
TransactionBroker(BrokerFilter).addConsumer(ConnectionContext, ConsumerInfo) line: 102
StatisticsBroker(BrokerFilter).addConsumer(ConnectionContext, ConsumerInfo) line: 102
BrokerService$5(MutableBrokerFilter).addConsumer(ConnectionContext,ConsumerInfo) line: 107
TransportConnection.processAddConsumer(ConsumerInfo) line: 663
ConsumerInfo.visit(CommandVisitor) line: 348
TransportConnection.service(Command) line: 334
TransportConnection$1.onCommand(Object) line: 188
ResponseCorrelator.onCommand(Object) line: 116
MutexTransport.onCommand(Object) line: 50
VMTransport.iterate() line: 248
DedicatedTaskRunner.runTask() line: 112
DedicatedTaskRunner$1.run() line: 42
{noformat}
Setting queueBrowsePrefetch="1" and queuePrefetch="1" in the
PolicyEntry for queue=">" also has no effect.
> No more browse/consume possible after #checkpoint run
> -----------------------------------------------------
>
> Key: AMQ-6115
> URL: https://issues.apache.org/jira/browse/AMQ-6115
> Project: ActiveMQ
> Issue Type: Bug
> Components: activemq-leveldb-store, Broker, KahaDB
> Affects Versions: 5.5.1, 5.11.2, 5.13.0
> Environment: OS=Linux,MacOS,Windows, Java=1.7,1.8, Xmx=1024m, SystemUsage Memory Limit 500 MB, Temp Limit 1 GB, Storage 80 GB
> Reporter: Klaus Pittig
> Attachments: Bildschirmfoto 2016-01-08 um 12.09.34.png, Bildschirmfoto 2016-01-08 um 13.29.08.png
>
>
> We are currently facing a problem when Using ActiveMQ with a large number of Persistence Queues (250) á 1000 persistent TextMessages á 10 KB.
> Our scenario requires these messages to remain in the storage over a long time (days), until they are consumed (large amounts of data are staged for distribution for many consumer, that could be offline for some days).
> This issue is independent of the JVM, OS and PersistentAdapter (KahaDB, LevelDB) with enough free space and memory.
> We tested this behaviour with ActiveMQ: 5.11.2, 5.13.0 and 5.5.1.
> After the Persistence Store is filled with these Messages (we use a simple unit test for production always the same message) and a broker restart, we can browse/consume some Queues _until_ the #checkpoint call after 30 seconds.
> This call causes the broker to use all available memory and never releases it for other tasks such as Queue browse/consume. Internally the MessageCursor seems to decide, that there is not enough memory and stops delivery of queue content to browsers/consumers.
> => Is there a way to avoid this behaviour of fix this?
> The expectation is, that we can consume/browse any queue under all circumstances.
> Besides the above mentioned settings we use the following settings for the broker (btw: changing the memoryLimit to a lower value like 1mb does not change the situation):
> {code:xml}
> <destinationPolicy>
> <policyMap>
> <policyEntries>
> <policyEntry queue=">" producerFlowControl="false"
> optimizedDispatch="true" memoryLimit="128mb">
> <dispatchPolicy>
> <strictOrderDispatchPolicy />
> </dispatchPolicy>
> <pendingQueuePolicy>
> <storeCursor/>
> </pendingQueuePolicy>
> </policyEntry>
> </policyEntries>
> </policyMap>
> </destinationPolicy>
> <systemUsage>
> <systemUsage sendFailIfNoSpace="true">
> <memoryUsage>
> <memoryUsage limit="500 mb"/>
> </memoryUsage>
> <storeUsage>
> <storeUsage limit="80000 mb"/>
> </storeUsage>
> <tempUsage>
> <tempUsage limit="1000 mb"/>
> </tempUsage>
> </systemUsage>
> </systemUsage>
> {code}
> If we set the *cursorMemoryHighWaterMark* in the destinationPolicy to a higher value like *150* or *600* depending on the difference between memoryUsage and the available heap space relieves the situation a bit for a workaround, but this is not really an option for production systems in my point of view.
> Attached some information from Oracle Mission Control and JProfiler showing those ActiveMQTextMessage instances that are never released from memory.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)