You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by Yarek Tyshchenko <ya...@affiliatewindow.com> on 2014/05/19 15:51:28 UTC
Hanging consumers when storeCursor gets to watermark
Hi all
I'm seeing a really strange issue with ActiveMQ that is present in the latest version 5.9.1 pretty much out of the box.
When one of the queues uses up all the cursor memory all consumers (except on that queue) stop consuming messages.
Producing messages is still fine, and there doesn't seem to be any upper limits on number of messages that
ActiveMQ can persist (below the limits on the persistent store's disk size obviously)
To replicate the problem:
0. Unpack fresh ActiveMQ release
1. Start ActiveMQ
2. Stop ActiveMQ
3. Start ActiveMQ again.
4. Start one producer on queue "bigQueue"
5. Start one consumer on queue "smallQueue"
6. Start one producer sending 4 messages per second to queue "smallQueue"
7. Wait until queue memory and cursor usage gets to the water mark
8. Observe that "smallQueue" dequeue count stops going up, and messages pile up
Notes: Configure a small global memory limit to make the test faster (<memoryUsage limit="32 mb" />).
The bug happens with any value. Consuming from "bigQueue" seems to free up some
resources which allows "smallQueue" to be consumed, until bigQueue fills up gain,
which is when everything stops.
The really strange part is this behaviour only shows up if activemq has been started
with an existing persistent database (KahaDB or LevelDB, happens with both).
Omitting steps #2 and #3 seems to keep the broker in perfect working condition.
The cursor usage goes up to the high water mark, in my case floats consistently around 69%.
When however ActiveMQ has been restarted and usage hits 70% on that queue consuming stops immediately.
I've tested several variations of options: useCache, splitSystemUsageForProducersConsumers,
producerFlowControl and optimizedDispatch to no avail.
Kind Regards
Yarek Tyshchenko
Affiliate Window
__________________________________________________________________________
Affiliate Window (a division of Digital Window Limited) disclaimer:
This e-mail is confidential to the intended recipient. If you have
received it in error, please notify the sender and delete it from your
system. Any unauthorised use, disclosure, or copying is not permitted.
This e-mail has been scanned for viruses, but no liability is accepted
for any damage caused by any content transmitted by this e-mail.
Digital Window Limited is registered as a private limited company in
England and Wales with number 04010229.
Registered Address:
3rd Floor
100 Leman Street
London
E1 8EU
United Kingdom
VAT No: GB766 0309 30
__________________________________________________________________________
This email message has been scanned by the Mimecast Email Security System.
__________________________________________________________________________
Re: Hanging consumers when storeCursor gets to watermark
Posted by jvbrandis <jo...@mnemonic.no>.
Yarek Tyshchenko wrote
> 4. Start one producer on queue "bigQueue"
> 5. Start one consumer on queue "smallQueue"
> 6. Start one producer sending 4 messages per second to queue "smallQueue"
> 7. Wait until queue memory and cursor usage gets to the water mark
> 8. Observe that "smallQueue" dequeue count stops going up, and messages
> pile up
Just posted my (similar) problem to the user list. The essence of the
problem is:
* start one producer and consumer on queue "Queue"
* stop the consumer
* let the queue reach the watermark, the producer will hang on send()
* start the consumer again. It will not be able to consume messages.
Stopping the producer does not help.
--
View this message in context: http://activemq.2283324.n4.nabble.com/Hanging-consumers-when-storeCursor-gets-to-watermark-tp4681249p4681335.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.