[jira] [Commented] (AMQ-5202) Default cursor exceeds allocated memory on broker restart

["Ben O'Day (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/AMQ-5202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14019575#comment-14019575 ] 

Ben O'Day commented on AMQ-5202:
--------------------------------

we were able to deal with this issue by setting a per queue memoryLimit as follows:
    <policyEntry queue=">" producerFlowControl="false" memoryLimit="100mb">

that said, if you have a sufficient number of queues where your <memoryLimit> X <number of queues> approaches the high water mark (70% of memUsage), then I suspect you'll run into the issue again...

> Default cursor exceeds allocated memory on broker restart
> ---------------------------------------------------------
>
>                 Key: AMQ-5202
>                 URL: https://issues.apache.org/jira/browse/AMQ-5202
>             Project: ActiveMQ
>          Issue Type: Bug
>          Components: Broker
>    Affects Versions: 5.9.0, 5.9.1
>         Environment: Linux, OSX
>            Reporter: Jakub Korab
>
> This seems to be related to solution that was implemented in AMQ-4495.
> Background:
> Publishing to queue:one at a high rate (loop, 10 producers), no consumers
> Publishing to queue:two at a low rate (2 msg/s), 1 consumer
> When using a fresh broker (no data directory) with LevelDB/KahaDB:
> Memory usage + cursor usage rise together on queue:one until they hit 69%; this corresponds to disk usage of 90mb (69% of tempUsage - under 70% default memory watermark for the cursor);
> queue:one keeps accepting messages.
> queue:two keeps accepting messages, and handing them off to the consumer
> This is reached when ~10000 * 4kb messages are sent. All OK so far.
> Issue:
> Restart the broker.
> Memory usage + cursor usage rise together very slowly on queue:one until they hit 70% (default policyEntry@cursorMemoryHighWaterMark)
> queue:one keeps accepting messages.
> queue:two keeps accepting messages, but _stops handing them off to the consumer_
> It doesn't matter whether the number of messages in the store is under or over the cursor watermark when the broker is restarted - the cursor usage hits the limit and the messages stop flowing.
> Consuming some messages from queue:one at this point will free up space up on the cursor, and messages start flowing through to the queue:two consumers. The cursor will eventually hit the watermark again and the consumers will again block, so it is not a self-healing issue.
> Deleting the queues does not change that situation, and neither consumers on queue:one nor queue:two will receive messages. The cursor seems to keep a handle on the old queue:one's reference.



--
This message was sent by Atlassian JIRA
(v6.2#6252)