Flow control, memory limits and persistent store confusion

[Dragisa Krsmanovic <dk...@plos.org>]

We've set up AMQ to persist all messages:

<broker xmlns="http://activemq.apache.org/schema/core"
        brokerName="localhost"
        useJmx="true"
        persistent="true">

    <persistenceAdapter>
      <amqPersistenceAdapter directory="/foo"/>
    </persistenceAdapter>


We are using flow control for all queues (13 of them):
<policyEntry queue=">" producerFlowControl="true" memoryLimit="10mb"/>

System usage is set as follows:

  <systemUsage>
      <systemUsage>
        <memoryUsage>
          <memoryUsage limit="180 mb"/>
        </memoryUsage>
        <storeUsage>
          <storeUsage limit="3 gb" name="plos-queue"/>
        </storeUsage>
        <tempUsage>
          <tempUsage limit="100 mb"/>
        </tempUsage>
      </systemUsage>
    </systemUsage>

We have slow consumers and messages are sent in bursts of about 20,000.
I was expecting, since we persist all the messages, that I should be
able to put all 20k of them in the queue quickly and have consumers
slowly work on them.

Instead, it looks like none of the queues can have more than 140-150
messages at the time and flow control is making producer wait. It's like
queues are limited by the number of messages they can have in those 10m
of memory. Aren't persistent messages limited by store size ?

How do I achieve what I want ? 

What would be a consequence in turning off flow control ?

Thanks

-- 
Dragisa Krsmanovic
Java Developer
Public Library of Science

--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
This email is confidential to the intended recipient. If you have received it in error, please notify the sender and delete it from your
system. Any unauthorized use, disclosure or copying is not permitted. The views or opinions presented are solely those of the sender and do
not necessarily represent those of Public Library of Science unless otherwise specifically stated. Please note that neither Public Library
of Science nor any of its agents accept any responsibility for any viruses that may be contained in this e-mail or its attachments and it
is your responsibility to scan the e-mail and attachments (if any).

Re: Flow control, memory limits and persistent store confusion

[Joe Fernandez <jo...@ttmsolutions.com>]

AMQ's producer flow control mechanism kicks in if one of the following
conditions occurs:

- The corresponding destination’s (queue) maximum memory usage limit has
been reached 
- The message store’s maximum capacity has been reached. 

If you don’t want your producer to block or hang when flow control kicks in,
you can enable the “sendFailIfNoSpace” system usage property. When this
property is enabled and the broker reaches a store usage limit, instead of
blocking the producer’s transport it will force an exception to be thrown by
the producer’s send method. Thus your producer will not hang and can instead
react accordingly to the thrown exception. 

Joe 
http://www.ttmsolutions.com



Dragisa Krsmanovic wrote:
> 
> We've set up AMQ to persist all messages:
> 
> <broker xmlns="http://activemq.apache.org/schema/core"
>         brokerName="localhost"
>         useJmx="true"
>         persistent="true">
> 
>     <persistenceAdapter>
>       <amqPersistenceAdapter directory="/foo"/>
>     </persistenceAdapter>
> 
> 
> We are using flow control for all queues (13 of them):
> <policyEntry queue=">" producerFlowControl="true" memoryLimit="10mb"/>
> 
> System usage is set as follows:
> 
>   <systemUsage>
>       <systemUsage>
>         <memoryUsage>
>           <memoryUsage limit="180 mb"/>
>         </memoryUsage>
>         <storeUsage>
>           <storeUsage limit="3 gb" name="plos-queue"/>
>         </storeUsage>
>         <tempUsage>
>           <tempUsage limit="100 mb"/>
>         </tempUsage>
>       </systemUsage>
>     </systemUsage>
> 
> We have slow consumers and messages are sent in bursts of about 20,000.
> I was expecting, since we persist all the messages, that I should be
> able to put all 20k of them in the queue quickly and have consumers
> slowly work on them.
> 
> Instead, it looks like none of the queues can have more than 140-150
> messages at the time and flow control is making producer wait. It's like
> queues are limited by the number of messages they can have in those 10m
> of memory. Aren't persistent messages limited by store size ?
> 
> How do I achieve what I want ? 
> 
> What would be a consequence in turning off flow control ?
> 
> Thanks
> 
> -- 
> Dragisa Krsmanovic
> Java Developer
> Public Library of Science
> 
> --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
> This email is confidential to the intended recipient. If you have received
> it in error, please notify the sender and delete it from your
> system. Any unauthorized use, disclosure or copying is not permitted. The
> views or opinions presented are solely those of the sender and do
> not necessarily represent those of Public Library of Science unless
> otherwise specifically stated. Please note that neither Public Library
> of Science nor any of its agents accept any responsibility for any viruses
> that may be contained in this e-mail or its attachments and it
> is your responsibility to scan the e-mail and attachments (if any).
> 
> 

-- 
View this message in context: http://old.nabble.com/Flow-control%2C-memory-limits-and-persistent-store-confusion-tp28833086p28833452.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.