PolicyEntry Memory Limits

[Brett Humphreys <br...@gmail.com>]

Hello all,
I had a question about a policy entry memory limit.  I'm creating a broker
with the following attributes:
        <systemUsage>
            <systemUsage sendFailIfNoSpace="true">
                <memoryUsage>
                    <memoryUsage limit="8 MB"/>
                </memoryUsage>
                <storeUsage>
                    <storeUsage limit="32 MB" store="#bridgeStore" />
                </storeUsage>
            </systemUsage>
        </systemUsage>

        <destinationPolicy>
            <policyMap>
                <policyEntries>
                    <policyEntry queue="EVICTIONTESTING" memoryLimit="4 MB"
advisoryWhenFull="true"/>
                </policyEntries>
            </policyMap>
        </destinationPolicy>

(full XML left out for brevity's sake)

This is in a unit test where I'm testing certain boundary cases
(specifically producers with no consumer).  My test is sending persistent
object messages with a payload of 1K, the total message size (according to
AMQ's calculation) is roughly 2.5K.  Since there is no consumer, and i'm
sending a lot of messages, I'm expecting resource allocation exceptions.
What I notice in this situation is that I get ResourceAllocation exceptions
from memory being full.  The exception description is:  "SystemUsage memory
limit reached".  However if I change my memory limit (on the policy entry)
from 4MB to >=6MB (or don't set it altogether) I don't get a resource
allocation exception from the memory limit being reached, but instead I get
it from the store being filled up (which is what my junit test case is
expecting).

Doing more digging, it seems that this occurs when the queue memory limit is
less than 70% of the systemUsage memory limit.  This seems to happen when
o.a.a.broker.region.cursors.StoreQueueCursor#addMessageLast adds a message
to the QueueStorePrefetch, the AbstractPendingMessageCursor's determination
of whether or not there is space available in memory is done via the
systemUsage's memory limit, not the memory limit set on the policy entry for
the particular queue.  So it keeps adding until the highwatermark is
reached, however since it calculating it from the systemusage, but adding it
to the specific queue's memoryUsage it runs out of memory.

Is this expected behavior?  I'm not quite sure I'm using the memory limits
correctly in the first place to assume there is a bug here.

As a side note, when I do get a  ResourceAllocationException from memory
being full I get an advisory message on ActiveMQ.Advisory.FULL.  However
when I get a ResourceAllocationException from the Store being full I don't
get an advisory (BaseDestination#isFull(...) only seems to be called by
o.a.a.broker.region.Queue for one of those cases, again, wasn't sure if I
was expecting something that I shouldn't or if this was an issue.

I'm currently using 5.2.0, I have 5.3.0, however from the digging I've done,
I believe 5.3.0 behaves the same way, however I haven't tested it yet.

Thanks a lot.  If these are issues, I'll certainly be happy to file them
with Jira.

-Brett Humphreys

Re: PolicyEntry Memory Limits

[Gary Tully <ga...@gmail.com>]

comments inline:

2009/10/29 Brett Humphreys <br...@gmail.com>
>
> Hello all,
> I had a question about a policy entry memory limit.  I'm creating a broker
> with the following attributes:
>        <systemUsage>
>            <systemUsage sendFailIfNoSpace="true">
>                <memoryUsage>
>                    <memoryUsage limit="8 MB"/>
>                </memoryUsage>
>                <storeUsage>
>                    <storeUsage limit="32 MB" store="#bridgeStore" />
>                </storeUsage>
>            </systemUsage>
>        </systemUsage>
>
>        <destinationPolicy>
>            <policyMap>
>                <policyEntries>
>                    <policyEntry queue="EVICTIONTESTING" memoryLimit="4 MB"
> advisoryWhenFull="true"/>
>                </policyEntries>
>            </policyMap>
>        </destinationPolicy>
>
> (full XML left out for brevity's sake)
>
> This is in a unit test where I'm testing certain boundary cases
> (specifically producers with no consumer).  My test is sending persistent
> object messages with a payload of 1K, the total message size (according to
> AMQ's calculation) is roughly 2.5K.  Since there is no consumer, and i'm
> sending a lot of messages, I'm expecting resource allocation exceptions.
> What I notice in this situation is that I get ResourceAllocation exceptions
> from memory being full.  The exception description is:  "SystemUsage memory
> limit reached".  However if I change my memory limit (on the policy entry)
> from 4MB to >=6MB (or don't set it altogether) I don't get a resource
> allocation exception from the memory limit being reached, but instead I get
> it from the store being filled up (which is what my junit test case is
> expecting).
>
This is expected behavior.

Spooling to disk is determined based on shared or global SystemUsage
memory limits (of 70% of). If this occurs on a queue before it's
memory limit is reached, it's memory limit will never be reached as
memory will be cleared as messages are spooled to disk.
The separation of concerns allows individual memory limits to
configure blocking etc without the possibility of any individual queue
exceeding SystemMemory usage and pushing the broker to an OOM.

If you want to have more deterministic behaviour w.r.t memory usage
you can use a vmQueueCursor that will never spool to disk. However
there is a risk that dynamically adding destinations could result in
an OOM as the there is now no consequence to exceeding the
SystemMemory usage.

> Doing more digging, it seems that this occurs when the queue memory limit is
> less than 70% of the systemUsage memory limit.  This seems to happen when
> o.a.a.broker.region.cursors.StoreQueueCursor#addMessageLast adds a message
> to the QueueStorePrefetch, the AbstractPendingMessageCursor's determination
> of whether or not there is space available in memory is done via the
> systemUsage's memory limit, not the memory limit set on the policy entry for
> the particular queue.  So it keeps adding until the highwatermark is
> reached, however since it calculating it from the systemusage, but adding it
> to the specific queue's memoryUsage it runs out of memory.
>
>
> Is this expected behavior?  I'm not quite sure I'm using the memory limits
> correctly in the first place to assume there is a bug here.

Do you see an OOM that is reproducibly here? If so, I think a jira is
in order  if it is reproducible with 5.3.

One thought, possibly the FilePendingMessageCursor will behave a little better.

> As a side note, when I do get a  ResourceAllocationException from memory
> being full I get an advisory message on ActiveMQ.Advisory.FULL.  However
> when I get a ResourceAllocationException from the Store being full I don't
> get an advisory (BaseDestination#isFull(...) only seems to be called by
> o.a.a.broker.region.Queue for one of those cases, again, wasn't sure if I
> was expecting something that I shouldn't or if this was an issue.
>
It would make sense to get an advisory when the store is full also I
guess but that is not implemented. That would be a different advisory
or a property on the same one. Probabally ActiveMQ.Advisory.FULL.MEM
and ActiveMQ.Advisory.FULL.STORE
Can you raise an jira for this requirement.

> I'm currently using 5.2.0, I have 5.3.0, however from the digging I've done,
> I believe 5.3.0 behaves the same way, however I haven't tested it yet.
>
Raise the issues against 5.3.0 if they are still present (which I
think they may be in these cases)

thanks.

> Thanks a lot.  If these are issues, I'll certainly be happy to file them
> with Jira.
>
> -Brett Humphreys



--
http://blog.garytully.com

Open Source Integration
http://fusesource.com