You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by Juan Nin <ju...@gmail.com> on 2012/11/16 18:29:00 UTC
Understanding memoryUsage (once again!)
Hi!
After some heavy digging about Producer Flow control and the systemUsage
properties a couple of years ago, I thought I quite understood it.
But yesterday I found that one of my configs was not behaving exactly as I
expected, so started doing some tests, and I see certain behaviours which
don't seem to match what the docs and posts that I find on the list or
other forums say.
"storeUsage" is perfectly clear, it's the max space that persistent
messages can use to be stored in disk.
"tempUsage"" applies to file cursors on non-persistent messages, so as to
flush to disk if memory limits are reached (I don't care much about this
one anyway, I always use persistent messages).
Now, according to most posts, memoryUsage would be the maximum memory that
the broker would be available to use.
On this post:
http://stackoverflow.com/questions/7646057/activemq-destinationpolicy-and-systemusage-configurationit
says that "memoryUsage corresponds to the amount of memory that's
assigned to the in-memory store".
For example, on my tests using the following config (only showing relevant
parts):
<policyEntry queue=">" producerFlowControl="false" optimizedDispatch="true">
<deadLetterStrategy>
<individualDeadLetterStrategy queuePrefix="DLQ."
useQueueForQueueMessages="true" />
</deadLetterStrategy>
</policyEntry>
<systemUsage>
<systemUsage>
<memoryUsage>
<memoryUsage limit="100 mb"/>
</memoryUsage>
<storeUsage>
<storeUsage limit="1 gb" name="foo"/>
</storeUsage>
<tempUsage>
<tempUsage limit="3 gb"/>
</tempUsage>
</systemUsage>
</systemUsage>
With that config I would expect the broker to use 100 mb of maximum memory
among all queues. So it could maybe use 30mb in one queue and 70mb in
second queue.
1) What I'm seeing is that if I start feeding a queue without consuming it,
the "Memory percent used" grows up to 70%, after that it doesn't grow
anymore.
What is it doing exactly there? The first 70% is stored in memory (apart
from disk since it's persistent), and all the rest that continues being fed
goes just to disk?
2) If then I start feeding a 2nd queue, "Memory percent used" continues
growing until it reaches 140%. So it looks like memoryUsage does not apply
globally, but on a per queue basis?
Using memoryLimit on the queue's policyEntry gives more control over this,
but it's just a variation, "Memory percent used" can grow more than 100%
anyway.
3) If #2 is true, then how would I prevent the broker from running out of
memory in case queues would continue to be created?
Maybe I'm misunderstanding and some of these settings make no sense when
producerFlowControl is disabled?
Thanks in advance.
Juan
Re: Understanding memoryUsage (once again!)
Posted by Torsten Mielke <to...@fusesource.com>.
ActiveMQ 5.3 is fairly old now.
I would have though that 5.7 should honor the memoryUsage settings correctly.
Unfortunately I don't have the time at the moment to test this myself.
If you are certain that it also does not work in 5.7, perhaps raise a JIRA ticket on the ActiveMQ project. Ideally, attach a little unit test that reproduces this behavior.
Regards,
Torsten Mielke
torsten@fusesource.com
tmielke.blogspot.com
On Nov 16, 2012, at 7:38 PM, Juan Nin wrote:
> nope, adding a 3rd queue the 3rd one also gets this same value, so even if
> it's the memory usage of the queue it's anyway going beyond..
>
>
> On Fri, Nov 16, 2012 at 4:32 PM, Juan Nin <ju...@gmail.com> wrote:
>
>> Might it be just a bug on how the MemoryPercentUsage is calculated?
>>
>> If I connect via JMX using console, I can see the MemoryPercentUsage as
>> 112 right now.
>> If I go to each of the 2 queues on them I see CursorMemoryUsage with value
>> 29360604, which would be 28mb each, summing a total of 56mb (just a bit
>> more than the specified memoryUsage of 50mb).
>>
>> Not sure I'm interpreting these values correctly though, first time I
>> access it via jconsole...
>>
>>
>> On Fri, Nov 16, 2012 at 4:07 PM, Juan Nin <ju...@gmail.com> wrote:
>>
>>> On that config there's a 40mb memoryLimit per queue, but also tested it
>>> without it with same results.
>>>
>>>
>>> On Fri, Nov 16, 2012 at 4:05 PM, Juan Nin <ju...@gmail.com> wrote:
>>>
>>>> Hi Torsten!
>>>>
>>>> I'm using ActiveMQ 5.3.2, but also tested it on 5.7.0 with the same
>>>> results...
>>>> This is my 5.3.2 config:
>>>>
>>>> <beans
>>>> xmlns="http://www.springframework.org/schema/beans"
>>>> xmlns:amq="http://activemq.apache.org/schema/core"
>>>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
>>>> xsi:schemaLocation="http://www.springframework.org/schema/beans
>>>> http://www.springframework.org/schema/beans/spring-beans-2.0.xsd
>>>> http://activemq.apache.org/schema/core
>>>> http://activemq.apache.org/schema/core/activemq-core.xsd">
>>>>
>>>> <bean
>>>> class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
>>>> <property name="locations">
>>>>
>>>> <value>file:${activemq.base}/conf/credentials.properties</value>
>>>> </property>
>>>> </bean>
>>>>
>>>> <broker xmlns="http://activemq.apache.org/schema/core"
>>>> brokerName="localhost" dataDirectory="${activemq.base}/data"
>>>> destroyApplicationContextOnStop="true" advisorySupport="false">
>>>>
>>>> <destinationPolicy>
>>>> <policyMap>
>>>> <policyEntries>
>>>> <policyEntry topic=">" producerFlowControl="true"
>>>> memoryLimit="5mb">
>>>> <pendingSubscriberPolicy>
>>>> <vmCursor />
>>>> </pendingSubscriberPolicy>
>>>> </policyEntry>
>>>> <policyEntry queue=">" producerFlowControl="false"
>>>> optimizedDispatch="true" memoryLimit="40mb">
>>>> <deadLetterStrategy>
>>>> <individualDeadLetterStrategy
>>>> queuePrefix="DLQ." useQueueForQueueMessages="true" />
>>>> </deadLetterStrategy>
>>>> </policyEntry>
>>>> </policyEntries>
>>>> </policyMap>
>>>> </destinationPolicy>
>>>>
>>>> <managementContext>
>>>> <managementContext connectorPort="2011"/>
>>>> </managementContext>
>>>>
>>>> <persistenceAdapter>
>>>> <kahaDB directory="${activemq.base}/data/kahadb"
>>>> enableJournalDiskSyncs="false" indexWriteBatchSize="10000"
>>>> indexCacheSize="1000"/>
>>>> </persistenceAdapter>
>>>>
>>>> <systemUsage>
>>>> <systemUsage>
>>>> <memoryUsage>
>>>> <memoryUsage limit="50 mb"/>
>>>> </memoryUsage>
>>>> <storeUsage>
>>>> <storeUsage limit="1 gb" name="foo"/>
>>>> </storeUsage>
>>>> <tempUsage>
>>>> <tempUsage limit="3 gb"/>
>>>> </tempUsage>
>>>> </systemUsage>
>>>> </systemUsage>
>>>>
>>>> <transportConnectors>
>>>> <transportConnector name="openwire" uri="tcp://0.0.0.0:61616
>>>> "/>
>>>> <transportConnector name="stomp" uri="stomp://0.0.0.0:61613"/>
>>>> </transportConnectors>
>>>>
>>>> </broker>
>>>>
>>>> <import resource="jetty.xml"/>
>>>>
>>>> </beans>
>>>>
>>>>
>>>> Using just a simple PHP script with Stomp for feeding the queues
>>>> (running it twice with different queue name):
>>>>
>>>> <?php
>>>>
>>>> require_once("Stomp.php");
>>>>
>>>> $amq = new Stomp("tcp://localhost:61613");
>>>> $amq->connect();
>>>>
>>>> for($i=1; $i <= 100000; $i++)
>>>> {
>>>> if($i%1000 == 0)
>>>> {
>>>> echo "\nmsg #: $i";
>>>> }
>>>> $amq->send("/queue/test", "this is test message # $i"
>>>> ,array('persistent' => 'true'));
>>>> }
>>>>
>>>> $amq->disconnect();
>>>>
>>>> ?>
>>>>
>>>>
>>>>
>>>> On Fri, Nov 16, 2012 at 3:47 PM, Torsten Mielke <to...@fusesource.com>wrote:
>>>>
>>>>> Hello,
>>>>>
>>>>> See in-line response.
>>>>>
>>>>> On Nov 16, 2012, at 6:29 PM, Juan Nin wrote:
>>>>>
>>>>>> Hi!
>>>>>>
>>>>>> After some heavy digging about Producer Flow control and the
>>>>> systemUsage
>>>>>> properties a couple of years ago, I thought I quite understood it.
>>>>>> But yesterday I found that one of my configs was not behaving exactly
>>>>> as I
>>>>>> expected, so started doing some tests, and I see certain behaviours
>>>>> which
>>>>>> don't seem to match what the docs and posts that I find on the list or
>>>>>> other forums say.
>>>>>>
>>>>>> "storeUsage" is perfectly clear, it's the max space that persistent
>>>>>> messages can use to be stored in disk.
>>>>>> "tempUsage"" applies to file cursors on non-persistent messages, so
>>>>> as to
>>>>>> flush to disk if memory limits are reached (I don't care much about
>>>>> this
>>>>>> one anyway, I always use persistent messages).
>>>>>
>>>>> Correct.
>>>>>
>>>>>>
>>>>>> Now, according to most posts, memoryUsage would be the maximum memory
>>>>> that
>>>>>> the broker would be available to use.
>>>>>> On this post:
>>>>>>
>>>>> http://stackoverflow.com/questions/7646057/activemq-destinationpolicy-and-systemusage-configurationit
>>>>>> says that "memoryUsage corresponds to the amount of memory that's
>>>>>> assigned to the in-memory store".
>>>>>
>>>>> Correct.
>>>>>
>>>>>>
>>>>>> For example, on my tests using the following config (only showing
>>>>> relevant
>>>>>> parts):
>>>>>>
>>>>>> <policyEntry queue=">" producerFlowControl="false"
>>>>> optimizedDispatch="true">
>>>>>> <deadLetterStrategy>
>>>>>> <individualDeadLetterStrategy queuePrefix="DLQ."
>>>>>> useQueueForQueueMessages="true" />
>>>>>> </deadLetterStrategy>
>>>>>> </policyEntry>
>>>>>>
>>>>>> <systemUsage>
>>>>>> <systemUsage>
>>>>>> <memoryUsage>
>>>>>> <memoryUsage limit="100 mb"/>
>>>>>> </memoryUsage>
>>>>>> <storeUsage>
>>>>>> <storeUsage limit="1 gb" name="foo"/>
>>>>>> </storeUsage>
>>>>>> <tempUsage>
>>>>>> <tempUsage limit="3 gb"/>
>>>>>> </tempUsage>
>>>>>> </systemUsage>
>>>>>> </systemUsage>
>>>>>>
>>>>>> With that config I would expect the broker to use 100 mb of maximum
>>>>> memory
>>>>>> among all queues. So it could maybe use 30mb in one queue and 70mb in
>>>>>> second queue.
>>>>>>
>>>>>>
>>>>>> 1) What I'm seeing is that if I start feeding a queue without
>>>>> consuming it,
>>>>>> the "Memory percent used" grows up to 70%, after that it doesn't grow
>>>>>> anymore.
>>>>>> What is it doing exactly there? The first 70% is stored in memory
>>>>> (apart
>>>>>> from disk since it's persistent), and all the rest that continues
>>>>> being fed
>>>>>> goes just to disk?
>>>>>
>>>>> This behavior is correct. For queues the default cursor is store
>>>>> cursor. It keeps any newly arrived msgs in memory as long as it does not
>>>>> reach the configured memory limit (either configured on the queue per
>>>>> destination or globally in memoryUsage settings).
>>>>> Once the cursor reaches 70% of the configured limit (in your case of
>>>>> the memoryUsage limit since you don't specify a per-destination limit), it
>>>>> will not keep any more messages in memory.
>>>>> Instead it will reload these messages from the store when its time to
>>>>> dispatch them. The broker anyway persists any msgs it receives before
>>>>> passing on to the cursor.
>>>>> This limit of 70% can be configured and raised to e..g 100%.
>>>>> This behavior is kind of an optimization. That way you run less often
>>>>> into producer-flow-control.
>>>>> As long as the persistence store is not running full, there is no need
>>>>> to block producers, since the cursor can also load the messages from the
>>>>> store and does not necessarily have to keep them in memory.
>>>>> If you configure the vmQueueCursor, then the behavior is different.
>>>>> This cursor will not be able to load msgs to the store but needs to keep
>>>>> them all in memory. The vmQueueCursor used to be the default cursor in
>>>>> older version of AMQ.
>>>>>
>>>>> Also note that topic msgs and non-persistent queue messages are not
>>>>> handled by the store cursor. These msgs are held in memory and if memory
>>>>> runs low, get swapped out to temp storage.
>>>>>
>>>>>> 2) If then I start feeding a 2nd queue, "Memory percent used"
>>>>> continues
>>>>>> growing until it reaches 140%. So it looks like memoryUsage does not
>>>>> apply
>>>>>> globally, but on a per queue basis?
>>>>>
>>>>> What version of AMQ do you use? The sum of the memory usage of all
>>>>> queues should not go any higher than the configured memoryUsage limit. If
>>>>> you're not on 5.5.1 or higher releases, then I suggest to upgrade.
>>>>>
>>>>>> Using memoryLimit on the queue's policyEntry gives more control over
>>>>> this,
>>>>>> but it's just a variation, "Memory percent used" can grow more than
>>>>> 100%
>>>>>> anyway.
>>>>>
>>>>> With the default store cursor this should not be the case from what I
>>>>> know.
>>>>>
>>>>>
>>>>>>
>>>>>> 3) If #2 is true, then how would I prevent the broker from running
>>>>> out of
>>>>>> memory in case queues would continue to be created?
>>>>>
>>>>> Just like above comment. I would expect the brokers MemoryPercentUsage
>>>>> won't grow over 100% and the destinations MemoryPercentUsage remains fairly
>>>>> much at 70%.
>>>>> Not sure why you would see a different behavior? Using an old version
>>>>> of AMQ perhaps? Or explicitly configuring for the vmQueueCursor?
>>>>> Could you perhaps also test with
>>>>>
>>>>>>
>>>>>>
>>>>>> Maybe I'm misunderstanding and some of these settings make no sense
>>>>> when
>>>>>> producerFlowControl is disabled?
>>>>>>
>>>>>> Thanks in advance.
>>>>>>
>>>>>> Juan
>>>>>
>>>>>
>>>>> Regards,
>>>>>
>>>>> Torsten Mielke
>>>>> torsten@fusesource.com
>>>>> tmielke.blogspot.com
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
Re: Understanding memoryUsage (once again!)
Posted by Juan Nin <ju...@gmail.com>.
Awesome, thanks Gary!
On Fri, Nov 30, 2012 at 8:41 AM, Gary Tully <ga...@gmail.com> wrote:
> use the destinationpolicy entry cursorMemoryHighWaterMark, it default to
> 70% of the system usage. so set it to something low like 2% and it will
> spool to disk very early.
>
> When reading back in from disk, it pages in messages in batches, this is
> controlled by the maxPageSize policy enry for a destination. defaults to
> 200
>
>
> On 29 November 2012 23:42, Juan Nin <ju...@gmail.com> wrote:
>
> > Right now I set it to 5mb per queue, but I guess could be even less.
> >
> > I tested using "0 mb" to see if it completely flushed to disk without
> using
> > memory, but didn't work, in that case seems it's the same as not putting
> > the memoryLimit at all. Is there any way to make them just go to disk
> > without using memory at all?
> >
> > Now, once they go to disk, how does ActiveMQ process them upon
> consumption?
> > Does it load to memory in chunks or something like that and keeps them in
> > memory, or just grabs from disk as required?
> >
> > I'll have cases where consumptions will be one by one, sporadic by
> > different processes (intermittent).
> > But there will be other cases where even millions will be consumed at
> once
> > by a multithreaded Java app.
> >
> > So for the intermittent ones it would be ok if it just loads them from
> > disk, but for the other ones would be interesting if can load bigger
> chunks
> > into memory for faster consumption. Not sure if possible though.
> >
> > Thanks again.
> >
> >
> > On Thu, Nov 29, 2012 at 6:49 PM, Christian Posta
> > <ch...@gmail.com>wrote:
> >
> > > Another thought.. if you know they won't be immediately consumed (slow,
> > or
> > > even intermittent consumers), why keep them in memory at all? Or at
> least
> > > why keep such a large number of them? Maybe turn down the memory limits
> > on
> > > the destinations with knowingly slow consumers so fewer messages are
> kept
> > > in memory (they will be kept in store if they are persistent messages
> and
> > > then recovered when they are ready to be dispatched). The only downside
> > is
> > > if the consumers speed up, you'll be fetching from disk more often.
> > >
> > > Would be nice if the broker could auto-tune its memory usage... uh
> oh...
> > >
> > >
> > >
> > >
> > >
> > >
> > > On Thu, Nov 29, 2012 at 12:45 PM, Juan Nin <ju...@gmail.com> wrote:
> > >
> > > > Hi Christian!
> > > >
> > > > Yes, actually that's what I'm doing, just setting per destination
> > > policies
> > > > which work for me.
> > > > I anyway needed them because I'm creating queues with lots of
> messages,
> > > > which won't be immediately consumed, so having them store a lot into
> > > memory
> > > > ended up slowing things up.
> > > >
> > > > So I just assigned enough memory to the broker so as not run into
> > issues.
> > > >
> > > > Thanks again.
> > > >
> > > >
> > > > On Tue, Nov 27, 2012 at 9:40 PM, Christian Posta
> > > > <ch...@gmail.com>wrote:
> > > >
> > > > > See inline...
> > > > >
> > > > >
> > > > > On Wed, Nov 21, 2012 at 12:04 PM, Juan Nin <ju...@gmail.com>
> wrote:
> > > > >
> > > > > > Hi!
> > > > > >
> > > > > > Sorry for the delay in replying, buried on a project.
> > > > > >
> > > > > > As I mentioned before, I had tested this with 5.7.0 with the same
> > > > > > behaviour.
> > > > > > I just tested it again (both with 5.3.2 and 5.7.0) and same
> thing,
> > > and
> > > > on
> > > > > > my case it doesn't matter if there are consumers or not, it
> always
> > > > seems
> > > > > to
> > > > > > make usage of the memory.
> > > > > >
> > > > > > Although I guess in theory that should not affect, did you use
> > Stomp
> > > > for
> > > > > > your testing, or maybe you used Openwire?
> > > > > > I'm using Stomp for my testing.
> > > > > >
> > > > > > Might be though that the broker's memory itself is not going
> beyond
> > > 70%
> > > > > of
> > > > > > memoryUsage, but this is just per destination counters as you
> > > > mentioned.
> > > > > > In which case I guess the value shown as "Memory percent used"
> is a
> > > bit
> > > > > > confusing... But haven't had much time to really test the
> > possibility
> > > > of
> > > > > > exhausting the broker's memory.
> > > > > >
> > > > > No, i believe what you're seeing is correct. The broker's memory
> > limit
> > > is
> > > > > going beyond memoryUsage (way beyond). When a queue checks whether
> > > memory
> > > > > is full, it will only do something interesting if producer flow
> > control
> > > > is
> > > > > enabled. Otherwise, it will continue on. You are seeing that it
> will
> > > > > continue to add messages until the Queue's memory limit (40MB)
> > reaches
> > > > the
> > > > > 70% mark. Since MemoryUsages are hierarchical, this means it will
> > also
> > > > > account for messages in the overall broker memory as well. For each
> > > > queue,
> > > > > you'll see that it will continue to hold 70% of 40MB of memory.
> What
> > > you
> > > > > want in this case (if there are no consumers, or slow consumers) is
> > to
> > > > > raise your system usage memory limit OR lower your per-destination
> > > limits
> > > > > OR lower your cursor highwatermark or a combination of all three.
> > > > >
> > > > > http://activemq.apache.org/per-destination-policies.html
> > > > >
> > > > > With PFC turned off, you're essentially telling the broker to take
> > the
> > > > > message no matter what. There is a point at which you will run out
> of
> > > > > resources (memory, disk, etc). The trick is to find your use case
> and
> > > > tune
> > > > > for that.
> > > > >
> > > > >
> > > > > >
> > > > > > Will try to do some more testing soon...
> > > > > >
> > > > > > Thanks
> > > > > >
> > > > > >
> > > > > > On Wed, Nov 21, 2012 at 2:28 PM, Christian Posta
> > > > > > <ch...@gmail.com>wrote:
> > > > > >
> > > > > > > Can you please try on 5.7?
> > > > > > > I just tried a test, and if there are no consumers to the queue
> > > then
> > > > > the
> > > > > > > memory usage will stay at 0%. The message will not be retained,
> > ie,
> > > > it
> > > > > > will
> > > > > > > be put into the store and kept there. If I add a consumer, and
> > not
> > > > try
> > > > > to
> > > > > > > consume, the message will be kept around in memory up to the
> > cursor
> > > > > high
> > > > > > > watermark (70 by default).
> > > > > > >
> > > > > > > As I add more queues the same behavior as described above will
> > > > happen.
> > > > > > If I
> > > > > > > attach consumers to the queues without consuming them (so no
> > > messages
> > > > > are
> > > > > > > consumed), then messages are kept in the cursor up to the
> > > high-water
> > > > > > > mark... note.. the high-water mark is relative to the
> > > > > > Destination/Cursor's
> > > > > > > MemoryUsage, not the global memory usage.
> > > > > > >
> > > > > > > If I continue adding queues, and with producer flow control set
> > to
> > > > > > false, I
> > > > > > > too will see the *Global* memory usage go much higher than
> 100%.
> > > This
> > > > > is
> > > > > > > not surprising though, because as I understand, these usage
> > memory
> > > > > > objects
> > > > > > > are really just counters. They don't enforce anything. When
> > coupled
> > > > > with
> > > > > > > producer flow control, they can be used to determine when to
> > enable
> > > > > PFC.
> > > > > > If
> > > > > > > PFC is false, it's up to the cursor to determine when to flush
> > out
> > > to
> > > > > > disk.
> > > > > > > But each destination/cursor will have it's own system usage
> (with
> > > the
> > > > > > > global as the parent).
> > > > > > >
> > > > > > > Hope this helps. Can you please try with 5.7 and give us a
> report
> > > > back?
> > > > > > > Thanks,
> > > > > > > Christian
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Fri, Nov 16, 2012 at 11:38 AM, Juan Nin <ju...@gmail.com>
> > > wrote:
> > > > > > >
> > > > > > > > nope, adding a 3rd queue the 3rd one also gets this same
> value,
> > > so
> > > > > even
> > > > > > > if
> > > > > > > > it's the memory usage of the queue it's anyway going beyond..
> > > > > > > >
> > > > > > > >
> > > > > > > > On Fri, Nov 16, 2012 at 4:32 PM, Juan Nin <ju...@gmail.com>
> > > > wrote:
> > > > > > > >
> > > > > > > > > Might it be just a bug on how the MemoryPercentUsage is
> > > > calculated?
> > > > > > > > >
> > > > > > > > > If I connect via JMX using console, I can see the
> > > > > MemoryPercentUsage
> > > > > > as
> > > > > > > > > 112 right now.
> > > > > > > > > If I go to each of the 2 queues on them I see
> > CursorMemoryUsage
> > > > > with
> > > > > > > > value
> > > > > > > > > 29360604, which would be 28mb each, summing a total of 56mb
> > > > (just a
> > > > > > bit
> > > > > > > > > more than the specified memoryUsage of 50mb).
> > > > > > > > >
> > > > > > > > > Not sure I'm interpreting these values correctly though,
> > first
> > > > > time I
> > > > > > > > > access it via jconsole...
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Fri, Nov 16, 2012 at 4:07 PM, Juan Nin <
> juanin@gmail.com>
> > > > > wrote:
> > > > > > > > >
> > > > > > > > >> On that config there's a 40mb memoryLimit per queue, but
> > also
> > > > > tested
> > > > > > > it
> > > > > > > > >> without it with same results.
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >> On Fri, Nov 16, 2012 at 4:05 PM, Juan Nin <
> juanin@gmail.com
> > >
> > > > > wrote:
> > > > > > > > >>
> > > > > > > > >>> Hi Torsten!
> > > > > > > > >>>
> > > > > > > > >>> I'm using ActiveMQ 5.3.2, but also tested it on 5.7.0
> with
> > > the
> > > > > same
> > > > > > > > >>> results...
> > > > > > > > >>> This is my 5.3.2 config:
> > > > > > > > >>>
> > > > > > > > >>> <beans
> > > > > > > > >>> xmlns="http://www.springframework.org/schema/beans"
> > > > > > > > >>> xmlns:amq="http://activemq.apache.org/schema/core"
> > > > > > > > >>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
> > > > > > > > >>> xsi:schemaLocation="
> > > > > http://www.springframework.org/schema/beans
> > > > > > > > >>>
> > > > http://www.springframework.org/schema/beans/spring-beans-2.0.xsd
> > > > > > > > >>> http://activemq.apache.org/schema/core
> > > > > > > > >>> http://activemq.apache.org/schema/core/activemq-core.xsd
> ">
> > > > > > > > >>>
> > > > > > > > >>> <bean
> > > > > > > > >>>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
> > > > > > > > >>> <property name="locations">
> > > > > > > > >>>
> > > > > > > > >>>
> > > > <value>file:${activemq.base}/conf/credentials.properties</value>
> > > > > > > > >>> </property>
> > > > > > > > >>> </bean>
> > > > > > > > >>>
> > > > > > > > >>> <broker xmlns="
> http://activemq.apache.org/schema/core"
> > > > > > > > >>> brokerName="localhost"
> > dataDirectory="${activemq.base}/data"
> > > > > > > > >>> destroyApplicationContextOnStop="true"
> > > advisorySupport="false">
> > > > > > > > >>>
> > > > > > > > >>> <destinationPolicy>
> > > > > > > > >>> <policyMap>
> > > > > > > > >>> <policyEntries>
> > > > > > > > >>> <policyEntry topic=">"
> > > > > > producerFlowControl="true"
> > > > > > > > >>> memoryLimit="5mb">
> > > > > > > > >>> <pendingSubscriberPolicy>
> > > > > > > > >>> <vmCursor />
> > > > > > > > >>> </pendingSubscriberPolicy>
> > > > > > > > >>> </policyEntry>
> > > > > > > > >>> <policyEntry queue=">"
> > > > > > > producerFlowControl="false"
> > > > > > > > >>> optimizedDispatch="true" memoryLimit="40mb">
> > > > > > > > >>> <deadLetterStrategy>
> > > > > > > > >>> <individualDeadLetterStrategy
> > > > > > > > >>> queuePrefix="DLQ." useQueueForQueueMessages="true" />
> > > > > > > > >>> </deadLetterStrategy>
> > > > > > > > >>> </policyEntry>
> > > > > > > > >>> </policyEntries>
> > > > > > > > >>> </policyMap>
> > > > > > > > >>> </destinationPolicy>
> > > > > > > > >>>
> > > > > > > > >>> <managementContext>
> > > > > > > > >>> <managementContext connectorPort="2011"/>
> > > > > > > > >>> </managementContext>
> > > > > > > > >>>
> > > > > > > > >>> <persistenceAdapter>
> > > > > > > > >>> <kahaDB
> > directory="${activemq.base}/data/kahadb"
> > > > > > > > >>> enableJournalDiskSyncs="false"
> indexWriteBatchSize="10000"
> > > > > > > > >>> indexCacheSize="1000"/>
> > > > > > > > >>> </persistenceAdapter>
> > > > > > > > >>>
> > > > > > > > >>> <systemUsage>
> > > > > > > > >>> <systemUsage>
> > > > > > > > >>> <memoryUsage>
> > > > > > > > >>> <memoryUsage limit="50 mb"/>
> > > > > > > > >>> </memoryUsage>
> > > > > > > > >>> <storeUsage>
> > > > > > > > >>> <storeUsage limit="1 gb" name="foo"/>
> > > > > > > > >>> </storeUsage>
> > > > > > > > >>> <tempUsage>
> > > > > > > > >>> <tempUsage limit="3 gb"/>
> > > > > > > > >>> </tempUsage>
> > > > > > > > >>> </systemUsage>
> > > > > > > > >>> </systemUsage>
> > > > > > > > >>>
> > > > > > > > >>> <transportConnectors>
> > > > > > > > >>> <transportConnector name="openwire"
> uri="tcp://
> > > > > > > > 0.0.0.0:61616
> > > > > > > > >>> "/>
> > > > > > > > >>> <transportConnector name="stomp" uri="stomp://
> > > > 0.0.0.0:61613
> > > > > "/>
> > > > > > > > >>> </transportConnectors>
> > > > > > > > >>>
> > > > > > > > >>> </broker>
> > > > > > > > >>>
> > > > > > > > >>> <import resource="jetty.xml"/>
> > > > > > > > >>>
> > > > > > > > >>> </beans>
> > > > > > > > >>>
> > > > > > > > >>>
> > > > > > > > >>> Using just a simple PHP script with Stomp for feeding the
> > > > queues
> > > > > > > > >>> (running it twice with different queue name):
> > > > > > > > >>>
> > > > > > > > >>> <?php
> > > > > > > > >>>
> > > > > > > > >>> require_once("Stomp.php");
> > > > > > > > >>>
> > > > > > > > >>> $amq = new Stomp("tcp://localhost:61613");
> > > > > > > > >>> $amq->connect();
> > > > > > > > >>>
> > > > > > > > >>> for($i=1; $i <= 100000; $i++)
> > > > > > > > >>> {
> > > > > > > > >>> if($i%1000 == 0)
> > > > > > > > >>> {
> > > > > > > > >>> echo "\nmsg #: $i";
> > > > > > > > >>> }
> > > > > > > > >>> $amq->send("/queue/test", "this is test message # $i"
> > > > > > > > >>> ,array('persistent' => 'true'));
> > > > > > > > >>> }
> > > > > > > > >>>
> > > > > > > > >>> $amq->disconnect();
> > > > > > > > >>>
> > > > > > > > >>> ?>
> > > > > > > > >>>
> > > > > > > > >>>
> > > > > > > > >>>
> > > > > > > > >>> On Fri, Nov 16, 2012 at 3:47 PM, Torsten Mielke <
> > > > > > > > torsten@fusesource.com>wrote:
> > > > > > > > >>>
> > > > > > > > >>>> Hello,
> > > > > > > > >>>>
> > > > > > > > >>>> See in-line response.
> > > > > > > > >>>>
> > > > > > > > >>>> On Nov 16, 2012, at 6:29 PM, Juan Nin wrote:
> > > > > > > > >>>>
> > > > > > > > >>>> > Hi!
> > > > > > > > >>>> >
> > > > > > > > >>>> > After some heavy digging about Producer Flow control
> and
> > > the
> > > > > > > > >>>> systemUsage
> > > > > > > > >>>> > properties a couple of years ago, I thought I quite
> > > > understood
> > > > > > it.
> > > > > > > > >>>> > But yesterday I found that one of my configs was not
> > > > behaving
> > > > > > > > exactly
> > > > > > > > >>>> as I
> > > > > > > > >>>> > expected, so started doing some tests, and I see
> certain
> > > > > > > behaviours
> > > > > > > > >>>> which
> > > > > > > > >>>> > don't seem to match what the docs and posts that I
> find
> > on
> > > > the
> > > > > > > list
> > > > > > > > or
> > > > > > > > >>>> > other forums say.
> > > > > > > > >>>> >
> > > > > > > > >>>> > "storeUsage" is perfectly clear, it's the max space
> that
> > > > > > > persistent
> > > > > > > > >>>> > messages can use to be stored in disk.
> > > > > > > > >>>> > "tempUsage"" applies to file cursors on non-persistent
> > > > > messages,
> > > > > > > so
> > > > > > > > >>>> as to
> > > > > > > > >>>> > flush to disk if memory limits are reached (I don't
> care
> > > > much
> > > > > > > about
> > > > > > > > >>>> this
> > > > > > > > >>>> > one anyway, I always use persistent messages).
> > > > > > > > >>>>
> > > > > > > > >>>> Correct.
> > > > > > > > >>>>
> > > > > > > > >>>> >
> > > > > > > > >>>> > Now, according to most posts, memoryUsage would be the
> > > > maximum
> > > > > > > > memory
> > > > > > > > >>>> that
> > > > > > > > >>>> > the broker would be available to use.
> > > > > > > > >>>> > On this post:
> > > > > > > > >>>> >
> > > > > > > > >>>>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://stackoverflow.com/questions/7646057/activemq-destinationpolicy-and-systemusage-configurationit
> > > > > > > > >>>> > says that "memoryUsage corresponds to the amount of
> > memory
> > > > > > that's
> > > > > > > > >>>> > assigned to the in-memory store".
> > > > > > > > >>>>
> > > > > > > > >>>> Correct.
> > > > > > > > >>>>
> > > > > > > > >>>> >
> > > > > > > > >>>> > For example, on my tests using the following config
> > (only
> > > > > > showing
> > > > > > > > >>>> relevant
> > > > > > > > >>>> > parts):
> > > > > > > > >>>> >
> > > > > > > > >>>> > <policyEntry queue=">" producerFlowControl="false"
> > > > > > > > >>>> optimizedDispatch="true">
> > > > > > > > >>>> > <deadLetterStrategy>
> > > > > > > > >>>> > <individualDeadLetterStrategy
> queuePrefix="DLQ."
> > > > > > > > >>>> > useQueueForQueueMessages="true" />
> > > > > > > > >>>> > </deadLetterStrategy>
> > > > > > > > >>>> > </policyEntry>
> > > > > > > > >>>> >
> > > > > > > > >>>> > <systemUsage>
> > > > > > > > >>>> > <systemUsage>
> > > > > > > > >>>> > <memoryUsage>
> > > > > > > > >>>> > <memoryUsage limit="100 mb"/>
> > > > > > > > >>>> > </memoryUsage>
> > > > > > > > >>>> > <storeUsage>
> > > > > > > > >>>> > <storeUsage limit="1 gb" name="foo"/>
> > > > > > > > >>>> > </storeUsage>
> > > > > > > > >>>> > <tempUsage>
> > > > > > > > >>>> > <tempUsage limit="3 gb"/>
> > > > > > > > >>>> > </tempUsage>
> > > > > > > > >>>> > </systemUsage>
> > > > > > > > >>>> > </systemUsage>
> > > > > > > > >>>> >
> > > > > > > > >>>> > With that config I would expect the broker to use 100
> mb
> > > of
> > > > > > > maximum
> > > > > > > > >>>> memory
> > > > > > > > >>>> > among all queues. So it could maybe use 30mb in one
> > queue
> > > > and
> > > > > > 70mb
> > > > > > > > in
> > > > > > > > >>>> > second queue.
> > > > > > > > >>>> >
> > > > > > > > >>>> >
> > > > > > > > >>>> > 1) What I'm seeing is that if I start feeding a queue
> > > > without
> > > > > > > > >>>> consuming it,
> > > > > > > > >>>> > the "Memory percent used" grows up to 70%, after that
> it
> > > > > doesn't
> > > > > > > > grow
> > > > > > > > >>>> > anymore.
> > > > > > > > >>>> > What is it doing exactly there? The first 70% is
> stored
> > in
> > > > > > memory
> > > > > > > > >>>> (apart
> > > > > > > > >>>> > from disk since it's persistent), and all the rest
> that
> > > > > > continues
> > > > > > > > >>>> being fed
> > > > > > > > >>>> > goes just to disk?
> > > > > > > > >>>>
> > > > > > > > >>>> This behavior is correct. For queues the default cursor
> is
> > > > store
> > > > > > > > >>>> cursor. It keeps any newly arrived msgs in memory as
> long
> > as
> > > > it
> > > > > > does
> > > > > > > > not
> > > > > > > > >>>> reach the configured memory limit (either configured on
> > the
> > > > > queue
> > > > > > > per
> > > > > > > > >>>> destination or globally in memoryUsage settings).
> > > > > > > > >>>> Once the cursor reaches 70% of the configured limit (in
> > your
> > > > > case
> > > > > > of
> > > > > > > > >>>> the memoryUsage limit since you don't specify a
> > > > per-destination
> > > > > > > > limit), it
> > > > > > > > >>>> will not keep any more messages in memory.
> > > > > > > > >>>> Instead it will reload these messages from the store
> when
> > > its
> > > > > time
> > > > > > > to
> > > > > > > > >>>> dispatch them. The broker anyway persists any msgs it
> > > receives
> > > > > > > before
> > > > > > > > >>>> passing on to the cursor.
> > > > > > > > >>>> This limit of 70% can be configured and raised to e..g
> > 100%.
> > > > > > > > >>>> This behavior is kind of an optimization. That way you
> run
> > > > less
> > > > > > > often
> > > > > > > > >>>> into producer-flow-control.
> > > > > > > > >>>> As long as the persistence store is not running full,
> > there
> > > is
> > > > > no
> > > > > > > need
> > > > > > > > >>>> to block producers, since the cursor can also load the
> > > > messages
> > > > > > from
> > > > > > > > the
> > > > > > > > >>>> store and does not necessarily have to keep them in
> > memory.
> > > > > > > > >>>> If you configure the vmQueueCursor, then the behavior is
> > > > > > different.
> > > > > > > > >>>> This cursor will not be able to load msgs to the store
> but
> > > > needs
> > > > > > to
> > > > > > > > keep
> > > > > > > > >>>> them all in memory. The vmQueueCursor used to be the
> > default
> > > > > > cursor
> > > > > > > in
> > > > > > > > >>>> older version of AMQ.
> > > > > > > > >>>>
> > > > > > > > >>>> Also note that topic msgs and non-persistent queue
> > messages
> > > > are
> > > > > > not
> > > > > > > > >>>> handled by the store cursor. These msgs are held in
> memory
> > > and
> > > > > if
> > > > > > > > memory
> > > > > > > > >>>> runs low, get swapped out to temp storage.
> > > > > > > > >>>>
> > > > > > > > >>>> > 2) If then I start feeding a 2nd queue, "Memory
> percent
> > > > used"
> > > > > > > > >>>> continues
> > > > > > > > >>>> > growing until it reaches 140%. So it looks like
> > > memoryUsage
> > > > > does
> > > > > > > not
> > > > > > > > >>>> apply
> > > > > > > > >>>> > globally, but on a per queue basis?
> > > > > > > > >>>>
> > > > > > > > >>>> What version of AMQ do you use? The sum of the memory
> > usage
> > > of
> > > > > all
> > > > > > > > >>>> queues should not go any higher than the configured
> > > > memoryUsage
> > > > > > > > limit. If
> > > > > > > > >>>> you're not on 5.5.1 or higher releases, then I suggest
> to
> > > > > upgrade.
> > > > > > > > >>>>
> > > > > > > > >>>> > Using memoryLimit on the queue's policyEntry gives
> more
> > > > > control
> > > > > > > over
> > > > > > > > >>>> this,
> > > > > > > > >>>> > but it's just a variation, "Memory percent used" can
> > grow
> > > > more
> > > > > > > than
> > > > > > > > >>>> 100%
> > > > > > > > >>>> > anyway.
> > > > > > > > >>>>
> > > > > > > > >>>> With the default store cursor this should not be the
> case
> > > from
> > > > > > what
> > > > > > > I
> > > > > > > > >>>> know.
> > > > > > > > >>>>
> > > > > > > > >>>>
> > > > > > > > >>>> >
> > > > > > > > >>>> > 3) If #2 is true, then how would I prevent the broker
> > from
> > > > > > running
> > > > > > > > >>>> out of
> > > > > > > > >>>> > memory in case queues would continue to be created?
> > > > > > > > >>>>
> > > > > > > > >>>> Just like above comment. I would expect the brokers
> > > > > > > MemoryPercentUsage
> > > > > > > > >>>> won't grow over 100% and the destinations
> > MemoryPercentUsage
> > > > > > remains
> > > > > > > > fairly
> > > > > > > > >>>> much at 70%.
> > > > > > > > >>>> Not sure why you would see a different behavior? Using
> an
> > > old
> > > > > > > version
> > > > > > > > >>>> of AMQ perhaps? Or explicitly configuring for the
> > > > vmQueueCursor?
> > > > > > > > >>>> Could you perhaps also test with
> > > > > > > > >>>>
> > > > > > > > >>>> >
> > > > > > > > >>>> >
> > > > > > > > >>>> > Maybe I'm misunderstanding and some of these settings
> > make
> > > > no
> > > > > > > sense
> > > > > > > > >>>> when
> > > > > > > > >>>> > producerFlowControl is disabled?
> > > > > > > > >>>> >
> > > > > > > > >>>> > Thanks in advance.
> > > > > > > > >>>> >
> > > > > > > > >>>> > Juan
> > > > > > > > >>>>
> > > > > > > > >>>>
> > > > > > > > >>>> Regards,
> > > > > > > > >>>>
> > > > > > > > >>>> Torsten Mielke
> > > > > > > > >>>> torsten@fusesource.com
> > > > > > > > >>>> tmielke.blogspot.com
> > > > > > > > >>>>
> > > > > > > > >>>>
> > > > > > > > >>>>
> > > > > > > > >>>>
> > > > > > > > >>>>
> > > > > > > > >>>
> > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > *Christian Posta*
> > > > > > > http://www.christianposta.com/blog
> > > > > > > twitter: @christianposta
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > *Christian Posta*
> > > > > http://www.christianposta.com/blog
> > > > > twitter: @christianposta
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > *Christian Posta*
> > > http://www.christianposta.com/blog
> > > twitter: @christianposta
> > >
> >
>
>
>
> --
> http://redhat.com
> http://blog.garytully.com
>
Re: Understanding memoryUsage (once again!)
Posted by Gary Tully <ga...@gmail.com>.
use the destinationpolicy entry cursorMemoryHighWaterMark, it default to
70% of the system usage. so set it to something low like 2% and it will
spool to disk very early.
When reading back in from disk, it pages in messages in batches, this is
controlled by the maxPageSize policy enry for a destination. defaults to
200
On 29 November 2012 23:42, Juan Nin <ju...@gmail.com> wrote:
> Right now I set it to 5mb per queue, but I guess could be even less.
>
> I tested using "0 mb" to see if it completely flushed to disk without using
> memory, but didn't work, in that case seems it's the same as not putting
> the memoryLimit at all. Is there any way to make them just go to disk
> without using memory at all?
>
> Now, once they go to disk, how does ActiveMQ process them upon consumption?
> Does it load to memory in chunks or something like that and keeps them in
> memory, or just grabs from disk as required?
>
> I'll have cases where consumptions will be one by one, sporadic by
> different processes (intermittent).
> But there will be other cases where even millions will be consumed at once
> by a multithreaded Java app.
>
> So for the intermittent ones it would be ok if it just loads them from
> disk, but for the other ones would be interesting if can load bigger chunks
> into memory for faster consumption. Not sure if possible though.
>
> Thanks again.
>
>
> On Thu, Nov 29, 2012 at 6:49 PM, Christian Posta
> <ch...@gmail.com>wrote:
>
> > Another thought.. if you know they won't be immediately consumed (slow,
> or
> > even intermittent consumers), why keep them in memory at all? Or at least
> > why keep such a large number of them? Maybe turn down the memory limits
> on
> > the destinations with knowingly slow consumers so fewer messages are kept
> > in memory (they will be kept in store if they are persistent messages and
> > then recovered when they are ready to be dispatched). The only downside
> is
> > if the consumers speed up, you'll be fetching from disk more often.
> >
> > Would be nice if the broker could auto-tune its memory usage... uh oh...
> >
> >
> >
> >
> >
> >
> > On Thu, Nov 29, 2012 at 12:45 PM, Juan Nin <ju...@gmail.com> wrote:
> >
> > > Hi Christian!
> > >
> > > Yes, actually that's what I'm doing, just setting per destination
> > policies
> > > which work for me.
> > > I anyway needed them because I'm creating queues with lots of messages,
> > > which won't be immediately consumed, so having them store a lot into
> > memory
> > > ended up slowing things up.
> > >
> > > So I just assigned enough memory to the broker so as not run into
> issues.
> > >
> > > Thanks again.
> > >
> > >
> > > On Tue, Nov 27, 2012 at 9:40 PM, Christian Posta
> > > <ch...@gmail.com>wrote:
> > >
> > > > See inline...
> > > >
> > > >
> > > > On Wed, Nov 21, 2012 at 12:04 PM, Juan Nin <ju...@gmail.com> wrote:
> > > >
> > > > > Hi!
> > > > >
> > > > > Sorry for the delay in replying, buried on a project.
> > > > >
> > > > > As I mentioned before, I had tested this with 5.7.0 with the same
> > > > > behaviour.
> > > > > I just tested it again (both with 5.3.2 and 5.7.0) and same thing,
> > and
> > > on
> > > > > my case it doesn't matter if there are consumers or not, it always
> > > seems
> > > > to
> > > > > make usage of the memory.
> > > > >
> > > > > Although I guess in theory that should not affect, did you use
> Stomp
> > > for
> > > > > your testing, or maybe you used Openwire?
> > > > > I'm using Stomp for my testing.
> > > > >
> > > > > Might be though that the broker's memory itself is not going beyond
> > 70%
> > > > of
> > > > > memoryUsage, but this is just per destination counters as you
> > > mentioned.
> > > > > In which case I guess the value shown as "Memory percent used" is a
> > bit
> > > > > confusing... But haven't had much time to really test the
> possibility
> > > of
> > > > > exhausting the broker's memory.
> > > > >
> > > > No, i believe what you're seeing is correct. The broker's memory
> limit
> > is
> > > > going beyond memoryUsage (way beyond). When a queue checks whether
> > memory
> > > > is full, it will only do something interesting if producer flow
> control
> > > is
> > > > enabled. Otherwise, it will continue on. You are seeing that it will
> > > > continue to add messages until the Queue's memory limit (40MB)
> reaches
> > > the
> > > > 70% mark. Since MemoryUsages are hierarchical, this means it will
> also
> > > > account for messages in the overall broker memory as well. For each
> > > queue,
> > > > you'll see that it will continue to hold 70% of 40MB of memory. What
> > you
> > > > want in this case (if there are no consumers, or slow consumers) is
> to
> > > > raise your system usage memory limit OR lower your per-destination
> > limits
> > > > OR lower your cursor highwatermark or a combination of all three.
> > > >
> > > > http://activemq.apache.org/per-destination-policies.html
> > > >
> > > > With PFC turned off, you're essentially telling the broker to take
> the
> > > > message no matter what. There is a point at which you will run out of
> > > > resources (memory, disk, etc). The trick is to find your use case and
> > > tune
> > > > for that.
> > > >
> > > >
> > > > >
> > > > > Will try to do some more testing soon...
> > > > >
> > > > > Thanks
> > > > >
> > > > >
> > > > > On Wed, Nov 21, 2012 at 2:28 PM, Christian Posta
> > > > > <ch...@gmail.com>wrote:
> > > > >
> > > > > > Can you please try on 5.7?
> > > > > > I just tried a test, and if there are no consumers to the queue
> > then
> > > > the
> > > > > > memory usage will stay at 0%. The message will not be retained,
> ie,
> > > it
> > > > > will
> > > > > > be put into the store and kept there. If I add a consumer, and
> not
> > > try
> > > > to
> > > > > > consume, the message will be kept around in memory up to the
> cursor
> > > > high
> > > > > > watermark (70 by default).
> > > > > >
> > > > > > As I add more queues the same behavior as described above will
> > > happen.
> > > > > If I
> > > > > > attach consumers to the queues without consuming them (so no
> > messages
> > > > are
> > > > > > consumed), then messages are kept in the cursor up to the
> > high-water
> > > > > > mark... note.. the high-water mark is relative to the
> > > > > Destination/Cursor's
> > > > > > MemoryUsage, not the global memory usage.
> > > > > >
> > > > > > If I continue adding queues, and with producer flow control set
> to
> > > > > false, I
> > > > > > too will see the *Global* memory usage go much higher than 100%.
> > This
> > > > is
> > > > > > not surprising though, because as I understand, these usage
> memory
> > > > > objects
> > > > > > are really just counters. They don't enforce anything. When
> coupled
> > > > with
> > > > > > producer flow control, they can be used to determine when to
> enable
> > > > PFC.
> > > > > If
> > > > > > PFC is false, it's up to the cursor to determine when to flush
> out
> > to
> > > > > disk.
> > > > > > But each destination/cursor will have it's own system usage (with
> > the
> > > > > > global as the parent).
> > > > > >
> > > > > > Hope this helps. Can you please try with 5.7 and give us a report
> > > back?
> > > > > > Thanks,
> > > > > > Christian
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Fri, Nov 16, 2012 at 11:38 AM, Juan Nin <ju...@gmail.com>
> > wrote:
> > > > > >
> > > > > > > nope, adding a 3rd queue the 3rd one also gets this same value,
> > so
> > > > even
> > > > > > if
> > > > > > > it's the memory usage of the queue it's anyway going beyond..
> > > > > > >
> > > > > > >
> > > > > > > On Fri, Nov 16, 2012 at 4:32 PM, Juan Nin <ju...@gmail.com>
> > > wrote:
> > > > > > >
> > > > > > > > Might it be just a bug on how the MemoryPercentUsage is
> > > calculated?
> > > > > > > >
> > > > > > > > If I connect via JMX using console, I can see the
> > > > MemoryPercentUsage
> > > > > as
> > > > > > > > 112 right now.
> > > > > > > > If I go to each of the 2 queues on them I see
> CursorMemoryUsage
> > > > with
> > > > > > > value
> > > > > > > > 29360604, which would be 28mb each, summing a total of 56mb
> > > (just a
> > > > > bit
> > > > > > > > more than the specified memoryUsage of 50mb).
> > > > > > > >
> > > > > > > > Not sure I'm interpreting these values correctly though,
> first
> > > > time I
> > > > > > > > access it via jconsole...
> > > > > > > >
> > > > > > > >
> > > > > > > > On Fri, Nov 16, 2012 at 4:07 PM, Juan Nin <ju...@gmail.com>
> > > > wrote:
> > > > > > > >
> > > > > > > >> On that config there's a 40mb memoryLimit per queue, but
> also
> > > > tested
> > > > > > it
> > > > > > > >> without it with same results.
> > > > > > > >>
> > > > > > > >>
> > > > > > > >> On Fri, Nov 16, 2012 at 4:05 PM, Juan Nin <juanin@gmail.com
> >
> > > > wrote:
> > > > > > > >>
> > > > > > > >>> Hi Torsten!
> > > > > > > >>>
> > > > > > > >>> I'm using ActiveMQ 5.3.2, but also tested it on 5.7.0 with
> > the
> > > > same
> > > > > > > >>> results...
> > > > > > > >>> This is my 5.3.2 config:
> > > > > > > >>>
> > > > > > > >>> <beans
> > > > > > > >>> xmlns="http://www.springframework.org/schema/beans"
> > > > > > > >>> xmlns:amq="http://activemq.apache.org/schema/core"
> > > > > > > >>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
> > > > > > > >>> xsi:schemaLocation="
> > > > http://www.springframework.org/schema/beans
> > > > > > > >>>
> > > http://www.springframework.org/schema/beans/spring-beans-2.0.xsd
> > > > > > > >>> http://activemq.apache.org/schema/core
> > > > > > > >>> http://activemq.apache.org/schema/core/activemq-core.xsd">
> > > > > > > >>>
> > > > > > > >>> <bean
> > > > > > > >>>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
> > > > > > > >>> <property name="locations">
> > > > > > > >>>
> > > > > > > >>>
> > > <value>file:${activemq.base}/conf/credentials.properties</value>
> > > > > > > >>> </property>
> > > > > > > >>> </bean>
> > > > > > > >>>
> > > > > > > >>> <broker xmlns="http://activemq.apache.org/schema/core"
> > > > > > > >>> brokerName="localhost"
> dataDirectory="${activemq.base}/data"
> > > > > > > >>> destroyApplicationContextOnStop="true"
> > advisorySupport="false">
> > > > > > > >>>
> > > > > > > >>> <destinationPolicy>
> > > > > > > >>> <policyMap>
> > > > > > > >>> <policyEntries>
> > > > > > > >>> <policyEntry topic=">"
> > > > > producerFlowControl="true"
> > > > > > > >>> memoryLimit="5mb">
> > > > > > > >>> <pendingSubscriberPolicy>
> > > > > > > >>> <vmCursor />
> > > > > > > >>> </pendingSubscriberPolicy>
> > > > > > > >>> </policyEntry>
> > > > > > > >>> <policyEntry queue=">"
> > > > > > producerFlowControl="false"
> > > > > > > >>> optimizedDispatch="true" memoryLimit="40mb">
> > > > > > > >>> <deadLetterStrategy>
> > > > > > > >>> <individualDeadLetterStrategy
> > > > > > > >>> queuePrefix="DLQ." useQueueForQueueMessages="true" />
> > > > > > > >>> </deadLetterStrategy>
> > > > > > > >>> </policyEntry>
> > > > > > > >>> </policyEntries>
> > > > > > > >>> </policyMap>
> > > > > > > >>> </destinationPolicy>
> > > > > > > >>>
> > > > > > > >>> <managementContext>
> > > > > > > >>> <managementContext connectorPort="2011"/>
> > > > > > > >>> </managementContext>
> > > > > > > >>>
> > > > > > > >>> <persistenceAdapter>
> > > > > > > >>> <kahaDB
> directory="${activemq.base}/data/kahadb"
> > > > > > > >>> enableJournalDiskSyncs="false" indexWriteBatchSize="10000"
> > > > > > > >>> indexCacheSize="1000"/>
> > > > > > > >>> </persistenceAdapter>
> > > > > > > >>>
> > > > > > > >>> <systemUsage>
> > > > > > > >>> <systemUsage>
> > > > > > > >>> <memoryUsage>
> > > > > > > >>> <memoryUsage limit="50 mb"/>
> > > > > > > >>> </memoryUsage>
> > > > > > > >>> <storeUsage>
> > > > > > > >>> <storeUsage limit="1 gb" name="foo"/>
> > > > > > > >>> </storeUsage>
> > > > > > > >>> <tempUsage>
> > > > > > > >>> <tempUsage limit="3 gb"/>
> > > > > > > >>> </tempUsage>
> > > > > > > >>> </systemUsage>
> > > > > > > >>> </systemUsage>
> > > > > > > >>>
> > > > > > > >>> <transportConnectors>
> > > > > > > >>> <transportConnector name="openwire" uri="tcp://
> > > > > > > 0.0.0.0:61616
> > > > > > > >>> "/>
> > > > > > > >>> <transportConnector name="stomp" uri="stomp://
> > > 0.0.0.0:61613
> > > > "/>
> > > > > > > >>> </transportConnectors>
> > > > > > > >>>
> > > > > > > >>> </broker>
> > > > > > > >>>
> > > > > > > >>> <import resource="jetty.xml"/>
> > > > > > > >>>
> > > > > > > >>> </beans>
> > > > > > > >>>
> > > > > > > >>>
> > > > > > > >>> Using just a simple PHP script with Stomp for feeding the
> > > queues
> > > > > > > >>> (running it twice with different queue name):
> > > > > > > >>>
> > > > > > > >>> <?php
> > > > > > > >>>
> > > > > > > >>> require_once("Stomp.php");
> > > > > > > >>>
> > > > > > > >>> $amq = new Stomp("tcp://localhost:61613");
> > > > > > > >>> $amq->connect();
> > > > > > > >>>
> > > > > > > >>> for($i=1; $i <= 100000; $i++)
> > > > > > > >>> {
> > > > > > > >>> if($i%1000 == 0)
> > > > > > > >>> {
> > > > > > > >>> echo "\nmsg #: $i";
> > > > > > > >>> }
> > > > > > > >>> $amq->send("/queue/test", "this is test message # $i"
> > > > > > > >>> ,array('persistent' => 'true'));
> > > > > > > >>> }
> > > > > > > >>>
> > > > > > > >>> $amq->disconnect();
> > > > > > > >>>
> > > > > > > >>> ?>
> > > > > > > >>>
> > > > > > > >>>
> > > > > > > >>>
> > > > > > > >>> On Fri, Nov 16, 2012 at 3:47 PM, Torsten Mielke <
> > > > > > > torsten@fusesource.com>wrote:
> > > > > > > >>>
> > > > > > > >>>> Hello,
> > > > > > > >>>>
> > > > > > > >>>> See in-line response.
> > > > > > > >>>>
> > > > > > > >>>> On Nov 16, 2012, at 6:29 PM, Juan Nin wrote:
> > > > > > > >>>>
> > > > > > > >>>> > Hi!
> > > > > > > >>>> >
> > > > > > > >>>> > After some heavy digging about Producer Flow control and
> > the
> > > > > > > >>>> systemUsage
> > > > > > > >>>> > properties a couple of years ago, I thought I quite
> > > understood
> > > > > it.
> > > > > > > >>>> > But yesterday I found that one of my configs was not
> > > behaving
> > > > > > > exactly
> > > > > > > >>>> as I
> > > > > > > >>>> > expected, so started doing some tests, and I see certain
> > > > > > behaviours
> > > > > > > >>>> which
> > > > > > > >>>> > don't seem to match what the docs and posts that I find
> on
> > > the
> > > > > > list
> > > > > > > or
> > > > > > > >>>> > other forums say.
> > > > > > > >>>> >
> > > > > > > >>>> > "storeUsage" is perfectly clear, it's the max space that
> > > > > > persistent
> > > > > > > >>>> > messages can use to be stored in disk.
> > > > > > > >>>> > "tempUsage"" applies to file cursors on non-persistent
> > > > messages,
> > > > > > so
> > > > > > > >>>> as to
> > > > > > > >>>> > flush to disk if memory limits are reached (I don't care
> > > much
> > > > > > about
> > > > > > > >>>> this
> > > > > > > >>>> > one anyway, I always use persistent messages).
> > > > > > > >>>>
> > > > > > > >>>> Correct.
> > > > > > > >>>>
> > > > > > > >>>> >
> > > > > > > >>>> > Now, according to most posts, memoryUsage would be the
> > > maximum
> > > > > > > memory
> > > > > > > >>>> that
> > > > > > > >>>> > the broker would be available to use.
> > > > > > > >>>> > On this post:
> > > > > > > >>>> >
> > > > > > > >>>>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://stackoverflow.com/questions/7646057/activemq-destinationpolicy-and-systemusage-configurationit
> > > > > > > >>>> > says that "memoryUsage corresponds to the amount of
> memory
> > > > > that's
> > > > > > > >>>> > assigned to the in-memory store".
> > > > > > > >>>>
> > > > > > > >>>> Correct.
> > > > > > > >>>>
> > > > > > > >>>> >
> > > > > > > >>>> > For example, on my tests using the following config
> (only
> > > > > showing
> > > > > > > >>>> relevant
> > > > > > > >>>> > parts):
> > > > > > > >>>> >
> > > > > > > >>>> > <policyEntry queue=">" producerFlowControl="false"
> > > > > > > >>>> optimizedDispatch="true">
> > > > > > > >>>> > <deadLetterStrategy>
> > > > > > > >>>> > <individualDeadLetterStrategy queuePrefix="DLQ."
> > > > > > > >>>> > useQueueForQueueMessages="true" />
> > > > > > > >>>> > </deadLetterStrategy>
> > > > > > > >>>> > </policyEntry>
> > > > > > > >>>> >
> > > > > > > >>>> > <systemUsage>
> > > > > > > >>>> > <systemUsage>
> > > > > > > >>>> > <memoryUsage>
> > > > > > > >>>> > <memoryUsage limit="100 mb"/>
> > > > > > > >>>> > </memoryUsage>
> > > > > > > >>>> > <storeUsage>
> > > > > > > >>>> > <storeUsage limit="1 gb" name="foo"/>
> > > > > > > >>>> > </storeUsage>
> > > > > > > >>>> > <tempUsage>
> > > > > > > >>>> > <tempUsage limit="3 gb"/>
> > > > > > > >>>> > </tempUsage>
> > > > > > > >>>> > </systemUsage>
> > > > > > > >>>> > </systemUsage>
> > > > > > > >>>> >
> > > > > > > >>>> > With that config I would expect the broker to use 100 mb
> > of
> > > > > > maximum
> > > > > > > >>>> memory
> > > > > > > >>>> > among all queues. So it could maybe use 30mb in one
> queue
> > > and
> > > > > 70mb
> > > > > > > in
> > > > > > > >>>> > second queue.
> > > > > > > >>>> >
> > > > > > > >>>> >
> > > > > > > >>>> > 1) What I'm seeing is that if I start feeding a queue
> > > without
> > > > > > > >>>> consuming it,
> > > > > > > >>>> > the "Memory percent used" grows up to 70%, after that it
> > > > doesn't
> > > > > > > grow
> > > > > > > >>>> > anymore.
> > > > > > > >>>> > What is it doing exactly there? The first 70% is stored
> in
> > > > > memory
> > > > > > > >>>> (apart
> > > > > > > >>>> > from disk since it's persistent), and all the rest that
> > > > > continues
> > > > > > > >>>> being fed
> > > > > > > >>>> > goes just to disk?
> > > > > > > >>>>
> > > > > > > >>>> This behavior is correct. For queues the default cursor is
> > > store
> > > > > > > >>>> cursor. It keeps any newly arrived msgs in memory as long
> as
> > > it
> > > > > does
> > > > > > > not
> > > > > > > >>>> reach the configured memory limit (either configured on
> the
> > > > queue
> > > > > > per
> > > > > > > >>>> destination or globally in memoryUsage settings).
> > > > > > > >>>> Once the cursor reaches 70% of the configured limit (in
> your
> > > > case
> > > > > of
> > > > > > > >>>> the memoryUsage limit since you don't specify a
> > > per-destination
> > > > > > > limit), it
> > > > > > > >>>> will not keep any more messages in memory.
> > > > > > > >>>> Instead it will reload these messages from the store when
> > its
> > > > time
> > > > > > to
> > > > > > > >>>> dispatch them. The broker anyway persists any msgs it
> > receives
> > > > > > before
> > > > > > > >>>> passing on to the cursor.
> > > > > > > >>>> This limit of 70% can be configured and raised to e..g
> 100%.
> > > > > > > >>>> This behavior is kind of an optimization. That way you run
> > > less
> > > > > > often
> > > > > > > >>>> into producer-flow-control.
> > > > > > > >>>> As long as the persistence store is not running full,
> there
> > is
> > > > no
> > > > > > need
> > > > > > > >>>> to block producers, since the cursor can also load the
> > > messages
> > > > > from
> > > > > > > the
> > > > > > > >>>> store and does not necessarily have to keep them in
> memory.
> > > > > > > >>>> If you configure the vmQueueCursor, then the behavior is
> > > > > different.
> > > > > > > >>>> This cursor will not be able to load msgs to the store but
> > > needs
> > > > > to
> > > > > > > keep
> > > > > > > >>>> them all in memory. The vmQueueCursor used to be the
> default
> > > > > cursor
> > > > > > in
> > > > > > > >>>> older version of AMQ.
> > > > > > > >>>>
> > > > > > > >>>> Also note that topic msgs and non-persistent queue
> messages
> > > are
> > > > > not
> > > > > > > >>>> handled by the store cursor. These msgs are held in memory
> > and
> > > > if
> > > > > > > memory
> > > > > > > >>>> runs low, get swapped out to temp storage.
> > > > > > > >>>>
> > > > > > > >>>> > 2) If then I start feeding a 2nd queue, "Memory percent
> > > used"
> > > > > > > >>>> continues
> > > > > > > >>>> > growing until it reaches 140%. So it looks like
> > memoryUsage
> > > > does
> > > > > > not
> > > > > > > >>>> apply
> > > > > > > >>>> > globally, but on a per queue basis?
> > > > > > > >>>>
> > > > > > > >>>> What version of AMQ do you use? The sum of the memory
> usage
> > of
> > > > all
> > > > > > > >>>> queues should not go any higher than the configured
> > > memoryUsage
> > > > > > > limit. If
> > > > > > > >>>> you're not on 5.5.1 or higher releases, then I suggest to
> > > > upgrade.
> > > > > > > >>>>
> > > > > > > >>>> > Using memoryLimit on the queue's policyEntry gives more
> > > > control
> > > > > > over
> > > > > > > >>>> this,
> > > > > > > >>>> > but it's just a variation, "Memory percent used" can
> grow
> > > more
> > > > > > than
> > > > > > > >>>> 100%
> > > > > > > >>>> > anyway.
> > > > > > > >>>>
> > > > > > > >>>> With the default store cursor this should not be the case
> > from
> > > > > what
> > > > > > I
> > > > > > > >>>> know.
> > > > > > > >>>>
> > > > > > > >>>>
> > > > > > > >>>> >
> > > > > > > >>>> > 3) If #2 is true, then how would I prevent the broker
> from
> > > > > running
> > > > > > > >>>> out of
> > > > > > > >>>> > memory in case queues would continue to be created?
> > > > > > > >>>>
> > > > > > > >>>> Just like above comment. I would expect the brokers
> > > > > > MemoryPercentUsage
> > > > > > > >>>> won't grow over 100% and the destinations
> MemoryPercentUsage
> > > > > remains
> > > > > > > fairly
> > > > > > > >>>> much at 70%.
> > > > > > > >>>> Not sure why you would see a different behavior? Using an
> > old
> > > > > > version
> > > > > > > >>>> of AMQ perhaps? Or explicitly configuring for the
> > > vmQueueCursor?
> > > > > > > >>>> Could you perhaps also test with
> > > > > > > >>>>
> > > > > > > >>>> >
> > > > > > > >>>> >
> > > > > > > >>>> > Maybe I'm misunderstanding and some of these settings
> make
> > > no
> > > > > > sense
> > > > > > > >>>> when
> > > > > > > >>>> > producerFlowControl is disabled?
> > > > > > > >>>> >
> > > > > > > >>>> > Thanks in advance.
> > > > > > > >>>> >
> > > > > > > >>>> > Juan
> > > > > > > >>>>
> > > > > > > >>>>
> > > > > > > >>>> Regards,
> > > > > > > >>>>
> > > > > > > >>>> Torsten Mielke
> > > > > > > >>>> torsten@fusesource.com
> > > > > > > >>>> tmielke.blogspot.com
> > > > > > > >>>>
> > > > > > > >>>>
> > > > > > > >>>>
> > > > > > > >>>>
> > > > > > > >>>>
> > > > > > > >>>
> > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > *Christian Posta*
> > > > > > http://www.christianposta.com/blog
> > > > > > twitter: @christianposta
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > *Christian Posta*
> > > > http://www.christianposta.com/blog
> > > > twitter: @christianposta
> > > >
> > >
> >
> >
> >
> > --
> > *Christian Posta*
> > http://www.christianposta.com/blog
> > twitter: @christianposta
> >
>
--
http://redhat.com
http://blog.garytully.com
Re: Understanding memoryUsage (once again!)
Posted by Juan Nin <ju...@gmail.com>.
Right now I set it to 5mb per queue, but I guess could be even less.
I tested using "0 mb" to see if it completely flushed to disk without using
memory, but didn't work, in that case seems it's the same as not putting
the memoryLimit at all. Is there any way to make them just go to disk
without using memory at all?
Now, once they go to disk, how does ActiveMQ process them upon consumption?
Does it load to memory in chunks or something like that and keeps them in
memory, or just grabs from disk as required?
I'll have cases where consumptions will be one by one, sporadic by
different processes (intermittent).
But there will be other cases where even millions will be consumed at once
by a multithreaded Java app.
So for the intermittent ones it would be ok if it just loads them from
disk, but for the other ones would be interesting if can load bigger chunks
into memory for faster consumption. Not sure if possible though.
Thanks again.
On Thu, Nov 29, 2012 at 6:49 PM, Christian Posta
<ch...@gmail.com>wrote:
> Another thought.. if you know they won't be immediately consumed (slow, or
> even intermittent consumers), why keep them in memory at all? Or at least
> why keep such a large number of them? Maybe turn down the memory limits on
> the destinations with knowingly slow consumers so fewer messages are kept
> in memory (they will be kept in store if they are persistent messages and
> then recovered when they are ready to be dispatched). The only downside is
> if the consumers speed up, you'll be fetching from disk more often.
>
> Would be nice if the broker could auto-tune its memory usage... uh oh...
>
>
>
>
>
>
> On Thu, Nov 29, 2012 at 12:45 PM, Juan Nin <ju...@gmail.com> wrote:
>
> > Hi Christian!
> >
> > Yes, actually that's what I'm doing, just setting per destination
> policies
> > which work for me.
> > I anyway needed them because I'm creating queues with lots of messages,
> > which won't be immediately consumed, so having them store a lot into
> memory
> > ended up slowing things up.
> >
> > So I just assigned enough memory to the broker so as not run into issues.
> >
> > Thanks again.
> >
> >
> > On Tue, Nov 27, 2012 at 9:40 PM, Christian Posta
> > <ch...@gmail.com>wrote:
> >
> > > See inline...
> > >
> > >
> > > On Wed, Nov 21, 2012 at 12:04 PM, Juan Nin <ju...@gmail.com> wrote:
> > >
> > > > Hi!
> > > >
> > > > Sorry for the delay in replying, buried on a project.
> > > >
> > > > As I mentioned before, I had tested this with 5.7.0 with the same
> > > > behaviour.
> > > > I just tested it again (both with 5.3.2 and 5.7.0) and same thing,
> and
> > on
> > > > my case it doesn't matter if there are consumers or not, it always
> > seems
> > > to
> > > > make usage of the memory.
> > > >
> > > > Although I guess in theory that should not affect, did you use Stomp
> > for
> > > > your testing, or maybe you used Openwire?
> > > > I'm using Stomp for my testing.
> > > >
> > > > Might be though that the broker's memory itself is not going beyond
> 70%
> > > of
> > > > memoryUsage, but this is just per destination counters as you
> > mentioned.
> > > > In which case I guess the value shown as "Memory percent used" is a
> bit
> > > > confusing... But haven't had much time to really test the possibility
> > of
> > > > exhausting the broker's memory.
> > > >
> > > No, i believe what you're seeing is correct. The broker's memory limit
> is
> > > going beyond memoryUsage (way beyond). When a queue checks whether
> memory
> > > is full, it will only do something interesting if producer flow control
> > is
> > > enabled. Otherwise, it will continue on. You are seeing that it will
> > > continue to add messages until the Queue's memory limit (40MB) reaches
> > the
> > > 70% mark. Since MemoryUsages are hierarchical, this means it will also
> > > account for messages in the overall broker memory as well. For each
> > queue,
> > > you'll see that it will continue to hold 70% of 40MB of memory. What
> you
> > > want in this case (if there are no consumers, or slow consumers) is to
> > > raise your system usage memory limit OR lower your per-destination
> limits
> > > OR lower your cursor highwatermark or a combination of all three.
> > >
> > > http://activemq.apache.org/per-destination-policies.html
> > >
> > > With PFC turned off, you're essentially telling the broker to take the
> > > message no matter what. There is a point at which you will run out of
> > > resources (memory, disk, etc). The trick is to find your use case and
> > tune
> > > for that.
> > >
> > >
> > > >
> > > > Will try to do some more testing soon...
> > > >
> > > > Thanks
> > > >
> > > >
> > > > On Wed, Nov 21, 2012 at 2:28 PM, Christian Posta
> > > > <ch...@gmail.com>wrote:
> > > >
> > > > > Can you please try on 5.7?
> > > > > I just tried a test, and if there are no consumers to the queue
> then
> > > the
> > > > > memory usage will stay at 0%. The message will not be retained, ie,
> > it
> > > > will
> > > > > be put into the store and kept there. If I add a consumer, and not
> > try
> > > to
> > > > > consume, the message will be kept around in memory up to the cursor
> > > high
> > > > > watermark (70 by default).
> > > > >
> > > > > As I add more queues the same behavior as described above will
> > happen.
> > > > If I
> > > > > attach consumers to the queues without consuming them (so no
> messages
> > > are
> > > > > consumed), then messages are kept in the cursor up to the
> high-water
> > > > > mark... note.. the high-water mark is relative to the
> > > > Destination/Cursor's
> > > > > MemoryUsage, not the global memory usage.
> > > > >
> > > > > If I continue adding queues, and with producer flow control set to
> > > > false, I
> > > > > too will see the *Global* memory usage go much higher than 100%.
> This
> > > is
> > > > > not surprising though, because as I understand, these usage memory
> > > > objects
> > > > > are really just counters. They don't enforce anything. When coupled
> > > with
> > > > > producer flow control, they can be used to determine when to enable
> > > PFC.
> > > > If
> > > > > PFC is false, it's up to the cursor to determine when to flush out
> to
> > > > disk.
> > > > > But each destination/cursor will have it's own system usage (with
> the
> > > > > global as the parent).
> > > > >
> > > > > Hope this helps. Can you please try with 5.7 and give us a report
> > back?
> > > > > Thanks,
> > > > > Christian
> > > > >
> > > > >
> > > > >
> > > > > On Fri, Nov 16, 2012 at 11:38 AM, Juan Nin <ju...@gmail.com>
> wrote:
> > > > >
> > > > > > nope, adding a 3rd queue the 3rd one also gets this same value,
> so
> > > even
> > > > > if
> > > > > > it's the memory usage of the queue it's anyway going beyond..
> > > > > >
> > > > > >
> > > > > > On Fri, Nov 16, 2012 at 4:32 PM, Juan Nin <ju...@gmail.com>
> > wrote:
> > > > > >
> > > > > > > Might it be just a bug on how the MemoryPercentUsage is
> > calculated?
> > > > > > >
> > > > > > > If I connect via JMX using console, I can see the
> > > MemoryPercentUsage
> > > > as
> > > > > > > 112 right now.
> > > > > > > If I go to each of the 2 queues on them I see CursorMemoryUsage
> > > with
> > > > > > value
> > > > > > > 29360604, which would be 28mb each, summing a total of 56mb
> > (just a
> > > > bit
> > > > > > > more than the specified memoryUsage of 50mb).
> > > > > > >
> > > > > > > Not sure I'm interpreting these values correctly though, first
> > > time I
> > > > > > > access it via jconsole...
> > > > > > >
> > > > > > >
> > > > > > > On Fri, Nov 16, 2012 at 4:07 PM, Juan Nin <ju...@gmail.com>
> > > wrote:
> > > > > > >
> > > > > > >> On that config there's a 40mb memoryLimit per queue, but also
> > > tested
> > > > > it
> > > > > > >> without it with same results.
> > > > > > >>
> > > > > > >>
> > > > > > >> On Fri, Nov 16, 2012 at 4:05 PM, Juan Nin <ju...@gmail.com>
> > > wrote:
> > > > > > >>
> > > > > > >>> Hi Torsten!
> > > > > > >>>
> > > > > > >>> I'm using ActiveMQ 5.3.2, but also tested it on 5.7.0 with
> the
> > > same
> > > > > > >>> results...
> > > > > > >>> This is my 5.3.2 config:
> > > > > > >>>
> > > > > > >>> <beans
> > > > > > >>> xmlns="http://www.springframework.org/schema/beans"
> > > > > > >>> xmlns:amq="http://activemq.apache.org/schema/core"
> > > > > > >>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
> > > > > > >>> xsi:schemaLocation="
> > > http://www.springframework.org/schema/beans
> > > > > > >>>
> > http://www.springframework.org/schema/beans/spring-beans-2.0.xsd
> > > > > > >>> http://activemq.apache.org/schema/core
> > > > > > >>> http://activemq.apache.org/schema/core/activemq-core.xsd">
> > > > > > >>>
> > > > > > >>> <bean
> > > > > > >>>
> > > > > >
> > > > >
> > > >
> > >
> >
> class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
> > > > > > >>> <property name="locations">
> > > > > > >>>
> > > > > > >>>
> > <value>file:${activemq.base}/conf/credentials.properties</value>
> > > > > > >>> </property>
> > > > > > >>> </bean>
> > > > > > >>>
> > > > > > >>> <broker xmlns="http://activemq.apache.org/schema/core"
> > > > > > >>> brokerName="localhost" dataDirectory="${activemq.base}/data"
> > > > > > >>> destroyApplicationContextOnStop="true"
> advisorySupport="false">
> > > > > > >>>
> > > > > > >>> <destinationPolicy>
> > > > > > >>> <policyMap>
> > > > > > >>> <policyEntries>
> > > > > > >>> <policyEntry topic=">"
> > > > producerFlowControl="true"
> > > > > > >>> memoryLimit="5mb">
> > > > > > >>> <pendingSubscriberPolicy>
> > > > > > >>> <vmCursor />
> > > > > > >>> </pendingSubscriberPolicy>
> > > > > > >>> </policyEntry>
> > > > > > >>> <policyEntry queue=">"
> > > > > producerFlowControl="false"
> > > > > > >>> optimizedDispatch="true" memoryLimit="40mb">
> > > > > > >>> <deadLetterStrategy>
> > > > > > >>> <individualDeadLetterStrategy
> > > > > > >>> queuePrefix="DLQ." useQueueForQueueMessages="true" />
> > > > > > >>> </deadLetterStrategy>
> > > > > > >>> </policyEntry>
> > > > > > >>> </policyEntries>
> > > > > > >>> </policyMap>
> > > > > > >>> </destinationPolicy>
> > > > > > >>>
> > > > > > >>> <managementContext>
> > > > > > >>> <managementContext connectorPort="2011"/>
> > > > > > >>> </managementContext>
> > > > > > >>>
> > > > > > >>> <persistenceAdapter>
> > > > > > >>> <kahaDB directory="${activemq.base}/data/kahadb"
> > > > > > >>> enableJournalDiskSyncs="false" indexWriteBatchSize="10000"
> > > > > > >>> indexCacheSize="1000"/>
> > > > > > >>> </persistenceAdapter>
> > > > > > >>>
> > > > > > >>> <systemUsage>
> > > > > > >>> <systemUsage>
> > > > > > >>> <memoryUsage>
> > > > > > >>> <memoryUsage limit="50 mb"/>
> > > > > > >>> </memoryUsage>
> > > > > > >>> <storeUsage>
> > > > > > >>> <storeUsage limit="1 gb" name="foo"/>
> > > > > > >>> </storeUsage>
> > > > > > >>> <tempUsage>
> > > > > > >>> <tempUsage limit="3 gb"/>
> > > > > > >>> </tempUsage>
> > > > > > >>> </systemUsage>
> > > > > > >>> </systemUsage>
> > > > > > >>>
> > > > > > >>> <transportConnectors>
> > > > > > >>> <transportConnector name="openwire" uri="tcp://
> > > > > > 0.0.0.0:61616
> > > > > > >>> "/>
> > > > > > >>> <transportConnector name="stomp" uri="stomp://
> > 0.0.0.0:61613
> > > "/>
> > > > > > >>> </transportConnectors>
> > > > > > >>>
> > > > > > >>> </broker>
> > > > > > >>>
> > > > > > >>> <import resource="jetty.xml"/>
> > > > > > >>>
> > > > > > >>> </beans>
> > > > > > >>>
> > > > > > >>>
> > > > > > >>> Using just a simple PHP script with Stomp for feeding the
> > queues
> > > > > > >>> (running it twice with different queue name):
> > > > > > >>>
> > > > > > >>> <?php
> > > > > > >>>
> > > > > > >>> require_once("Stomp.php");
> > > > > > >>>
> > > > > > >>> $amq = new Stomp("tcp://localhost:61613");
> > > > > > >>> $amq->connect();
> > > > > > >>>
> > > > > > >>> for($i=1; $i <= 100000; $i++)
> > > > > > >>> {
> > > > > > >>> if($i%1000 == 0)
> > > > > > >>> {
> > > > > > >>> echo "\nmsg #: $i";
> > > > > > >>> }
> > > > > > >>> $amq->send("/queue/test", "this is test message # $i"
> > > > > > >>> ,array('persistent' => 'true'));
> > > > > > >>> }
> > > > > > >>>
> > > > > > >>> $amq->disconnect();
> > > > > > >>>
> > > > > > >>> ?>
> > > > > > >>>
> > > > > > >>>
> > > > > > >>>
> > > > > > >>> On Fri, Nov 16, 2012 at 3:47 PM, Torsten Mielke <
> > > > > > torsten@fusesource.com>wrote:
> > > > > > >>>
> > > > > > >>>> Hello,
> > > > > > >>>>
> > > > > > >>>> See in-line response.
> > > > > > >>>>
> > > > > > >>>> On Nov 16, 2012, at 6:29 PM, Juan Nin wrote:
> > > > > > >>>>
> > > > > > >>>> > Hi!
> > > > > > >>>> >
> > > > > > >>>> > After some heavy digging about Producer Flow control and
> the
> > > > > > >>>> systemUsage
> > > > > > >>>> > properties a couple of years ago, I thought I quite
> > understood
> > > > it.
> > > > > > >>>> > But yesterday I found that one of my configs was not
> > behaving
> > > > > > exactly
> > > > > > >>>> as I
> > > > > > >>>> > expected, so started doing some tests, and I see certain
> > > > > behaviours
> > > > > > >>>> which
> > > > > > >>>> > don't seem to match what the docs and posts that I find on
> > the
> > > > > list
> > > > > > or
> > > > > > >>>> > other forums say.
> > > > > > >>>> >
> > > > > > >>>> > "storeUsage" is perfectly clear, it's the max space that
> > > > > persistent
> > > > > > >>>> > messages can use to be stored in disk.
> > > > > > >>>> > "tempUsage"" applies to file cursors on non-persistent
> > > messages,
> > > > > so
> > > > > > >>>> as to
> > > > > > >>>> > flush to disk if memory limits are reached (I don't care
> > much
> > > > > about
> > > > > > >>>> this
> > > > > > >>>> > one anyway, I always use persistent messages).
> > > > > > >>>>
> > > > > > >>>> Correct.
> > > > > > >>>>
> > > > > > >>>> >
> > > > > > >>>> > Now, according to most posts, memoryUsage would be the
> > maximum
> > > > > > memory
> > > > > > >>>> that
> > > > > > >>>> > the broker would be available to use.
> > > > > > >>>> > On this post:
> > > > > > >>>> >
> > > > > > >>>>
> > > > > >
> > > > >
> > > >
> > >
> >
> http://stackoverflow.com/questions/7646057/activemq-destinationpolicy-and-systemusage-configurationit
> > > > > > >>>> > says that "memoryUsage corresponds to the amount of memory
> > > > that's
> > > > > > >>>> > assigned to the in-memory store".
> > > > > > >>>>
> > > > > > >>>> Correct.
> > > > > > >>>>
> > > > > > >>>> >
> > > > > > >>>> > For example, on my tests using the following config (only
> > > > showing
> > > > > > >>>> relevant
> > > > > > >>>> > parts):
> > > > > > >>>> >
> > > > > > >>>> > <policyEntry queue=">" producerFlowControl="false"
> > > > > > >>>> optimizedDispatch="true">
> > > > > > >>>> > <deadLetterStrategy>
> > > > > > >>>> > <individualDeadLetterStrategy queuePrefix="DLQ."
> > > > > > >>>> > useQueueForQueueMessages="true" />
> > > > > > >>>> > </deadLetterStrategy>
> > > > > > >>>> > </policyEntry>
> > > > > > >>>> >
> > > > > > >>>> > <systemUsage>
> > > > > > >>>> > <systemUsage>
> > > > > > >>>> > <memoryUsage>
> > > > > > >>>> > <memoryUsage limit="100 mb"/>
> > > > > > >>>> > </memoryUsage>
> > > > > > >>>> > <storeUsage>
> > > > > > >>>> > <storeUsage limit="1 gb" name="foo"/>
> > > > > > >>>> > </storeUsage>
> > > > > > >>>> > <tempUsage>
> > > > > > >>>> > <tempUsage limit="3 gb"/>
> > > > > > >>>> > </tempUsage>
> > > > > > >>>> > </systemUsage>
> > > > > > >>>> > </systemUsage>
> > > > > > >>>> >
> > > > > > >>>> > With that config I would expect the broker to use 100 mb
> of
> > > > > maximum
> > > > > > >>>> memory
> > > > > > >>>> > among all queues. So it could maybe use 30mb in one queue
> > and
> > > > 70mb
> > > > > > in
> > > > > > >>>> > second queue.
> > > > > > >>>> >
> > > > > > >>>> >
> > > > > > >>>> > 1) What I'm seeing is that if I start feeding a queue
> > without
> > > > > > >>>> consuming it,
> > > > > > >>>> > the "Memory percent used" grows up to 70%, after that it
> > > doesn't
> > > > > > grow
> > > > > > >>>> > anymore.
> > > > > > >>>> > What is it doing exactly there? The first 70% is stored in
> > > > memory
> > > > > > >>>> (apart
> > > > > > >>>> > from disk since it's persistent), and all the rest that
> > > > continues
> > > > > > >>>> being fed
> > > > > > >>>> > goes just to disk?
> > > > > > >>>>
> > > > > > >>>> This behavior is correct. For queues the default cursor is
> > store
> > > > > > >>>> cursor. It keeps any newly arrived msgs in memory as long as
> > it
> > > > does
> > > > > > not
> > > > > > >>>> reach the configured memory limit (either configured on the
> > > queue
> > > > > per
> > > > > > >>>> destination or globally in memoryUsage settings).
> > > > > > >>>> Once the cursor reaches 70% of the configured limit (in your
> > > case
> > > > of
> > > > > > >>>> the memoryUsage limit since you don't specify a
> > per-destination
> > > > > > limit), it
> > > > > > >>>> will not keep any more messages in memory.
> > > > > > >>>> Instead it will reload these messages from the store when
> its
> > > time
> > > > > to
> > > > > > >>>> dispatch them. The broker anyway persists any msgs it
> receives
> > > > > before
> > > > > > >>>> passing on to the cursor.
> > > > > > >>>> This limit of 70% can be configured and raised to e..g 100%.
> > > > > > >>>> This behavior is kind of an optimization. That way you run
> > less
> > > > > often
> > > > > > >>>> into producer-flow-control.
> > > > > > >>>> As long as the persistence store is not running full, there
> is
> > > no
> > > > > need
> > > > > > >>>> to block producers, since the cursor can also load the
> > messages
> > > > from
> > > > > > the
> > > > > > >>>> store and does not necessarily have to keep them in memory.
> > > > > > >>>> If you configure the vmQueueCursor, then the behavior is
> > > > different.
> > > > > > >>>> This cursor will not be able to load msgs to the store but
> > needs
> > > > to
> > > > > > keep
> > > > > > >>>> them all in memory. The vmQueueCursor used to be the default
> > > > cursor
> > > > > in
> > > > > > >>>> older version of AMQ.
> > > > > > >>>>
> > > > > > >>>> Also note that topic msgs and non-persistent queue messages
> > are
> > > > not
> > > > > > >>>> handled by the store cursor. These msgs are held in memory
> and
> > > if
> > > > > > memory
> > > > > > >>>> runs low, get swapped out to temp storage.
> > > > > > >>>>
> > > > > > >>>> > 2) If then I start feeding a 2nd queue, "Memory percent
> > used"
> > > > > > >>>> continues
> > > > > > >>>> > growing until it reaches 140%. So it looks like
> memoryUsage
> > > does
> > > > > not
> > > > > > >>>> apply
> > > > > > >>>> > globally, but on a per queue basis?
> > > > > > >>>>
> > > > > > >>>> What version of AMQ do you use? The sum of the memory usage
> of
> > > all
> > > > > > >>>> queues should not go any higher than the configured
> > memoryUsage
> > > > > > limit. If
> > > > > > >>>> you're not on 5.5.1 or higher releases, then I suggest to
> > > upgrade.
> > > > > > >>>>
> > > > > > >>>> > Using memoryLimit on the queue's policyEntry gives more
> > > control
> > > > > over
> > > > > > >>>> this,
> > > > > > >>>> > but it's just a variation, "Memory percent used" can grow
> > more
> > > > > than
> > > > > > >>>> 100%
> > > > > > >>>> > anyway.
> > > > > > >>>>
> > > > > > >>>> With the default store cursor this should not be the case
> from
> > > > what
> > > > > I
> > > > > > >>>> know.
> > > > > > >>>>
> > > > > > >>>>
> > > > > > >>>> >
> > > > > > >>>> > 3) If #2 is true, then how would I prevent the broker from
> > > > running
> > > > > > >>>> out of
> > > > > > >>>> > memory in case queues would continue to be created?
> > > > > > >>>>
> > > > > > >>>> Just like above comment. I would expect the brokers
> > > > > MemoryPercentUsage
> > > > > > >>>> won't grow over 100% and the destinations MemoryPercentUsage
> > > > remains
> > > > > > fairly
> > > > > > >>>> much at 70%.
> > > > > > >>>> Not sure why you would see a different behavior? Using an
> old
> > > > > version
> > > > > > >>>> of AMQ perhaps? Or explicitly configuring for the
> > vmQueueCursor?
> > > > > > >>>> Could you perhaps also test with
> > > > > > >>>>
> > > > > > >>>> >
> > > > > > >>>> >
> > > > > > >>>> > Maybe I'm misunderstanding and some of these settings make
> > no
> > > > > sense
> > > > > > >>>> when
> > > > > > >>>> > producerFlowControl is disabled?
> > > > > > >>>> >
> > > > > > >>>> > Thanks in advance.
> > > > > > >>>> >
> > > > > > >>>> > Juan
> > > > > > >>>>
> > > > > > >>>>
> > > > > > >>>> Regards,
> > > > > > >>>>
> > > > > > >>>> Torsten Mielke
> > > > > > >>>> torsten@fusesource.com
> > > > > > >>>> tmielke.blogspot.com
> > > > > > >>>>
> > > > > > >>>>
> > > > > > >>>>
> > > > > > >>>>
> > > > > > >>>>
> > > > > > >>>
> > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > *Christian Posta*
> > > > > http://www.christianposta.com/blog
> > > > > twitter: @christianposta
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > *Christian Posta*
> > > http://www.christianposta.com/blog
> > > twitter: @christianposta
> > >
> >
>
>
>
> --
> *Christian Posta*
> http://www.christianposta.com/blog
> twitter: @christianposta
>
Re: Understanding memoryUsage (once again!)
Posted by Christian Posta <ch...@gmail.com>.
Another thought.. if you know they won't be immediately consumed (slow, or
even intermittent consumers), why keep them in memory at all? Or at least
why keep such a large number of them? Maybe turn down the memory limits on
the destinations with knowingly slow consumers so fewer messages are kept
in memory (they will be kept in store if they are persistent messages and
then recovered when they are ready to be dispatched). The only downside is
if the consumers speed up, you'll be fetching from disk more often.
Would be nice if the broker could auto-tune its memory usage... uh oh...
On Thu, Nov 29, 2012 at 12:45 PM, Juan Nin <ju...@gmail.com> wrote:
> Hi Christian!
>
> Yes, actually that's what I'm doing, just setting per destination policies
> which work for me.
> I anyway needed them because I'm creating queues with lots of messages,
> which won't be immediately consumed, so having them store a lot into memory
> ended up slowing things up.
>
> So I just assigned enough memory to the broker so as not run into issues.
>
> Thanks again.
>
>
> On Tue, Nov 27, 2012 at 9:40 PM, Christian Posta
> <ch...@gmail.com>wrote:
>
> > See inline...
> >
> >
> > On Wed, Nov 21, 2012 at 12:04 PM, Juan Nin <ju...@gmail.com> wrote:
> >
> > > Hi!
> > >
> > > Sorry for the delay in replying, buried on a project.
> > >
> > > As I mentioned before, I had tested this with 5.7.0 with the same
> > > behaviour.
> > > I just tested it again (both with 5.3.2 and 5.7.0) and same thing, and
> on
> > > my case it doesn't matter if there are consumers or not, it always
> seems
> > to
> > > make usage of the memory.
> > >
> > > Although I guess in theory that should not affect, did you use Stomp
> for
> > > your testing, or maybe you used Openwire?
> > > I'm using Stomp for my testing.
> > >
> > > Might be though that the broker's memory itself is not going beyond 70%
> > of
> > > memoryUsage, but this is just per destination counters as you
> mentioned.
> > > In which case I guess the value shown as "Memory percent used" is a bit
> > > confusing... But haven't had much time to really test the possibility
> of
> > > exhausting the broker's memory.
> > >
> > No, i believe what you're seeing is correct. The broker's memory limit is
> > going beyond memoryUsage (way beyond). When a queue checks whether memory
> > is full, it will only do something interesting if producer flow control
> is
> > enabled. Otherwise, it will continue on. You are seeing that it will
> > continue to add messages until the Queue's memory limit (40MB) reaches
> the
> > 70% mark. Since MemoryUsages are hierarchical, this means it will also
> > account for messages in the overall broker memory as well. For each
> queue,
> > you'll see that it will continue to hold 70% of 40MB of memory. What you
> > want in this case (if there are no consumers, or slow consumers) is to
> > raise your system usage memory limit OR lower your per-destination limits
> > OR lower your cursor highwatermark or a combination of all three.
> >
> > http://activemq.apache.org/per-destination-policies.html
> >
> > With PFC turned off, you're essentially telling the broker to take the
> > message no matter what. There is a point at which you will run out of
> > resources (memory, disk, etc). The trick is to find your use case and
> tune
> > for that.
> >
> >
> > >
> > > Will try to do some more testing soon...
> > >
> > > Thanks
> > >
> > >
> > > On Wed, Nov 21, 2012 at 2:28 PM, Christian Posta
> > > <ch...@gmail.com>wrote:
> > >
> > > > Can you please try on 5.7?
> > > > I just tried a test, and if there are no consumers to the queue then
> > the
> > > > memory usage will stay at 0%. The message will not be retained, ie,
> it
> > > will
> > > > be put into the store and kept there. If I add a consumer, and not
> try
> > to
> > > > consume, the message will be kept around in memory up to the cursor
> > high
> > > > watermark (70 by default).
> > > >
> > > > As I add more queues the same behavior as described above will
> happen.
> > > If I
> > > > attach consumers to the queues without consuming them (so no messages
> > are
> > > > consumed), then messages are kept in the cursor up to the high-water
> > > > mark... note.. the high-water mark is relative to the
> > > Destination/Cursor's
> > > > MemoryUsage, not the global memory usage.
> > > >
> > > > If I continue adding queues, and with producer flow control set to
> > > false, I
> > > > too will see the *Global* memory usage go much higher than 100%. This
> > is
> > > > not surprising though, because as I understand, these usage memory
> > > objects
> > > > are really just counters. They don't enforce anything. When coupled
> > with
> > > > producer flow control, they can be used to determine when to enable
> > PFC.
> > > If
> > > > PFC is false, it's up to the cursor to determine when to flush out to
> > > disk.
> > > > But each destination/cursor will have it's own system usage (with the
> > > > global as the parent).
> > > >
> > > > Hope this helps. Can you please try with 5.7 and give us a report
> back?
> > > > Thanks,
> > > > Christian
> > > >
> > > >
> > > >
> > > > On Fri, Nov 16, 2012 at 11:38 AM, Juan Nin <ju...@gmail.com> wrote:
> > > >
> > > > > nope, adding a 3rd queue the 3rd one also gets this same value, so
> > even
> > > > if
> > > > > it's the memory usage of the queue it's anyway going beyond..
> > > > >
> > > > >
> > > > > On Fri, Nov 16, 2012 at 4:32 PM, Juan Nin <ju...@gmail.com>
> wrote:
> > > > >
> > > > > > Might it be just a bug on how the MemoryPercentUsage is
> calculated?
> > > > > >
> > > > > > If I connect via JMX using console, I can see the
> > MemoryPercentUsage
> > > as
> > > > > > 112 right now.
> > > > > > If I go to each of the 2 queues on them I see CursorMemoryUsage
> > with
> > > > > value
> > > > > > 29360604, which would be 28mb each, summing a total of 56mb
> (just a
> > > bit
> > > > > > more than the specified memoryUsage of 50mb).
> > > > > >
> > > > > > Not sure I'm interpreting these values correctly though, first
> > time I
> > > > > > access it via jconsole...
> > > > > >
> > > > > >
> > > > > > On Fri, Nov 16, 2012 at 4:07 PM, Juan Nin <ju...@gmail.com>
> > wrote:
> > > > > >
> > > > > >> On that config there's a 40mb memoryLimit per queue, but also
> > tested
> > > > it
> > > > > >> without it with same results.
> > > > > >>
> > > > > >>
> > > > > >> On Fri, Nov 16, 2012 at 4:05 PM, Juan Nin <ju...@gmail.com>
> > wrote:
> > > > > >>
> > > > > >>> Hi Torsten!
> > > > > >>>
> > > > > >>> I'm using ActiveMQ 5.3.2, but also tested it on 5.7.0 with the
> > same
> > > > > >>> results...
> > > > > >>> This is my 5.3.2 config:
> > > > > >>>
> > > > > >>> <beans
> > > > > >>> xmlns="http://www.springframework.org/schema/beans"
> > > > > >>> xmlns:amq="http://activemq.apache.org/schema/core"
> > > > > >>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
> > > > > >>> xsi:schemaLocation="
> > http://www.springframework.org/schema/beans
> > > > > >>>
> http://www.springframework.org/schema/beans/spring-beans-2.0.xsd
> > > > > >>> http://activemq.apache.org/schema/core
> > > > > >>> http://activemq.apache.org/schema/core/activemq-core.xsd">
> > > > > >>>
> > > > > >>> <bean
> > > > > >>>
> > > > >
> > > >
> > >
> >
> class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
> > > > > >>> <property name="locations">
> > > > > >>>
> > > > > >>>
> <value>file:${activemq.base}/conf/credentials.properties</value>
> > > > > >>> </property>
> > > > > >>> </bean>
> > > > > >>>
> > > > > >>> <broker xmlns="http://activemq.apache.org/schema/core"
> > > > > >>> brokerName="localhost" dataDirectory="${activemq.base}/data"
> > > > > >>> destroyApplicationContextOnStop="true" advisorySupport="false">
> > > > > >>>
> > > > > >>> <destinationPolicy>
> > > > > >>> <policyMap>
> > > > > >>> <policyEntries>
> > > > > >>> <policyEntry topic=">"
> > > producerFlowControl="true"
> > > > > >>> memoryLimit="5mb">
> > > > > >>> <pendingSubscriberPolicy>
> > > > > >>> <vmCursor />
> > > > > >>> </pendingSubscriberPolicy>
> > > > > >>> </policyEntry>
> > > > > >>> <policyEntry queue=">"
> > > > producerFlowControl="false"
> > > > > >>> optimizedDispatch="true" memoryLimit="40mb">
> > > > > >>> <deadLetterStrategy>
> > > > > >>> <individualDeadLetterStrategy
> > > > > >>> queuePrefix="DLQ." useQueueForQueueMessages="true" />
> > > > > >>> </deadLetterStrategy>
> > > > > >>> </policyEntry>
> > > > > >>> </policyEntries>
> > > > > >>> </policyMap>
> > > > > >>> </destinationPolicy>
> > > > > >>>
> > > > > >>> <managementContext>
> > > > > >>> <managementContext connectorPort="2011"/>
> > > > > >>> </managementContext>
> > > > > >>>
> > > > > >>> <persistenceAdapter>
> > > > > >>> <kahaDB directory="${activemq.base}/data/kahadb"
> > > > > >>> enableJournalDiskSyncs="false" indexWriteBatchSize="10000"
> > > > > >>> indexCacheSize="1000"/>
> > > > > >>> </persistenceAdapter>
> > > > > >>>
> > > > > >>> <systemUsage>
> > > > > >>> <systemUsage>
> > > > > >>> <memoryUsage>
> > > > > >>> <memoryUsage limit="50 mb"/>
> > > > > >>> </memoryUsage>
> > > > > >>> <storeUsage>
> > > > > >>> <storeUsage limit="1 gb" name="foo"/>
> > > > > >>> </storeUsage>
> > > > > >>> <tempUsage>
> > > > > >>> <tempUsage limit="3 gb"/>
> > > > > >>> </tempUsage>
> > > > > >>> </systemUsage>
> > > > > >>> </systemUsage>
> > > > > >>>
> > > > > >>> <transportConnectors>
> > > > > >>> <transportConnector name="openwire" uri="tcp://
> > > > > 0.0.0.0:61616
> > > > > >>> "/>
> > > > > >>> <transportConnector name="stomp" uri="stomp://
> 0.0.0.0:61613
> > "/>
> > > > > >>> </transportConnectors>
> > > > > >>>
> > > > > >>> </broker>
> > > > > >>>
> > > > > >>> <import resource="jetty.xml"/>
> > > > > >>>
> > > > > >>> </beans>
> > > > > >>>
> > > > > >>>
> > > > > >>> Using just a simple PHP script with Stomp for feeding the
> queues
> > > > > >>> (running it twice with different queue name):
> > > > > >>>
> > > > > >>> <?php
> > > > > >>>
> > > > > >>> require_once("Stomp.php");
> > > > > >>>
> > > > > >>> $amq = new Stomp("tcp://localhost:61613");
> > > > > >>> $amq->connect();
> > > > > >>>
> > > > > >>> for($i=1; $i <= 100000; $i++)
> > > > > >>> {
> > > > > >>> if($i%1000 == 0)
> > > > > >>> {
> > > > > >>> echo "\nmsg #: $i";
> > > > > >>> }
> > > > > >>> $amq->send("/queue/test", "this is test message # $i"
> > > > > >>> ,array('persistent' => 'true'));
> > > > > >>> }
> > > > > >>>
> > > > > >>> $amq->disconnect();
> > > > > >>>
> > > > > >>> ?>
> > > > > >>>
> > > > > >>>
> > > > > >>>
> > > > > >>> On Fri, Nov 16, 2012 at 3:47 PM, Torsten Mielke <
> > > > > torsten@fusesource.com>wrote:
> > > > > >>>
> > > > > >>>> Hello,
> > > > > >>>>
> > > > > >>>> See in-line response.
> > > > > >>>>
> > > > > >>>> On Nov 16, 2012, at 6:29 PM, Juan Nin wrote:
> > > > > >>>>
> > > > > >>>> > Hi!
> > > > > >>>> >
> > > > > >>>> > After some heavy digging about Producer Flow control and the
> > > > > >>>> systemUsage
> > > > > >>>> > properties a couple of years ago, I thought I quite
> understood
> > > it.
> > > > > >>>> > But yesterday I found that one of my configs was not
> behaving
> > > > > exactly
> > > > > >>>> as I
> > > > > >>>> > expected, so started doing some tests, and I see certain
> > > > behaviours
> > > > > >>>> which
> > > > > >>>> > don't seem to match what the docs and posts that I find on
> the
> > > > list
> > > > > or
> > > > > >>>> > other forums say.
> > > > > >>>> >
> > > > > >>>> > "storeUsage" is perfectly clear, it's the max space that
> > > > persistent
> > > > > >>>> > messages can use to be stored in disk.
> > > > > >>>> > "tempUsage"" applies to file cursors on non-persistent
> > messages,
> > > > so
> > > > > >>>> as to
> > > > > >>>> > flush to disk if memory limits are reached (I don't care
> much
> > > > about
> > > > > >>>> this
> > > > > >>>> > one anyway, I always use persistent messages).
> > > > > >>>>
> > > > > >>>> Correct.
> > > > > >>>>
> > > > > >>>> >
> > > > > >>>> > Now, according to most posts, memoryUsage would be the
> maximum
> > > > > memory
> > > > > >>>> that
> > > > > >>>> > the broker would be available to use.
> > > > > >>>> > On this post:
> > > > > >>>> >
> > > > > >>>>
> > > > >
> > > >
> > >
> >
> http://stackoverflow.com/questions/7646057/activemq-destinationpolicy-and-systemusage-configurationit
> > > > > >>>> > says that "memoryUsage corresponds to the amount of memory
> > > that's
> > > > > >>>> > assigned to the in-memory store".
> > > > > >>>>
> > > > > >>>> Correct.
> > > > > >>>>
> > > > > >>>> >
> > > > > >>>> > For example, on my tests using the following config (only
> > > showing
> > > > > >>>> relevant
> > > > > >>>> > parts):
> > > > > >>>> >
> > > > > >>>> > <policyEntry queue=">" producerFlowControl="false"
> > > > > >>>> optimizedDispatch="true">
> > > > > >>>> > <deadLetterStrategy>
> > > > > >>>> > <individualDeadLetterStrategy queuePrefix="DLQ."
> > > > > >>>> > useQueueForQueueMessages="true" />
> > > > > >>>> > </deadLetterStrategy>
> > > > > >>>> > </policyEntry>
> > > > > >>>> >
> > > > > >>>> > <systemUsage>
> > > > > >>>> > <systemUsage>
> > > > > >>>> > <memoryUsage>
> > > > > >>>> > <memoryUsage limit="100 mb"/>
> > > > > >>>> > </memoryUsage>
> > > > > >>>> > <storeUsage>
> > > > > >>>> > <storeUsage limit="1 gb" name="foo"/>
> > > > > >>>> > </storeUsage>
> > > > > >>>> > <tempUsage>
> > > > > >>>> > <tempUsage limit="3 gb"/>
> > > > > >>>> > </tempUsage>
> > > > > >>>> > </systemUsage>
> > > > > >>>> > </systemUsage>
> > > > > >>>> >
> > > > > >>>> > With that config I would expect the broker to use 100 mb of
> > > > maximum
> > > > > >>>> memory
> > > > > >>>> > among all queues. So it could maybe use 30mb in one queue
> and
> > > 70mb
> > > > > in
> > > > > >>>> > second queue.
> > > > > >>>> >
> > > > > >>>> >
> > > > > >>>> > 1) What I'm seeing is that if I start feeding a queue
> without
> > > > > >>>> consuming it,
> > > > > >>>> > the "Memory percent used" grows up to 70%, after that it
> > doesn't
> > > > > grow
> > > > > >>>> > anymore.
> > > > > >>>> > What is it doing exactly there? The first 70% is stored in
> > > memory
> > > > > >>>> (apart
> > > > > >>>> > from disk since it's persistent), and all the rest that
> > > continues
> > > > > >>>> being fed
> > > > > >>>> > goes just to disk?
> > > > > >>>>
> > > > > >>>> This behavior is correct. For queues the default cursor is
> store
> > > > > >>>> cursor. It keeps any newly arrived msgs in memory as long as
> it
> > > does
> > > > > not
> > > > > >>>> reach the configured memory limit (either configured on the
> > queue
> > > > per
> > > > > >>>> destination or globally in memoryUsage settings).
> > > > > >>>> Once the cursor reaches 70% of the configured limit (in your
> > case
> > > of
> > > > > >>>> the memoryUsage limit since you don't specify a
> per-destination
> > > > > limit), it
> > > > > >>>> will not keep any more messages in memory.
> > > > > >>>> Instead it will reload these messages from the store when its
> > time
> > > > to
> > > > > >>>> dispatch them. The broker anyway persists any msgs it receives
> > > > before
> > > > > >>>> passing on to the cursor.
> > > > > >>>> This limit of 70% can be configured and raised to e..g 100%.
> > > > > >>>> This behavior is kind of an optimization. That way you run
> less
> > > > often
> > > > > >>>> into producer-flow-control.
> > > > > >>>> As long as the persistence store is not running full, there is
> > no
> > > > need
> > > > > >>>> to block producers, since the cursor can also load the
> messages
> > > from
> > > > > the
> > > > > >>>> store and does not necessarily have to keep them in memory.
> > > > > >>>> If you configure the vmQueueCursor, then the behavior is
> > > different.
> > > > > >>>> This cursor will not be able to load msgs to the store but
> needs
> > > to
> > > > > keep
> > > > > >>>> them all in memory. The vmQueueCursor used to be the default
> > > cursor
> > > > in
> > > > > >>>> older version of AMQ.
> > > > > >>>>
> > > > > >>>> Also note that topic msgs and non-persistent queue messages
> are
> > > not
> > > > > >>>> handled by the store cursor. These msgs are held in memory and
> > if
> > > > > memory
> > > > > >>>> runs low, get swapped out to temp storage.
> > > > > >>>>
> > > > > >>>> > 2) If then I start feeding a 2nd queue, "Memory percent
> used"
> > > > > >>>> continues
> > > > > >>>> > growing until it reaches 140%. So it looks like memoryUsage
> > does
> > > > not
> > > > > >>>> apply
> > > > > >>>> > globally, but on a per queue basis?
> > > > > >>>>
> > > > > >>>> What version of AMQ do you use? The sum of the memory usage of
> > all
> > > > > >>>> queues should not go any higher than the configured
> memoryUsage
> > > > > limit. If
> > > > > >>>> you're not on 5.5.1 or higher releases, then I suggest to
> > upgrade.
> > > > > >>>>
> > > > > >>>> > Using memoryLimit on the queue's policyEntry gives more
> > control
> > > > over
> > > > > >>>> this,
> > > > > >>>> > but it's just a variation, "Memory percent used" can grow
> more
> > > > than
> > > > > >>>> 100%
> > > > > >>>> > anyway.
> > > > > >>>>
> > > > > >>>> With the default store cursor this should not be the case from
> > > what
> > > > I
> > > > > >>>> know.
> > > > > >>>>
> > > > > >>>>
> > > > > >>>> >
> > > > > >>>> > 3) If #2 is true, then how would I prevent the broker from
> > > running
> > > > > >>>> out of
> > > > > >>>> > memory in case queues would continue to be created?
> > > > > >>>>
> > > > > >>>> Just like above comment. I would expect the brokers
> > > > MemoryPercentUsage
> > > > > >>>> won't grow over 100% and the destinations MemoryPercentUsage
> > > remains
> > > > > fairly
> > > > > >>>> much at 70%.
> > > > > >>>> Not sure why you would see a different behavior? Using an old
> > > > version
> > > > > >>>> of AMQ perhaps? Or explicitly configuring for the
> vmQueueCursor?
> > > > > >>>> Could you perhaps also test with
> > > > > >>>>
> > > > > >>>> >
> > > > > >>>> >
> > > > > >>>> > Maybe I'm misunderstanding and some of these settings make
> no
> > > > sense
> > > > > >>>> when
> > > > > >>>> > producerFlowControl is disabled?
> > > > > >>>> >
> > > > > >>>> > Thanks in advance.
> > > > > >>>> >
> > > > > >>>> > Juan
> > > > > >>>>
> > > > > >>>>
> > > > > >>>> Regards,
> > > > > >>>>
> > > > > >>>> Torsten Mielke
> > > > > >>>> torsten@fusesource.com
> > > > > >>>> tmielke.blogspot.com
> > > > > >>>>
> > > > > >>>>
> > > > > >>>>
> > > > > >>>>
> > > > > >>>>
> > > > > >>>
> > > > > >>
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > *Christian Posta*
> > > > http://www.christianposta.com/blog
> > > > twitter: @christianposta
> > > >
> > >
> >
> >
> >
> > --
> > *Christian Posta*
> > http://www.christianposta.com/blog
> > twitter: @christianposta
> >
>
--
*Christian Posta*
http://www.christianposta.com/blog
twitter: @christianposta
Re: Understanding memoryUsage (once again!)
Posted by Juan Nin <ju...@gmail.com>.
Hi Christian!
Yes, actually that's what I'm doing, just setting per destination policies
which work for me.
I anyway needed them because I'm creating queues with lots of messages,
which won't be immediately consumed, so having them store a lot into memory
ended up slowing things up.
So I just assigned enough memory to the broker so as not run into issues.
Thanks again.
On Tue, Nov 27, 2012 at 9:40 PM, Christian Posta
<ch...@gmail.com>wrote:
> See inline...
>
>
> On Wed, Nov 21, 2012 at 12:04 PM, Juan Nin <ju...@gmail.com> wrote:
>
> > Hi!
> >
> > Sorry for the delay in replying, buried on a project.
> >
> > As I mentioned before, I had tested this with 5.7.0 with the same
> > behaviour.
> > I just tested it again (both with 5.3.2 and 5.7.0) and same thing, and on
> > my case it doesn't matter if there are consumers or not, it always seems
> to
> > make usage of the memory.
> >
> > Although I guess in theory that should not affect, did you use Stomp for
> > your testing, or maybe you used Openwire?
> > I'm using Stomp for my testing.
> >
> > Might be though that the broker's memory itself is not going beyond 70%
> of
> > memoryUsage, but this is just per destination counters as you mentioned.
> > In which case I guess the value shown as "Memory percent used" is a bit
> > confusing... But haven't had much time to really test the possibility of
> > exhausting the broker's memory.
> >
> No, i believe what you're seeing is correct. The broker's memory limit is
> going beyond memoryUsage (way beyond). When a queue checks whether memory
> is full, it will only do something interesting if producer flow control is
> enabled. Otherwise, it will continue on. You are seeing that it will
> continue to add messages until the Queue's memory limit (40MB) reaches the
> 70% mark. Since MemoryUsages are hierarchical, this means it will also
> account for messages in the overall broker memory as well. For each queue,
> you'll see that it will continue to hold 70% of 40MB of memory. What you
> want in this case (if there are no consumers, or slow consumers) is to
> raise your system usage memory limit OR lower your per-destination limits
> OR lower your cursor highwatermark or a combination of all three.
>
> http://activemq.apache.org/per-destination-policies.html
>
> With PFC turned off, you're essentially telling the broker to take the
> message no matter what. There is a point at which you will run out of
> resources (memory, disk, etc). The trick is to find your use case and tune
> for that.
>
>
> >
> > Will try to do some more testing soon...
> >
> > Thanks
> >
> >
> > On Wed, Nov 21, 2012 at 2:28 PM, Christian Posta
> > <ch...@gmail.com>wrote:
> >
> > > Can you please try on 5.7?
> > > I just tried a test, and if there are no consumers to the queue then
> the
> > > memory usage will stay at 0%. The message will not be retained, ie, it
> > will
> > > be put into the store and kept there. If I add a consumer, and not try
> to
> > > consume, the message will be kept around in memory up to the cursor
> high
> > > watermark (70 by default).
> > >
> > > As I add more queues the same behavior as described above will happen.
> > If I
> > > attach consumers to the queues without consuming them (so no messages
> are
> > > consumed), then messages are kept in the cursor up to the high-water
> > > mark... note.. the high-water mark is relative to the
> > Destination/Cursor's
> > > MemoryUsage, not the global memory usage.
> > >
> > > If I continue adding queues, and with producer flow control set to
> > false, I
> > > too will see the *Global* memory usage go much higher than 100%. This
> is
> > > not surprising though, because as I understand, these usage memory
> > objects
> > > are really just counters. They don't enforce anything. When coupled
> with
> > > producer flow control, they can be used to determine when to enable
> PFC.
> > If
> > > PFC is false, it's up to the cursor to determine when to flush out to
> > disk.
> > > But each destination/cursor will have it's own system usage (with the
> > > global as the parent).
> > >
> > > Hope this helps. Can you please try with 5.7 and give us a report back?
> > > Thanks,
> > > Christian
> > >
> > >
> > >
> > > On Fri, Nov 16, 2012 at 11:38 AM, Juan Nin <ju...@gmail.com> wrote:
> > >
> > > > nope, adding a 3rd queue the 3rd one also gets this same value, so
> even
> > > if
> > > > it's the memory usage of the queue it's anyway going beyond..
> > > >
> > > >
> > > > On Fri, Nov 16, 2012 at 4:32 PM, Juan Nin <ju...@gmail.com> wrote:
> > > >
> > > > > Might it be just a bug on how the MemoryPercentUsage is calculated?
> > > > >
> > > > > If I connect via JMX using console, I can see the
> MemoryPercentUsage
> > as
> > > > > 112 right now.
> > > > > If I go to each of the 2 queues on them I see CursorMemoryUsage
> with
> > > > value
> > > > > 29360604, which would be 28mb each, summing a total of 56mb (just a
> > bit
> > > > > more than the specified memoryUsage of 50mb).
> > > > >
> > > > > Not sure I'm interpreting these values correctly though, first
> time I
> > > > > access it via jconsole...
> > > > >
> > > > >
> > > > > On Fri, Nov 16, 2012 at 4:07 PM, Juan Nin <ju...@gmail.com>
> wrote:
> > > > >
> > > > >> On that config there's a 40mb memoryLimit per queue, but also
> tested
> > > it
> > > > >> without it with same results.
> > > > >>
> > > > >>
> > > > >> On Fri, Nov 16, 2012 at 4:05 PM, Juan Nin <ju...@gmail.com>
> wrote:
> > > > >>
> > > > >>> Hi Torsten!
> > > > >>>
> > > > >>> I'm using ActiveMQ 5.3.2, but also tested it on 5.7.0 with the
> same
> > > > >>> results...
> > > > >>> This is my 5.3.2 config:
> > > > >>>
> > > > >>> <beans
> > > > >>> xmlns="http://www.springframework.org/schema/beans"
> > > > >>> xmlns:amq="http://activemq.apache.org/schema/core"
> > > > >>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
> > > > >>> xsi:schemaLocation="
> http://www.springframework.org/schema/beans
> > > > >>> http://www.springframework.org/schema/beans/spring-beans-2.0.xsd
> > > > >>> http://activemq.apache.org/schema/core
> > > > >>> http://activemq.apache.org/schema/core/activemq-core.xsd">
> > > > >>>
> > > > >>> <bean
> > > > >>>
> > > >
> > >
> >
> class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
> > > > >>> <property name="locations">
> > > > >>>
> > > > >>> <value>file:${activemq.base}/conf/credentials.properties</value>
> > > > >>> </property>
> > > > >>> </bean>
> > > > >>>
> > > > >>> <broker xmlns="http://activemq.apache.org/schema/core"
> > > > >>> brokerName="localhost" dataDirectory="${activemq.base}/data"
> > > > >>> destroyApplicationContextOnStop="true" advisorySupport="false">
> > > > >>>
> > > > >>> <destinationPolicy>
> > > > >>> <policyMap>
> > > > >>> <policyEntries>
> > > > >>> <policyEntry topic=">"
> > producerFlowControl="true"
> > > > >>> memoryLimit="5mb">
> > > > >>> <pendingSubscriberPolicy>
> > > > >>> <vmCursor />
> > > > >>> </pendingSubscriberPolicy>
> > > > >>> </policyEntry>
> > > > >>> <policyEntry queue=">"
> > > producerFlowControl="false"
> > > > >>> optimizedDispatch="true" memoryLimit="40mb">
> > > > >>> <deadLetterStrategy>
> > > > >>> <individualDeadLetterStrategy
> > > > >>> queuePrefix="DLQ." useQueueForQueueMessages="true" />
> > > > >>> </deadLetterStrategy>
> > > > >>> </policyEntry>
> > > > >>> </policyEntries>
> > > > >>> </policyMap>
> > > > >>> </destinationPolicy>
> > > > >>>
> > > > >>> <managementContext>
> > > > >>> <managementContext connectorPort="2011"/>
> > > > >>> </managementContext>
> > > > >>>
> > > > >>> <persistenceAdapter>
> > > > >>> <kahaDB directory="${activemq.base}/data/kahadb"
> > > > >>> enableJournalDiskSyncs="false" indexWriteBatchSize="10000"
> > > > >>> indexCacheSize="1000"/>
> > > > >>> </persistenceAdapter>
> > > > >>>
> > > > >>> <systemUsage>
> > > > >>> <systemUsage>
> > > > >>> <memoryUsage>
> > > > >>> <memoryUsage limit="50 mb"/>
> > > > >>> </memoryUsage>
> > > > >>> <storeUsage>
> > > > >>> <storeUsage limit="1 gb" name="foo"/>
> > > > >>> </storeUsage>
> > > > >>> <tempUsage>
> > > > >>> <tempUsage limit="3 gb"/>
> > > > >>> </tempUsage>
> > > > >>> </systemUsage>
> > > > >>> </systemUsage>
> > > > >>>
> > > > >>> <transportConnectors>
> > > > >>> <transportConnector name="openwire" uri="tcp://
> > > > 0.0.0.0:61616
> > > > >>> "/>
> > > > >>> <transportConnector name="stomp" uri="stomp://0.0.0.0:61613
> "/>
> > > > >>> </transportConnectors>
> > > > >>>
> > > > >>> </broker>
> > > > >>>
> > > > >>> <import resource="jetty.xml"/>
> > > > >>>
> > > > >>> </beans>
> > > > >>>
> > > > >>>
> > > > >>> Using just a simple PHP script with Stomp for feeding the queues
> > > > >>> (running it twice with different queue name):
> > > > >>>
> > > > >>> <?php
> > > > >>>
> > > > >>> require_once("Stomp.php");
> > > > >>>
> > > > >>> $amq = new Stomp("tcp://localhost:61613");
> > > > >>> $amq->connect();
> > > > >>>
> > > > >>> for($i=1; $i <= 100000; $i++)
> > > > >>> {
> > > > >>> if($i%1000 == 0)
> > > > >>> {
> > > > >>> echo "\nmsg #: $i";
> > > > >>> }
> > > > >>> $amq->send("/queue/test", "this is test message # $i"
> > > > >>> ,array('persistent' => 'true'));
> > > > >>> }
> > > > >>>
> > > > >>> $amq->disconnect();
> > > > >>>
> > > > >>> ?>
> > > > >>>
> > > > >>>
> > > > >>>
> > > > >>> On Fri, Nov 16, 2012 at 3:47 PM, Torsten Mielke <
> > > > torsten@fusesource.com>wrote:
> > > > >>>
> > > > >>>> Hello,
> > > > >>>>
> > > > >>>> See in-line response.
> > > > >>>>
> > > > >>>> On Nov 16, 2012, at 6:29 PM, Juan Nin wrote:
> > > > >>>>
> > > > >>>> > Hi!
> > > > >>>> >
> > > > >>>> > After some heavy digging about Producer Flow control and the
> > > > >>>> systemUsage
> > > > >>>> > properties a couple of years ago, I thought I quite understood
> > it.
> > > > >>>> > But yesterday I found that one of my configs was not behaving
> > > > exactly
> > > > >>>> as I
> > > > >>>> > expected, so started doing some tests, and I see certain
> > > behaviours
> > > > >>>> which
> > > > >>>> > don't seem to match what the docs and posts that I find on the
> > > list
> > > > or
> > > > >>>> > other forums say.
> > > > >>>> >
> > > > >>>> > "storeUsage" is perfectly clear, it's the max space that
> > > persistent
> > > > >>>> > messages can use to be stored in disk.
> > > > >>>> > "tempUsage"" applies to file cursors on non-persistent
> messages,
> > > so
> > > > >>>> as to
> > > > >>>> > flush to disk if memory limits are reached (I don't care much
> > > about
> > > > >>>> this
> > > > >>>> > one anyway, I always use persistent messages).
> > > > >>>>
> > > > >>>> Correct.
> > > > >>>>
> > > > >>>> >
> > > > >>>> > Now, according to most posts, memoryUsage would be the maximum
> > > > memory
> > > > >>>> that
> > > > >>>> > the broker would be available to use.
> > > > >>>> > On this post:
> > > > >>>> >
> > > > >>>>
> > > >
> > >
> >
> http://stackoverflow.com/questions/7646057/activemq-destinationpolicy-and-systemusage-configurationit
> > > > >>>> > says that "memoryUsage corresponds to the amount of memory
> > that's
> > > > >>>> > assigned to the in-memory store".
> > > > >>>>
> > > > >>>> Correct.
> > > > >>>>
> > > > >>>> >
> > > > >>>> > For example, on my tests using the following config (only
> > showing
> > > > >>>> relevant
> > > > >>>> > parts):
> > > > >>>> >
> > > > >>>> > <policyEntry queue=">" producerFlowControl="false"
> > > > >>>> optimizedDispatch="true">
> > > > >>>> > <deadLetterStrategy>
> > > > >>>> > <individualDeadLetterStrategy queuePrefix="DLQ."
> > > > >>>> > useQueueForQueueMessages="true" />
> > > > >>>> > </deadLetterStrategy>
> > > > >>>> > </policyEntry>
> > > > >>>> >
> > > > >>>> > <systemUsage>
> > > > >>>> > <systemUsage>
> > > > >>>> > <memoryUsage>
> > > > >>>> > <memoryUsage limit="100 mb"/>
> > > > >>>> > </memoryUsage>
> > > > >>>> > <storeUsage>
> > > > >>>> > <storeUsage limit="1 gb" name="foo"/>
> > > > >>>> > </storeUsage>
> > > > >>>> > <tempUsage>
> > > > >>>> > <tempUsage limit="3 gb"/>
> > > > >>>> > </tempUsage>
> > > > >>>> > </systemUsage>
> > > > >>>> > </systemUsage>
> > > > >>>> >
> > > > >>>> > With that config I would expect the broker to use 100 mb of
> > > maximum
> > > > >>>> memory
> > > > >>>> > among all queues. So it could maybe use 30mb in one queue and
> > 70mb
> > > > in
> > > > >>>> > second queue.
> > > > >>>> >
> > > > >>>> >
> > > > >>>> > 1) What I'm seeing is that if I start feeding a queue without
> > > > >>>> consuming it,
> > > > >>>> > the "Memory percent used" grows up to 70%, after that it
> doesn't
> > > > grow
> > > > >>>> > anymore.
> > > > >>>> > What is it doing exactly there? The first 70% is stored in
> > memory
> > > > >>>> (apart
> > > > >>>> > from disk since it's persistent), and all the rest that
> > continues
> > > > >>>> being fed
> > > > >>>> > goes just to disk?
> > > > >>>>
> > > > >>>> This behavior is correct. For queues the default cursor is store
> > > > >>>> cursor. It keeps any newly arrived msgs in memory as long as it
> > does
> > > > not
> > > > >>>> reach the configured memory limit (either configured on the
> queue
> > > per
> > > > >>>> destination or globally in memoryUsage settings).
> > > > >>>> Once the cursor reaches 70% of the configured limit (in your
> case
> > of
> > > > >>>> the memoryUsage limit since you don't specify a per-destination
> > > > limit), it
> > > > >>>> will not keep any more messages in memory.
> > > > >>>> Instead it will reload these messages from the store when its
> time
> > > to
> > > > >>>> dispatch them. The broker anyway persists any msgs it receives
> > > before
> > > > >>>> passing on to the cursor.
> > > > >>>> This limit of 70% can be configured and raised to e..g 100%.
> > > > >>>> This behavior is kind of an optimization. That way you run less
> > > often
> > > > >>>> into producer-flow-control.
> > > > >>>> As long as the persistence store is not running full, there is
> no
> > > need
> > > > >>>> to block producers, since the cursor can also load the messages
> > from
> > > > the
> > > > >>>> store and does not necessarily have to keep them in memory.
> > > > >>>> If you configure the vmQueueCursor, then the behavior is
> > different.
> > > > >>>> This cursor will not be able to load msgs to the store but needs
> > to
> > > > keep
> > > > >>>> them all in memory. The vmQueueCursor used to be the default
> > cursor
> > > in
> > > > >>>> older version of AMQ.
> > > > >>>>
> > > > >>>> Also note that topic msgs and non-persistent queue messages are
> > not
> > > > >>>> handled by the store cursor. These msgs are held in memory and
> if
> > > > memory
> > > > >>>> runs low, get swapped out to temp storage.
> > > > >>>>
> > > > >>>> > 2) If then I start feeding a 2nd queue, "Memory percent used"
> > > > >>>> continues
> > > > >>>> > growing until it reaches 140%. So it looks like memoryUsage
> does
> > > not
> > > > >>>> apply
> > > > >>>> > globally, but on a per queue basis?
> > > > >>>>
> > > > >>>> What version of AMQ do you use? The sum of the memory usage of
> all
> > > > >>>> queues should not go any higher than the configured memoryUsage
> > > > limit. If
> > > > >>>> you're not on 5.5.1 or higher releases, then I suggest to
> upgrade.
> > > > >>>>
> > > > >>>> > Using memoryLimit on the queue's policyEntry gives more
> control
> > > over
> > > > >>>> this,
> > > > >>>> > but it's just a variation, "Memory percent used" can grow more
> > > than
> > > > >>>> 100%
> > > > >>>> > anyway.
> > > > >>>>
> > > > >>>> With the default store cursor this should not be the case from
> > what
> > > I
> > > > >>>> know.
> > > > >>>>
> > > > >>>>
> > > > >>>> >
> > > > >>>> > 3) If #2 is true, then how would I prevent the broker from
> > running
> > > > >>>> out of
> > > > >>>> > memory in case queues would continue to be created?
> > > > >>>>
> > > > >>>> Just like above comment. I would expect the brokers
> > > MemoryPercentUsage
> > > > >>>> won't grow over 100% and the destinations MemoryPercentUsage
> > remains
> > > > fairly
> > > > >>>> much at 70%.
> > > > >>>> Not sure why you would see a different behavior? Using an old
> > > version
> > > > >>>> of AMQ perhaps? Or explicitly configuring for the vmQueueCursor?
> > > > >>>> Could you perhaps also test with
> > > > >>>>
> > > > >>>> >
> > > > >>>> >
> > > > >>>> > Maybe I'm misunderstanding and some of these settings make no
> > > sense
> > > > >>>> when
> > > > >>>> > producerFlowControl is disabled?
> > > > >>>> >
> > > > >>>> > Thanks in advance.
> > > > >>>> >
> > > > >>>> > Juan
> > > > >>>>
> > > > >>>>
> > > > >>>> Regards,
> > > > >>>>
> > > > >>>> Torsten Mielke
> > > > >>>> torsten@fusesource.com
> > > > >>>> tmielke.blogspot.com
> > > > >>>>
> > > > >>>>
> > > > >>>>
> > > > >>>>
> > > > >>>>
> > > > >>>
> > > > >>
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > *Christian Posta*
> > > http://www.christianposta.com/blog
> > > twitter: @christianposta
> > >
> >
>
>
>
> --
> *Christian Posta*
> http://www.christianposta.com/blog
> twitter: @christianposta
>
Re: Understanding memoryUsage (once again!)
Posted by Christian Posta <ch...@gmail.com>.
See inline...
On Wed, Nov 21, 2012 at 12:04 PM, Juan Nin <ju...@gmail.com> wrote:
> Hi!
>
> Sorry for the delay in replying, buried on a project.
>
> As I mentioned before, I had tested this with 5.7.0 with the same
> behaviour.
> I just tested it again (both with 5.3.2 and 5.7.0) and same thing, and on
> my case it doesn't matter if there are consumers or not, it always seems to
> make usage of the memory.
>
> Although I guess in theory that should not affect, did you use Stomp for
> your testing, or maybe you used Openwire?
> I'm using Stomp for my testing.
>
> Might be though that the broker's memory itself is not going beyond 70% of
> memoryUsage, but this is just per destination counters as you mentioned.
> In which case I guess the value shown as "Memory percent used" is a bit
> confusing... But haven't had much time to really test the possibility of
> exhausting the broker's memory.
>
No, i believe what you're seeing is correct. The broker's memory limit is
going beyond memoryUsage (way beyond). When a queue checks whether memory
is full, it will only do something interesting if producer flow control is
enabled. Otherwise, it will continue on. You are seeing that it will
continue to add messages until the Queue's memory limit (40MB) reaches the
70% mark. Since MemoryUsages are hierarchical, this means it will also
account for messages in the overall broker memory as well. For each queue,
you'll see that it will continue to hold 70% of 40MB of memory. What you
want in this case (if there are no consumers, or slow consumers) is to
raise your system usage memory limit OR lower your per-destination limits
OR lower your cursor highwatermark or a combination of all three.
http://activemq.apache.org/per-destination-policies.html
With PFC turned off, you're essentially telling the broker to take the
message no matter what. There is a point at which you will run out of
resources (memory, disk, etc). The trick is to find your use case and tune
for that.
>
> Will try to do some more testing soon...
>
> Thanks
>
>
> On Wed, Nov 21, 2012 at 2:28 PM, Christian Posta
> <ch...@gmail.com>wrote:
>
> > Can you please try on 5.7?
> > I just tried a test, and if there are no consumers to the queue then the
> > memory usage will stay at 0%. The message will not be retained, ie, it
> will
> > be put into the store and kept there. If I add a consumer, and not try to
> > consume, the message will be kept around in memory up to the cursor high
> > watermark (70 by default).
> >
> > As I add more queues the same behavior as described above will happen.
> If I
> > attach consumers to the queues without consuming them (so no messages are
> > consumed), then messages are kept in the cursor up to the high-water
> > mark... note.. the high-water mark is relative to the
> Destination/Cursor's
> > MemoryUsage, not the global memory usage.
> >
> > If I continue adding queues, and with producer flow control set to
> false, I
> > too will see the *Global* memory usage go much higher than 100%. This is
> > not surprising though, because as I understand, these usage memory
> objects
> > are really just counters. They don't enforce anything. When coupled with
> > producer flow control, they can be used to determine when to enable PFC.
> If
> > PFC is false, it's up to the cursor to determine when to flush out to
> disk.
> > But each destination/cursor will have it's own system usage (with the
> > global as the parent).
> >
> > Hope this helps. Can you please try with 5.7 and give us a report back?
> > Thanks,
> > Christian
> >
> >
> >
> > On Fri, Nov 16, 2012 at 11:38 AM, Juan Nin <ju...@gmail.com> wrote:
> >
> > > nope, adding a 3rd queue the 3rd one also gets this same value, so even
> > if
> > > it's the memory usage of the queue it's anyway going beyond..
> > >
> > >
> > > On Fri, Nov 16, 2012 at 4:32 PM, Juan Nin <ju...@gmail.com> wrote:
> > >
> > > > Might it be just a bug on how the MemoryPercentUsage is calculated?
> > > >
> > > > If I connect via JMX using console, I can see the MemoryPercentUsage
> as
> > > > 112 right now.
> > > > If I go to each of the 2 queues on them I see CursorMemoryUsage with
> > > value
> > > > 29360604, which would be 28mb each, summing a total of 56mb (just a
> bit
> > > > more than the specified memoryUsage of 50mb).
> > > >
> > > > Not sure I'm interpreting these values correctly though, first time I
> > > > access it via jconsole...
> > > >
> > > >
> > > > On Fri, Nov 16, 2012 at 4:07 PM, Juan Nin <ju...@gmail.com> wrote:
> > > >
> > > >> On that config there's a 40mb memoryLimit per queue, but also tested
> > it
> > > >> without it with same results.
> > > >>
> > > >>
> > > >> On Fri, Nov 16, 2012 at 4:05 PM, Juan Nin <ju...@gmail.com> wrote:
> > > >>
> > > >>> Hi Torsten!
> > > >>>
> > > >>> I'm using ActiveMQ 5.3.2, but also tested it on 5.7.0 with the same
> > > >>> results...
> > > >>> This is my 5.3.2 config:
> > > >>>
> > > >>> <beans
> > > >>> xmlns="http://www.springframework.org/schema/beans"
> > > >>> xmlns:amq="http://activemq.apache.org/schema/core"
> > > >>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
> > > >>> xsi:schemaLocation="http://www.springframework.org/schema/beans
> > > >>> http://www.springframework.org/schema/beans/spring-beans-2.0.xsd
> > > >>> http://activemq.apache.org/schema/core
> > > >>> http://activemq.apache.org/schema/core/activemq-core.xsd">
> > > >>>
> > > >>> <bean
> > > >>>
> > >
> >
> class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
> > > >>> <property name="locations">
> > > >>>
> > > >>> <value>file:${activemq.base}/conf/credentials.properties</value>
> > > >>> </property>
> > > >>> </bean>
> > > >>>
> > > >>> <broker xmlns="http://activemq.apache.org/schema/core"
> > > >>> brokerName="localhost" dataDirectory="${activemq.base}/data"
> > > >>> destroyApplicationContextOnStop="true" advisorySupport="false">
> > > >>>
> > > >>> <destinationPolicy>
> > > >>> <policyMap>
> > > >>> <policyEntries>
> > > >>> <policyEntry topic=">"
> producerFlowControl="true"
> > > >>> memoryLimit="5mb">
> > > >>> <pendingSubscriberPolicy>
> > > >>> <vmCursor />
> > > >>> </pendingSubscriberPolicy>
> > > >>> </policyEntry>
> > > >>> <policyEntry queue=">"
> > producerFlowControl="false"
> > > >>> optimizedDispatch="true" memoryLimit="40mb">
> > > >>> <deadLetterStrategy>
> > > >>> <individualDeadLetterStrategy
> > > >>> queuePrefix="DLQ." useQueueForQueueMessages="true" />
> > > >>> </deadLetterStrategy>
> > > >>> </policyEntry>
> > > >>> </policyEntries>
> > > >>> </policyMap>
> > > >>> </destinationPolicy>
> > > >>>
> > > >>> <managementContext>
> > > >>> <managementContext connectorPort="2011"/>
> > > >>> </managementContext>
> > > >>>
> > > >>> <persistenceAdapter>
> > > >>> <kahaDB directory="${activemq.base}/data/kahadb"
> > > >>> enableJournalDiskSyncs="false" indexWriteBatchSize="10000"
> > > >>> indexCacheSize="1000"/>
> > > >>> </persistenceAdapter>
> > > >>>
> > > >>> <systemUsage>
> > > >>> <systemUsage>
> > > >>> <memoryUsage>
> > > >>> <memoryUsage limit="50 mb"/>
> > > >>> </memoryUsage>
> > > >>> <storeUsage>
> > > >>> <storeUsage limit="1 gb" name="foo"/>
> > > >>> </storeUsage>
> > > >>> <tempUsage>
> > > >>> <tempUsage limit="3 gb"/>
> > > >>> </tempUsage>
> > > >>> </systemUsage>
> > > >>> </systemUsage>
> > > >>>
> > > >>> <transportConnectors>
> > > >>> <transportConnector name="openwire" uri="tcp://
> > > 0.0.0.0:61616
> > > >>> "/>
> > > >>> <transportConnector name="stomp" uri="stomp://0.0.0.0:61613"/>
> > > >>> </transportConnectors>
> > > >>>
> > > >>> </broker>
> > > >>>
> > > >>> <import resource="jetty.xml"/>
> > > >>>
> > > >>> </beans>
> > > >>>
> > > >>>
> > > >>> Using just a simple PHP script with Stomp for feeding the queues
> > > >>> (running it twice with different queue name):
> > > >>>
> > > >>> <?php
> > > >>>
> > > >>> require_once("Stomp.php");
> > > >>>
> > > >>> $amq = new Stomp("tcp://localhost:61613");
> > > >>> $amq->connect();
> > > >>>
> > > >>> for($i=1; $i <= 100000; $i++)
> > > >>> {
> > > >>> if($i%1000 == 0)
> > > >>> {
> > > >>> echo "\nmsg #: $i";
> > > >>> }
> > > >>> $amq->send("/queue/test", "this is test message # $i"
> > > >>> ,array('persistent' => 'true'));
> > > >>> }
> > > >>>
> > > >>> $amq->disconnect();
> > > >>>
> > > >>> ?>
> > > >>>
> > > >>>
> > > >>>
> > > >>> On Fri, Nov 16, 2012 at 3:47 PM, Torsten Mielke <
> > > torsten@fusesource.com>wrote:
> > > >>>
> > > >>>> Hello,
> > > >>>>
> > > >>>> See in-line response.
> > > >>>>
> > > >>>> On Nov 16, 2012, at 6:29 PM, Juan Nin wrote:
> > > >>>>
> > > >>>> > Hi!
> > > >>>> >
> > > >>>> > After some heavy digging about Producer Flow control and the
> > > >>>> systemUsage
> > > >>>> > properties a couple of years ago, I thought I quite understood
> it.
> > > >>>> > But yesterday I found that one of my configs was not behaving
> > > exactly
> > > >>>> as I
> > > >>>> > expected, so started doing some tests, and I see certain
> > behaviours
> > > >>>> which
> > > >>>> > don't seem to match what the docs and posts that I find on the
> > list
> > > or
> > > >>>> > other forums say.
> > > >>>> >
> > > >>>> > "storeUsage" is perfectly clear, it's the max space that
> > persistent
> > > >>>> > messages can use to be stored in disk.
> > > >>>> > "tempUsage"" applies to file cursors on non-persistent messages,
> > so
> > > >>>> as to
> > > >>>> > flush to disk if memory limits are reached (I don't care much
> > about
> > > >>>> this
> > > >>>> > one anyway, I always use persistent messages).
> > > >>>>
> > > >>>> Correct.
> > > >>>>
> > > >>>> >
> > > >>>> > Now, according to most posts, memoryUsage would be the maximum
> > > memory
> > > >>>> that
> > > >>>> > the broker would be available to use.
> > > >>>> > On this post:
> > > >>>> >
> > > >>>>
> > >
> >
> http://stackoverflow.com/questions/7646057/activemq-destinationpolicy-and-systemusage-configurationit
> > > >>>> > says that "memoryUsage corresponds to the amount of memory
> that's
> > > >>>> > assigned to the in-memory store".
> > > >>>>
> > > >>>> Correct.
> > > >>>>
> > > >>>> >
> > > >>>> > For example, on my tests using the following config (only
> showing
> > > >>>> relevant
> > > >>>> > parts):
> > > >>>> >
> > > >>>> > <policyEntry queue=">" producerFlowControl="false"
> > > >>>> optimizedDispatch="true">
> > > >>>> > <deadLetterStrategy>
> > > >>>> > <individualDeadLetterStrategy queuePrefix="DLQ."
> > > >>>> > useQueueForQueueMessages="true" />
> > > >>>> > </deadLetterStrategy>
> > > >>>> > </policyEntry>
> > > >>>> >
> > > >>>> > <systemUsage>
> > > >>>> > <systemUsage>
> > > >>>> > <memoryUsage>
> > > >>>> > <memoryUsage limit="100 mb"/>
> > > >>>> > </memoryUsage>
> > > >>>> > <storeUsage>
> > > >>>> > <storeUsage limit="1 gb" name="foo"/>
> > > >>>> > </storeUsage>
> > > >>>> > <tempUsage>
> > > >>>> > <tempUsage limit="3 gb"/>
> > > >>>> > </tempUsage>
> > > >>>> > </systemUsage>
> > > >>>> > </systemUsage>
> > > >>>> >
> > > >>>> > With that config I would expect the broker to use 100 mb of
> > maximum
> > > >>>> memory
> > > >>>> > among all queues. So it could maybe use 30mb in one queue and
> 70mb
> > > in
> > > >>>> > second queue.
> > > >>>> >
> > > >>>> >
> > > >>>> > 1) What I'm seeing is that if I start feeding a queue without
> > > >>>> consuming it,
> > > >>>> > the "Memory percent used" grows up to 70%, after that it doesn't
> > > grow
> > > >>>> > anymore.
> > > >>>> > What is it doing exactly there? The first 70% is stored in
> memory
> > > >>>> (apart
> > > >>>> > from disk since it's persistent), and all the rest that
> continues
> > > >>>> being fed
> > > >>>> > goes just to disk?
> > > >>>>
> > > >>>> This behavior is correct. For queues the default cursor is store
> > > >>>> cursor. It keeps any newly arrived msgs in memory as long as it
> does
> > > not
> > > >>>> reach the configured memory limit (either configured on the queue
> > per
> > > >>>> destination or globally in memoryUsage settings).
> > > >>>> Once the cursor reaches 70% of the configured limit (in your case
> of
> > > >>>> the memoryUsage limit since you don't specify a per-destination
> > > limit), it
> > > >>>> will not keep any more messages in memory.
> > > >>>> Instead it will reload these messages from the store when its time
> > to
> > > >>>> dispatch them. The broker anyway persists any msgs it receives
> > before
> > > >>>> passing on to the cursor.
> > > >>>> This limit of 70% can be configured and raised to e..g 100%.
> > > >>>> This behavior is kind of an optimization. That way you run less
> > often
> > > >>>> into producer-flow-control.
> > > >>>> As long as the persistence store is not running full, there is no
> > need
> > > >>>> to block producers, since the cursor can also load the messages
> from
> > > the
> > > >>>> store and does not necessarily have to keep them in memory.
> > > >>>> If you configure the vmQueueCursor, then the behavior is
> different.
> > > >>>> This cursor will not be able to load msgs to the store but needs
> to
> > > keep
> > > >>>> them all in memory. The vmQueueCursor used to be the default
> cursor
> > in
> > > >>>> older version of AMQ.
> > > >>>>
> > > >>>> Also note that topic msgs and non-persistent queue messages are
> not
> > > >>>> handled by the store cursor. These msgs are held in memory and if
> > > memory
> > > >>>> runs low, get swapped out to temp storage.
> > > >>>>
> > > >>>> > 2) If then I start feeding a 2nd queue, "Memory percent used"
> > > >>>> continues
> > > >>>> > growing until it reaches 140%. So it looks like memoryUsage does
> > not
> > > >>>> apply
> > > >>>> > globally, but on a per queue basis?
> > > >>>>
> > > >>>> What version of AMQ do you use? The sum of the memory usage of all
> > > >>>> queues should not go any higher than the configured memoryUsage
> > > limit. If
> > > >>>> you're not on 5.5.1 or higher releases, then I suggest to upgrade.
> > > >>>>
> > > >>>> > Using memoryLimit on the queue's policyEntry gives more control
> > over
> > > >>>> this,
> > > >>>> > but it's just a variation, "Memory percent used" can grow more
> > than
> > > >>>> 100%
> > > >>>> > anyway.
> > > >>>>
> > > >>>> With the default store cursor this should not be the case from
> what
> > I
> > > >>>> know.
> > > >>>>
> > > >>>>
> > > >>>> >
> > > >>>> > 3) If #2 is true, then how would I prevent the broker from
> running
> > > >>>> out of
> > > >>>> > memory in case queues would continue to be created?
> > > >>>>
> > > >>>> Just like above comment. I would expect the brokers
> > MemoryPercentUsage
> > > >>>> won't grow over 100% and the destinations MemoryPercentUsage
> remains
> > > fairly
> > > >>>> much at 70%.
> > > >>>> Not sure why you would see a different behavior? Using an old
> > version
> > > >>>> of AMQ perhaps? Or explicitly configuring for the vmQueueCursor?
> > > >>>> Could you perhaps also test with
> > > >>>>
> > > >>>> >
> > > >>>> >
> > > >>>> > Maybe I'm misunderstanding and some of these settings make no
> > sense
> > > >>>> when
> > > >>>> > producerFlowControl is disabled?
> > > >>>> >
> > > >>>> > Thanks in advance.
> > > >>>> >
> > > >>>> > Juan
> > > >>>>
> > > >>>>
> > > >>>> Regards,
> > > >>>>
> > > >>>> Torsten Mielke
> > > >>>> torsten@fusesource.com
> > > >>>> tmielke.blogspot.com
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>
> > > >>
> > > >
> > >
> >
> >
> >
> > --
> > *Christian Posta*
> > http://www.christianposta.com/blog
> > twitter: @christianposta
> >
>
--
*Christian Posta*
http://www.christianposta.com/blog
twitter: @christianposta
Re: Understanding memoryUsage (once again!)
Posted by Juan Nin <ju...@gmail.com>.
Hi!
Sorry for the delay in replying, buried on a project.
As I mentioned before, I had tested this with 5.7.0 with the same behaviour.
I just tested it again (both with 5.3.2 and 5.7.0) and same thing, and on
my case it doesn't matter if there are consumers or not, it always seems to
make usage of the memory.
Although I guess in theory that should not affect, did you use Stomp for
your testing, or maybe you used Openwire?
I'm using Stomp for my testing.
Might be though that the broker's memory itself is not going beyond 70% of
memoryUsage, but this is just per destination counters as you mentioned.
In which case I guess the value shown as "Memory percent used" is a bit
confusing... But haven't had much time to really test the possibility of
exhausting the broker's memory.
Will try to do some more testing soon...
Thanks
On Wed, Nov 21, 2012 at 2:28 PM, Christian Posta
<ch...@gmail.com>wrote:
> Can you please try on 5.7?
> I just tried a test, and if there are no consumers to the queue then the
> memory usage will stay at 0%. The message will not be retained, ie, it will
> be put into the store and kept there. If I add a consumer, and not try to
> consume, the message will be kept around in memory up to the cursor high
> watermark (70 by default).
>
> As I add more queues the same behavior as described above will happen. If I
> attach consumers to the queues without consuming them (so no messages are
> consumed), then messages are kept in the cursor up to the high-water
> mark... note.. the high-water mark is relative to the Destination/Cursor's
> MemoryUsage, not the global memory usage.
>
> If I continue adding queues, and with producer flow control set to false, I
> too will see the *Global* memory usage go much higher than 100%. This is
> not surprising though, because as I understand, these usage memory objects
> are really just counters. They don't enforce anything. When coupled with
> producer flow control, they can be used to determine when to enable PFC. If
> PFC is false, it's up to the cursor to determine when to flush out to disk.
> But each destination/cursor will have it's own system usage (with the
> global as the parent).
>
> Hope this helps. Can you please try with 5.7 and give us a report back?
> Thanks,
> Christian
>
>
>
> On Fri, Nov 16, 2012 at 11:38 AM, Juan Nin <ju...@gmail.com> wrote:
>
> > nope, adding a 3rd queue the 3rd one also gets this same value, so even
> if
> > it's the memory usage of the queue it's anyway going beyond..
> >
> >
> > On Fri, Nov 16, 2012 at 4:32 PM, Juan Nin <ju...@gmail.com> wrote:
> >
> > > Might it be just a bug on how the MemoryPercentUsage is calculated?
> > >
> > > If I connect via JMX using console, I can see the MemoryPercentUsage as
> > > 112 right now.
> > > If I go to each of the 2 queues on them I see CursorMemoryUsage with
> > value
> > > 29360604, which would be 28mb each, summing a total of 56mb (just a bit
> > > more than the specified memoryUsage of 50mb).
> > >
> > > Not sure I'm interpreting these values correctly though, first time I
> > > access it via jconsole...
> > >
> > >
> > > On Fri, Nov 16, 2012 at 4:07 PM, Juan Nin <ju...@gmail.com> wrote:
> > >
> > >> On that config there's a 40mb memoryLimit per queue, but also tested
> it
> > >> without it with same results.
> > >>
> > >>
> > >> On Fri, Nov 16, 2012 at 4:05 PM, Juan Nin <ju...@gmail.com> wrote:
> > >>
> > >>> Hi Torsten!
> > >>>
> > >>> I'm using ActiveMQ 5.3.2, but also tested it on 5.7.0 with the same
> > >>> results...
> > >>> This is my 5.3.2 config:
> > >>>
> > >>> <beans
> > >>> xmlns="http://www.springframework.org/schema/beans"
> > >>> xmlns:amq="http://activemq.apache.org/schema/core"
> > >>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
> > >>> xsi:schemaLocation="http://www.springframework.org/schema/beans
> > >>> http://www.springframework.org/schema/beans/spring-beans-2.0.xsd
> > >>> http://activemq.apache.org/schema/core
> > >>> http://activemq.apache.org/schema/core/activemq-core.xsd">
> > >>>
> > >>> <bean
> > >>>
> >
> class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
> > >>> <property name="locations">
> > >>>
> > >>> <value>file:${activemq.base}/conf/credentials.properties</value>
> > >>> </property>
> > >>> </bean>
> > >>>
> > >>> <broker xmlns="http://activemq.apache.org/schema/core"
> > >>> brokerName="localhost" dataDirectory="${activemq.base}/data"
> > >>> destroyApplicationContextOnStop="true" advisorySupport="false">
> > >>>
> > >>> <destinationPolicy>
> > >>> <policyMap>
> > >>> <policyEntries>
> > >>> <policyEntry topic=">" producerFlowControl="true"
> > >>> memoryLimit="5mb">
> > >>> <pendingSubscriberPolicy>
> > >>> <vmCursor />
> > >>> </pendingSubscriberPolicy>
> > >>> </policyEntry>
> > >>> <policyEntry queue=">"
> producerFlowControl="false"
> > >>> optimizedDispatch="true" memoryLimit="40mb">
> > >>> <deadLetterStrategy>
> > >>> <individualDeadLetterStrategy
> > >>> queuePrefix="DLQ." useQueueForQueueMessages="true" />
> > >>> </deadLetterStrategy>
> > >>> </policyEntry>
> > >>> </policyEntries>
> > >>> </policyMap>
> > >>> </destinationPolicy>
> > >>>
> > >>> <managementContext>
> > >>> <managementContext connectorPort="2011"/>
> > >>> </managementContext>
> > >>>
> > >>> <persistenceAdapter>
> > >>> <kahaDB directory="${activemq.base}/data/kahadb"
> > >>> enableJournalDiskSyncs="false" indexWriteBatchSize="10000"
> > >>> indexCacheSize="1000"/>
> > >>> </persistenceAdapter>
> > >>>
> > >>> <systemUsage>
> > >>> <systemUsage>
> > >>> <memoryUsage>
> > >>> <memoryUsage limit="50 mb"/>
> > >>> </memoryUsage>
> > >>> <storeUsage>
> > >>> <storeUsage limit="1 gb" name="foo"/>
> > >>> </storeUsage>
> > >>> <tempUsage>
> > >>> <tempUsage limit="3 gb"/>
> > >>> </tempUsage>
> > >>> </systemUsage>
> > >>> </systemUsage>
> > >>>
> > >>> <transportConnectors>
> > >>> <transportConnector name="openwire" uri="tcp://
> > 0.0.0.0:61616
> > >>> "/>
> > >>> <transportConnector name="stomp" uri="stomp://0.0.0.0:61613"/>
> > >>> </transportConnectors>
> > >>>
> > >>> </broker>
> > >>>
> > >>> <import resource="jetty.xml"/>
> > >>>
> > >>> </beans>
> > >>>
> > >>>
> > >>> Using just a simple PHP script with Stomp for feeding the queues
> > >>> (running it twice with different queue name):
> > >>>
> > >>> <?php
> > >>>
> > >>> require_once("Stomp.php");
> > >>>
> > >>> $amq = new Stomp("tcp://localhost:61613");
> > >>> $amq->connect();
> > >>>
> > >>> for($i=1; $i <= 100000; $i++)
> > >>> {
> > >>> if($i%1000 == 0)
> > >>> {
> > >>> echo "\nmsg #: $i";
> > >>> }
> > >>> $amq->send("/queue/test", "this is test message # $i"
> > >>> ,array('persistent' => 'true'));
> > >>> }
> > >>>
> > >>> $amq->disconnect();
> > >>>
> > >>> ?>
> > >>>
> > >>>
> > >>>
> > >>> On Fri, Nov 16, 2012 at 3:47 PM, Torsten Mielke <
> > torsten@fusesource.com>wrote:
> > >>>
> > >>>> Hello,
> > >>>>
> > >>>> See in-line response.
> > >>>>
> > >>>> On Nov 16, 2012, at 6:29 PM, Juan Nin wrote:
> > >>>>
> > >>>> > Hi!
> > >>>> >
> > >>>> > After some heavy digging about Producer Flow control and the
> > >>>> systemUsage
> > >>>> > properties a couple of years ago, I thought I quite understood it.
> > >>>> > But yesterday I found that one of my configs was not behaving
> > exactly
> > >>>> as I
> > >>>> > expected, so started doing some tests, and I see certain
> behaviours
> > >>>> which
> > >>>> > don't seem to match what the docs and posts that I find on the
> list
> > or
> > >>>> > other forums say.
> > >>>> >
> > >>>> > "storeUsage" is perfectly clear, it's the max space that
> persistent
> > >>>> > messages can use to be stored in disk.
> > >>>> > "tempUsage"" applies to file cursors on non-persistent messages,
> so
> > >>>> as to
> > >>>> > flush to disk if memory limits are reached (I don't care much
> about
> > >>>> this
> > >>>> > one anyway, I always use persistent messages).
> > >>>>
> > >>>> Correct.
> > >>>>
> > >>>> >
> > >>>> > Now, according to most posts, memoryUsage would be the maximum
> > memory
> > >>>> that
> > >>>> > the broker would be available to use.
> > >>>> > On this post:
> > >>>> >
> > >>>>
> >
> http://stackoverflow.com/questions/7646057/activemq-destinationpolicy-and-systemusage-configurationit
> > >>>> > says that "memoryUsage corresponds to the amount of memory that's
> > >>>> > assigned to the in-memory store".
> > >>>>
> > >>>> Correct.
> > >>>>
> > >>>> >
> > >>>> > For example, on my tests using the following config (only showing
> > >>>> relevant
> > >>>> > parts):
> > >>>> >
> > >>>> > <policyEntry queue=">" producerFlowControl="false"
> > >>>> optimizedDispatch="true">
> > >>>> > <deadLetterStrategy>
> > >>>> > <individualDeadLetterStrategy queuePrefix="DLQ."
> > >>>> > useQueueForQueueMessages="true" />
> > >>>> > </deadLetterStrategy>
> > >>>> > </policyEntry>
> > >>>> >
> > >>>> > <systemUsage>
> > >>>> > <systemUsage>
> > >>>> > <memoryUsage>
> > >>>> > <memoryUsage limit="100 mb"/>
> > >>>> > </memoryUsage>
> > >>>> > <storeUsage>
> > >>>> > <storeUsage limit="1 gb" name="foo"/>
> > >>>> > </storeUsage>
> > >>>> > <tempUsage>
> > >>>> > <tempUsage limit="3 gb"/>
> > >>>> > </tempUsage>
> > >>>> > </systemUsage>
> > >>>> > </systemUsage>
> > >>>> >
> > >>>> > With that config I would expect the broker to use 100 mb of
> maximum
> > >>>> memory
> > >>>> > among all queues. So it could maybe use 30mb in one queue and 70mb
> > in
> > >>>> > second queue.
> > >>>> >
> > >>>> >
> > >>>> > 1) What I'm seeing is that if I start feeding a queue without
> > >>>> consuming it,
> > >>>> > the "Memory percent used" grows up to 70%, after that it doesn't
> > grow
> > >>>> > anymore.
> > >>>> > What is it doing exactly there? The first 70% is stored in memory
> > >>>> (apart
> > >>>> > from disk since it's persistent), and all the rest that continues
> > >>>> being fed
> > >>>> > goes just to disk?
> > >>>>
> > >>>> This behavior is correct. For queues the default cursor is store
> > >>>> cursor. It keeps any newly arrived msgs in memory as long as it does
> > not
> > >>>> reach the configured memory limit (either configured on the queue
> per
> > >>>> destination or globally in memoryUsage settings).
> > >>>> Once the cursor reaches 70% of the configured limit (in your case of
> > >>>> the memoryUsage limit since you don't specify a per-destination
> > limit), it
> > >>>> will not keep any more messages in memory.
> > >>>> Instead it will reload these messages from the store when its time
> to
> > >>>> dispatch them. The broker anyway persists any msgs it receives
> before
> > >>>> passing on to the cursor.
> > >>>> This limit of 70% can be configured and raised to e..g 100%.
> > >>>> This behavior is kind of an optimization. That way you run less
> often
> > >>>> into producer-flow-control.
> > >>>> As long as the persistence store is not running full, there is no
> need
> > >>>> to block producers, since the cursor can also load the messages from
> > the
> > >>>> store and does not necessarily have to keep them in memory.
> > >>>> If you configure the vmQueueCursor, then the behavior is different.
> > >>>> This cursor will not be able to load msgs to the store but needs to
> > keep
> > >>>> them all in memory. The vmQueueCursor used to be the default cursor
> in
> > >>>> older version of AMQ.
> > >>>>
> > >>>> Also note that topic msgs and non-persistent queue messages are not
> > >>>> handled by the store cursor. These msgs are held in memory and if
> > memory
> > >>>> runs low, get swapped out to temp storage.
> > >>>>
> > >>>> > 2) If then I start feeding a 2nd queue, "Memory percent used"
> > >>>> continues
> > >>>> > growing until it reaches 140%. So it looks like memoryUsage does
> not
> > >>>> apply
> > >>>> > globally, but on a per queue basis?
> > >>>>
> > >>>> What version of AMQ do you use? The sum of the memory usage of all
> > >>>> queues should not go any higher than the configured memoryUsage
> > limit. If
> > >>>> you're not on 5.5.1 or higher releases, then I suggest to upgrade.
> > >>>>
> > >>>> > Using memoryLimit on the queue's policyEntry gives more control
> over
> > >>>> this,
> > >>>> > but it's just a variation, "Memory percent used" can grow more
> than
> > >>>> 100%
> > >>>> > anyway.
> > >>>>
> > >>>> With the default store cursor this should not be the case from what
> I
> > >>>> know.
> > >>>>
> > >>>>
> > >>>> >
> > >>>> > 3) If #2 is true, then how would I prevent the broker from running
> > >>>> out of
> > >>>> > memory in case queues would continue to be created?
> > >>>>
> > >>>> Just like above comment. I would expect the brokers
> MemoryPercentUsage
> > >>>> won't grow over 100% and the destinations MemoryPercentUsage remains
> > fairly
> > >>>> much at 70%.
> > >>>> Not sure why you would see a different behavior? Using an old
> version
> > >>>> of AMQ perhaps? Or explicitly configuring for the vmQueueCursor?
> > >>>> Could you perhaps also test with
> > >>>>
> > >>>> >
> > >>>> >
> > >>>> > Maybe I'm misunderstanding and some of these settings make no
> sense
> > >>>> when
> > >>>> > producerFlowControl is disabled?
> > >>>> >
> > >>>> > Thanks in advance.
> > >>>> >
> > >>>> > Juan
> > >>>>
> > >>>>
> > >>>> Regards,
> > >>>>
> > >>>> Torsten Mielke
> > >>>> torsten@fusesource.com
> > >>>> tmielke.blogspot.com
> > >>>>
> > >>>>
> > >>>>
> > >>>>
> > >>>>
> > >>>
> > >>
> > >
> >
>
>
>
> --
> *Christian Posta*
> http://www.christianposta.com/blog
> twitter: @christianposta
>
Re: Understanding memoryUsage (once again!)
Posted by Christian Posta <ch...@gmail.com>.
Can you please try on 5.7?
I just tried a test, and if there are no consumers to the queue then the
memory usage will stay at 0%. The message will not be retained, ie, it will
be put into the store and kept there. If I add a consumer, and not try to
consume, the message will be kept around in memory up to the cursor high
watermark (70 by default).
As I add more queues the same behavior as described above will happen. If I
attach consumers to the queues without consuming them (so no messages are
consumed), then messages are kept in the cursor up to the high-water
mark... note.. the high-water mark is relative to the Destination/Cursor's
MemoryUsage, not the global memory usage.
If I continue adding queues, and with producer flow control set to false, I
too will see the *Global* memory usage go much higher than 100%. This is
not surprising though, because as I understand, these usage memory objects
are really just counters. They don't enforce anything. When coupled with
producer flow control, they can be used to determine when to enable PFC. If
PFC is false, it's up to the cursor to determine when to flush out to disk.
But each destination/cursor will have it's own system usage (with the
global as the parent).
Hope this helps. Can you please try with 5.7 and give us a report back?
Thanks,
Christian
On Fri, Nov 16, 2012 at 11:38 AM, Juan Nin <ju...@gmail.com> wrote:
> nope, adding a 3rd queue the 3rd one also gets this same value, so even if
> it's the memory usage of the queue it's anyway going beyond..
>
>
> On Fri, Nov 16, 2012 at 4:32 PM, Juan Nin <ju...@gmail.com> wrote:
>
> > Might it be just a bug on how the MemoryPercentUsage is calculated?
> >
> > If I connect via JMX using console, I can see the MemoryPercentUsage as
> > 112 right now.
> > If I go to each of the 2 queues on them I see CursorMemoryUsage with
> value
> > 29360604, which would be 28mb each, summing a total of 56mb (just a bit
> > more than the specified memoryUsage of 50mb).
> >
> > Not sure I'm interpreting these values correctly though, first time I
> > access it via jconsole...
> >
> >
> > On Fri, Nov 16, 2012 at 4:07 PM, Juan Nin <ju...@gmail.com> wrote:
> >
> >> On that config there's a 40mb memoryLimit per queue, but also tested it
> >> without it with same results.
> >>
> >>
> >> On Fri, Nov 16, 2012 at 4:05 PM, Juan Nin <ju...@gmail.com> wrote:
> >>
> >>> Hi Torsten!
> >>>
> >>> I'm using ActiveMQ 5.3.2, but also tested it on 5.7.0 with the same
> >>> results...
> >>> This is my 5.3.2 config:
> >>>
> >>> <beans
> >>> xmlns="http://www.springframework.org/schema/beans"
> >>> xmlns:amq="http://activemq.apache.org/schema/core"
> >>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
> >>> xsi:schemaLocation="http://www.springframework.org/schema/beans
> >>> http://www.springframework.org/schema/beans/spring-beans-2.0.xsd
> >>> http://activemq.apache.org/schema/core
> >>> http://activemq.apache.org/schema/core/activemq-core.xsd">
> >>>
> >>> <bean
> >>>
> class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
> >>> <property name="locations">
> >>>
> >>> <value>file:${activemq.base}/conf/credentials.properties</value>
> >>> </property>
> >>> </bean>
> >>>
> >>> <broker xmlns="http://activemq.apache.org/schema/core"
> >>> brokerName="localhost" dataDirectory="${activemq.base}/data"
> >>> destroyApplicationContextOnStop="true" advisorySupport="false">
> >>>
> >>> <destinationPolicy>
> >>> <policyMap>
> >>> <policyEntries>
> >>> <policyEntry topic=">" producerFlowControl="true"
> >>> memoryLimit="5mb">
> >>> <pendingSubscriberPolicy>
> >>> <vmCursor />
> >>> </pendingSubscriberPolicy>
> >>> </policyEntry>
> >>> <policyEntry queue=">" producerFlowControl="false"
> >>> optimizedDispatch="true" memoryLimit="40mb">
> >>> <deadLetterStrategy>
> >>> <individualDeadLetterStrategy
> >>> queuePrefix="DLQ." useQueueForQueueMessages="true" />
> >>> </deadLetterStrategy>
> >>> </policyEntry>
> >>> </policyEntries>
> >>> </policyMap>
> >>> </destinationPolicy>
> >>>
> >>> <managementContext>
> >>> <managementContext connectorPort="2011"/>
> >>> </managementContext>
> >>>
> >>> <persistenceAdapter>
> >>> <kahaDB directory="${activemq.base}/data/kahadb"
> >>> enableJournalDiskSyncs="false" indexWriteBatchSize="10000"
> >>> indexCacheSize="1000"/>
> >>> </persistenceAdapter>
> >>>
> >>> <systemUsage>
> >>> <systemUsage>
> >>> <memoryUsage>
> >>> <memoryUsage limit="50 mb"/>
> >>> </memoryUsage>
> >>> <storeUsage>
> >>> <storeUsage limit="1 gb" name="foo"/>
> >>> </storeUsage>
> >>> <tempUsage>
> >>> <tempUsage limit="3 gb"/>
> >>> </tempUsage>
> >>> </systemUsage>
> >>> </systemUsage>
> >>>
> >>> <transportConnectors>
> >>> <transportConnector name="openwire" uri="tcp://
> 0.0.0.0:61616
> >>> "/>
> >>> <transportConnector name="stomp" uri="stomp://0.0.0.0:61613"/>
> >>> </transportConnectors>
> >>>
> >>> </broker>
> >>>
> >>> <import resource="jetty.xml"/>
> >>>
> >>> </beans>
> >>>
> >>>
> >>> Using just a simple PHP script with Stomp for feeding the queues
> >>> (running it twice with different queue name):
> >>>
> >>> <?php
> >>>
> >>> require_once("Stomp.php");
> >>>
> >>> $amq = new Stomp("tcp://localhost:61613");
> >>> $amq->connect();
> >>>
> >>> for($i=1; $i <= 100000; $i++)
> >>> {
> >>> if($i%1000 == 0)
> >>> {
> >>> echo "\nmsg #: $i";
> >>> }
> >>> $amq->send("/queue/test", "this is test message # $i"
> >>> ,array('persistent' => 'true'));
> >>> }
> >>>
> >>> $amq->disconnect();
> >>>
> >>> ?>
> >>>
> >>>
> >>>
> >>> On Fri, Nov 16, 2012 at 3:47 PM, Torsten Mielke <
> torsten@fusesource.com>wrote:
> >>>
> >>>> Hello,
> >>>>
> >>>> See in-line response.
> >>>>
> >>>> On Nov 16, 2012, at 6:29 PM, Juan Nin wrote:
> >>>>
> >>>> > Hi!
> >>>> >
> >>>> > After some heavy digging about Producer Flow control and the
> >>>> systemUsage
> >>>> > properties a couple of years ago, I thought I quite understood it.
> >>>> > But yesterday I found that one of my configs was not behaving
> exactly
> >>>> as I
> >>>> > expected, so started doing some tests, and I see certain behaviours
> >>>> which
> >>>> > don't seem to match what the docs and posts that I find on the list
> or
> >>>> > other forums say.
> >>>> >
> >>>> > "storeUsage" is perfectly clear, it's the max space that persistent
> >>>> > messages can use to be stored in disk.
> >>>> > "tempUsage"" applies to file cursors on non-persistent messages, so
> >>>> as to
> >>>> > flush to disk if memory limits are reached (I don't care much about
> >>>> this
> >>>> > one anyway, I always use persistent messages).
> >>>>
> >>>> Correct.
> >>>>
> >>>> >
> >>>> > Now, according to most posts, memoryUsage would be the maximum
> memory
> >>>> that
> >>>> > the broker would be available to use.
> >>>> > On this post:
> >>>> >
> >>>>
> http://stackoverflow.com/questions/7646057/activemq-destinationpolicy-and-systemusage-configurationit
> >>>> > says that "memoryUsage corresponds to the amount of memory that's
> >>>> > assigned to the in-memory store".
> >>>>
> >>>> Correct.
> >>>>
> >>>> >
> >>>> > For example, on my tests using the following config (only showing
> >>>> relevant
> >>>> > parts):
> >>>> >
> >>>> > <policyEntry queue=">" producerFlowControl="false"
> >>>> optimizedDispatch="true">
> >>>> > <deadLetterStrategy>
> >>>> > <individualDeadLetterStrategy queuePrefix="DLQ."
> >>>> > useQueueForQueueMessages="true" />
> >>>> > </deadLetterStrategy>
> >>>> > </policyEntry>
> >>>> >
> >>>> > <systemUsage>
> >>>> > <systemUsage>
> >>>> > <memoryUsage>
> >>>> > <memoryUsage limit="100 mb"/>
> >>>> > </memoryUsage>
> >>>> > <storeUsage>
> >>>> > <storeUsage limit="1 gb" name="foo"/>
> >>>> > </storeUsage>
> >>>> > <tempUsage>
> >>>> > <tempUsage limit="3 gb"/>
> >>>> > </tempUsage>
> >>>> > </systemUsage>
> >>>> > </systemUsage>
> >>>> >
> >>>> > With that config I would expect the broker to use 100 mb of maximum
> >>>> memory
> >>>> > among all queues. So it could maybe use 30mb in one queue and 70mb
> in
> >>>> > second queue.
> >>>> >
> >>>> >
> >>>> > 1) What I'm seeing is that if I start feeding a queue without
> >>>> consuming it,
> >>>> > the "Memory percent used" grows up to 70%, after that it doesn't
> grow
> >>>> > anymore.
> >>>> > What is it doing exactly there? The first 70% is stored in memory
> >>>> (apart
> >>>> > from disk since it's persistent), and all the rest that continues
> >>>> being fed
> >>>> > goes just to disk?
> >>>>
> >>>> This behavior is correct. For queues the default cursor is store
> >>>> cursor. It keeps any newly arrived msgs in memory as long as it does
> not
> >>>> reach the configured memory limit (either configured on the queue per
> >>>> destination or globally in memoryUsage settings).
> >>>> Once the cursor reaches 70% of the configured limit (in your case of
> >>>> the memoryUsage limit since you don't specify a per-destination
> limit), it
> >>>> will not keep any more messages in memory.
> >>>> Instead it will reload these messages from the store when its time to
> >>>> dispatch them. The broker anyway persists any msgs it receives before
> >>>> passing on to the cursor.
> >>>> This limit of 70% can be configured and raised to e..g 100%.
> >>>> This behavior is kind of an optimization. That way you run less often
> >>>> into producer-flow-control.
> >>>> As long as the persistence store is not running full, there is no need
> >>>> to block producers, since the cursor can also load the messages from
> the
> >>>> store and does not necessarily have to keep them in memory.
> >>>> If you configure the vmQueueCursor, then the behavior is different.
> >>>> This cursor will not be able to load msgs to the store but needs to
> keep
> >>>> them all in memory. The vmQueueCursor used to be the default cursor in
> >>>> older version of AMQ.
> >>>>
> >>>> Also note that topic msgs and non-persistent queue messages are not
> >>>> handled by the store cursor. These msgs are held in memory and if
> memory
> >>>> runs low, get swapped out to temp storage.
> >>>>
> >>>> > 2) If then I start feeding a 2nd queue, "Memory percent used"
> >>>> continues
> >>>> > growing until it reaches 140%. So it looks like memoryUsage does not
> >>>> apply
> >>>> > globally, but on a per queue basis?
> >>>>
> >>>> What version of AMQ do you use? The sum of the memory usage of all
> >>>> queues should not go any higher than the configured memoryUsage
> limit. If
> >>>> you're not on 5.5.1 or higher releases, then I suggest to upgrade.
> >>>>
> >>>> > Using memoryLimit on the queue's policyEntry gives more control over
> >>>> this,
> >>>> > but it's just a variation, "Memory percent used" can grow more than
> >>>> 100%
> >>>> > anyway.
> >>>>
> >>>> With the default store cursor this should not be the case from what I
> >>>> know.
> >>>>
> >>>>
> >>>> >
> >>>> > 3) If #2 is true, then how would I prevent the broker from running
> >>>> out of
> >>>> > memory in case queues would continue to be created?
> >>>>
> >>>> Just like above comment. I would expect the brokers MemoryPercentUsage
> >>>> won't grow over 100% and the destinations MemoryPercentUsage remains
> fairly
> >>>> much at 70%.
> >>>> Not sure why you would see a different behavior? Using an old version
> >>>> of AMQ perhaps? Or explicitly configuring for the vmQueueCursor?
> >>>> Could you perhaps also test with
> >>>>
> >>>> >
> >>>> >
> >>>> > Maybe I'm misunderstanding and some of these settings make no sense
> >>>> when
> >>>> > producerFlowControl is disabled?
> >>>> >
> >>>> > Thanks in advance.
> >>>> >
> >>>> > Juan
> >>>>
> >>>>
> >>>> Regards,
> >>>>
> >>>> Torsten Mielke
> >>>> torsten@fusesource.com
> >>>> tmielke.blogspot.com
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>
> >>
> >
>
--
*Christian Posta*
http://www.christianposta.com/blog
twitter: @christianposta
Re: Understanding memoryUsage (once again!)
Posted by Juan Nin <ju...@gmail.com>.
nope, adding a 3rd queue the 3rd one also gets this same value, so even if
it's the memory usage of the queue it's anyway going beyond..
On Fri, Nov 16, 2012 at 4:32 PM, Juan Nin <ju...@gmail.com> wrote:
> Might it be just a bug on how the MemoryPercentUsage is calculated?
>
> If I connect via JMX using console, I can see the MemoryPercentUsage as
> 112 right now.
> If I go to each of the 2 queues on them I see CursorMemoryUsage with value
> 29360604, which would be 28mb each, summing a total of 56mb (just a bit
> more than the specified memoryUsage of 50mb).
>
> Not sure I'm interpreting these values correctly though, first time I
> access it via jconsole...
>
>
> On Fri, Nov 16, 2012 at 4:07 PM, Juan Nin <ju...@gmail.com> wrote:
>
>> On that config there's a 40mb memoryLimit per queue, but also tested it
>> without it with same results.
>>
>>
>> On Fri, Nov 16, 2012 at 4:05 PM, Juan Nin <ju...@gmail.com> wrote:
>>
>>> Hi Torsten!
>>>
>>> I'm using ActiveMQ 5.3.2, but also tested it on 5.7.0 with the same
>>> results...
>>> This is my 5.3.2 config:
>>>
>>> <beans
>>> xmlns="http://www.springframework.org/schema/beans"
>>> xmlns:amq="http://activemq.apache.org/schema/core"
>>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
>>> xsi:schemaLocation="http://www.springframework.org/schema/beans
>>> http://www.springframework.org/schema/beans/spring-beans-2.0.xsd
>>> http://activemq.apache.org/schema/core
>>> http://activemq.apache.org/schema/core/activemq-core.xsd">
>>>
>>> <bean
>>> class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
>>> <property name="locations">
>>>
>>> <value>file:${activemq.base}/conf/credentials.properties</value>
>>> </property>
>>> </bean>
>>>
>>> <broker xmlns="http://activemq.apache.org/schema/core"
>>> brokerName="localhost" dataDirectory="${activemq.base}/data"
>>> destroyApplicationContextOnStop="true" advisorySupport="false">
>>>
>>> <destinationPolicy>
>>> <policyMap>
>>> <policyEntries>
>>> <policyEntry topic=">" producerFlowControl="true"
>>> memoryLimit="5mb">
>>> <pendingSubscriberPolicy>
>>> <vmCursor />
>>> </pendingSubscriberPolicy>
>>> </policyEntry>
>>> <policyEntry queue=">" producerFlowControl="false"
>>> optimizedDispatch="true" memoryLimit="40mb">
>>> <deadLetterStrategy>
>>> <individualDeadLetterStrategy
>>> queuePrefix="DLQ." useQueueForQueueMessages="true" />
>>> </deadLetterStrategy>
>>> </policyEntry>
>>> </policyEntries>
>>> </policyMap>
>>> </destinationPolicy>
>>>
>>> <managementContext>
>>> <managementContext connectorPort="2011"/>
>>> </managementContext>
>>>
>>> <persistenceAdapter>
>>> <kahaDB directory="${activemq.base}/data/kahadb"
>>> enableJournalDiskSyncs="false" indexWriteBatchSize="10000"
>>> indexCacheSize="1000"/>
>>> </persistenceAdapter>
>>>
>>> <systemUsage>
>>> <systemUsage>
>>> <memoryUsage>
>>> <memoryUsage limit="50 mb"/>
>>> </memoryUsage>
>>> <storeUsage>
>>> <storeUsage limit="1 gb" name="foo"/>
>>> </storeUsage>
>>> <tempUsage>
>>> <tempUsage limit="3 gb"/>
>>> </tempUsage>
>>> </systemUsage>
>>> </systemUsage>
>>>
>>> <transportConnectors>
>>> <transportConnector name="openwire" uri="tcp://0.0.0.0:61616
>>> "/>
>>> <transportConnector name="stomp" uri="stomp://0.0.0.0:61613"/>
>>> </transportConnectors>
>>>
>>> </broker>
>>>
>>> <import resource="jetty.xml"/>
>>>
>>> </beans>
>>>
>>>
>>> Using just a simple PHP script with Stomp for feeding the queues
>>> (running it twice with different queue name):
>>>
>>> <?php
>>>
>>> require_once("Stomp.php");
>>>
>>> $amq = new Stomp("tcp://localhost:61613");
>>> $amq->connect();
>>>
>>> for($i=1; $i <= 100000; $i++)
>>> {
>>> if($i%1000 == 0)
>>> {
>>> echo "\nmsg #: $i";
>>> }
>>> $amq->send("/queue/test", "this is test message # $i"
>>> ,array('persistent' => 'true'));
>>> }
>>>
>>> $amq->disconnect();
>>>
>>> ?>
>>>
>>>
>>>
>>> On Fri, Nov 16, 2012 at 3:47 PM, Torsten Mielke <to...@fusesource.com>wrote:
>>>
>>>> Hello,
>>>>
>>>> See in-line response.
>>>>
>>>> On Nov 16, 2012, at 6:29 PM, Juan Nin wrote:
>>>>
>>>> > Hi!
>>>> >
>>>> > After some heavy digging about Producer Flow control and the
>>>> systemUsage
>>>> > properties a couple of years ago, I thought I quite understood it.
>>>> > But yesterday I found that one of my configs was not behaving exactly
>>>> as I
>>>> > expected, so started doing some tests, and I see certain behaviours
>>>> which
>>>> > don't seem to match what the docs and posts that I find on the list or
>>>> > other forums say.
>>>> >
>>>> > "storeUsage" is perfectly clear, it's the max space that persistent
>>>> > messages can use to be stored in disk.
>>>> > "tempUsage"" applies to file cursors on non-persistent messages, so
>>>> as to
>>>> > flush to disk if memory limits are reached (I don't care much about
>>>> this
>>>> > one anyway, I always use persistent messages).
>>>>
>>>> Correct.
>>>>
>>>> >
>>>> > Now, according to most posts, memoryUsage would be the maximum memory
>>>> that
>>>> > the broker would be available to use.
>>>> > On this post:
>>>> >
>>>> http://stackoverflow.com/questions/7646057/activemq-destinationpolicy-and-systemusage-configurationit
>>>> > says that "memoryUsage corresponds to the amount of memory that's
>>>> > assigned to the in-memory store".
>>>>
>>>> Correct.
>>>>
>>>> >
>>>> > For example, on my tests using the following config (only showing
>>>> relevant
>>>> > parts):
>>>> >
>>>> > <policyEntry queue=">" producerFlowControl="false"
>>>> optimizedDispatch="true">
>>>> > <deadLetterStrategy>
>>>> > <individualDeadLetterStrategy queuePrefix="DLQ."
>>>> > useQueueForQueueMessages="true" />
>>>> > </deadLetterStrategy>
>>>> > </policyEntry>
>>>> >
>>>> > <systemUsage>
>>>> > <systemUsage>
>>>> > <memoryUsage>
>>>> > <memoryUsage limit="100 mb"/>
>>>> > </memoryUsage>
>>>> > <storeUsage>
>>>> > <storeUsage limit="1 gb" name="foo"/>
>>>> > </storeUsage>
>>>> > <tempUsage>
>>>> > <tempUsage limit="3 gb"/>
>>>> > </tempUsage>
>>>> > </systemUsage>
>>>> > </systemUsage>
>>>> >
>>>> > With that config I would expect the broker to use 100 mb of maximum
>>>> memory
>>>> > among all queues. So it could maybe use 30mb in one queue and 70mb in
>>>> > second queue.
>>>> >
>>>> >
>>>> > 1) What I'm seeing is that if I start feeding a queue without
>>>> consuming it,
>>>> > the "Memory percent used" grows up to 70%, after that it doesn't grow
>>>> > anymore.
>>>> > What is it doing exactly there? The first 70% is stored in memory
>>>> (apart
>>>> > from disk since it's persistent), and all the rest that continues
>>>> being fed
>>>> > goes just to disk?
>>>>
>>>> This behavior is correct. For queues the default cursor is store
>>>> cursor. It keeps any newly arrived msgs in memory as long as it does not
>>>> reach the configured memory limit (either configured on the queue per
>>>> destination or globally in memoryUsage settings).
>>>> Once the cursor reaches 70% of the configured limit (in your case of
>>>> the memoryUsage limit since you don't specify a per-destination limit), it
>>>> will not keep any more messages in memory.
>>>> Instead it will reload these messages from the store when its time to
>>>> dispatch them. The broker anyway persists any msgs it receives before
>>>> passing on to the cursor.
>>>> This limit of 70% can be configured and raised to e..g 100%.
>>>> This behavior is kind of an optimization. That way you run less often
>>>> into producer-flow-control.
>>>> As long as the persistence store is not running full, there is no need
>>>> to block producers, since the cursor can also load the messages from the
>>>> store and does not necessarily have to keep them in memory.
>>>> If you configure the vmQueueCursor, then the behavior is different.
>>>> This cursor will not be able to load msgs to the store but needs to keep
>>>> them all in memory. The vmQueueCursor used to be the default cursor in
>>>> older version of AMQ.
>>>>
>>>> Also note that topic msgs and non-persistent queue messages are not
>>>> handled by the store cursor. These msgs are held in memory and if memory
>>>> runs low, get swapped out to temp storage.
>>>>
>>>> > 2) If then I start feeding a 2nd queue, "Memory percent used"
>>>> continues
>>>> > growing until it reaches 140%. So it looks like memoryUsage does not
>>>> apply
>>>> > globally, but on a per queue basis?
>>>>
>>>> What version of AMQ do you use? The sum of the memory usage of all
>>>> queues should not go any higher than the configured memoryUsage limit. If
>>>> you're not on 5.5.1 or higher releases, then I suggest to upgrade.
>>>>
>>>> > Using memoryLimit on the queue's policyEntry gives more control over
>>>> this,
>>>> > but it's just a variation, "Memory percent used" can grow more than
>>>> 100%
>>>> > anyway.
>>>>
>>>> With the default store cursor this should not be the case from what I
>>>> know.
>>>>
>>>>
>>>> >
>>>> > 3) If #2 is true, then how would I prevent the broker from running
>>>> out of
>>>> > memory in case queues would continue to be created?
>>>>
>>>> Just like above comment. I would expect the brokers MemoryPercentUsage
>>>> won't grow over 100% and the destinations MemoryPercentUsage remains fairly
>>>> much at 70%.
>>>> Not sure why you would see a different behavior? Using an old version
>>>> of AMQ perhaps? Or explicitly configuring for the vmQueueCursor?
>>>> Could you perhaps also test with
>>>>
>>>> >
>>>> >
>>>> > Maybe I'm misunderstanding and some of these settings make no sense
>>>> when
>>>> > producerFlowControl is disabled?
>>>> >
>>>> > Thanks in advance.
>>>> >
>>>> > Juan
>>>>
>>>>
>>>> Regards,
>>>>
>>>> Torsten Mielke
>>>> torsten@fusesource.com
>>>> tmielke.blogspot.com
>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>
>
Re: Understanding memoryUsage (once again!)
Posted by Juan Nin <ju...@gmail.com>.
Might it be just a bug on how the MemoryPercentUsage is calculated?
If I connect via JMX using console, I can see the MemoryPercentUsage as 112
right now.
If I go to each of the 2 queues on them I see CursorMemoryUsage with value
29360604, which would be 28mb each, summing a total of 56mb (just a bit
more than the specified memoryUsage of 50mb).
Not sure I'm interpreting these values correctly though, first time I
access it via jconsole...
On Fri, Nov 16, 2012 at 4:07 PM, Juan Nin <ju...@gmail.com> wrote:
> On that config there's a 40mb memoryLimit per queue, but also tested it
> without it with same results.
>
>
> On Fri, Nov 16, 2012 at 4:05 PM, Juan Nin <ju...@gmail.com> wrote:
>
>> Hi Torsten!
>>
>> I'm using ActiveMQ 5.3.2, but also tested it on 5.7.0 with the same
>> results...
>> This is my 5.3.2 config:
>>
>> <beans
>> xmlns="http://www.springframework.org/schema/beans"
>> xmlns:amq="http://activemq.apache.org/schema/core"
>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
>> xsi:schemaLocation="http://www.springframework.org/schema/beans
>> http://www.springframework.org/schema/beans/spring-beans-2.0.xsd
>> http://activemq.apache.org/schema/core
>> http://activemq.apache.org/schema/core/activemq-core.xsd">
>>
>> <bean
>> class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
>> <property name="locations">
>>
>> <value>file:${activemq.base}/conf/credentials.properties</value>
>> </property>
>> </bean>
>>
>> <broker xmlns="http://activemq.apache.org/schema/core"
>> brokerName="localhost" dataDirectory="${activemq.base}/data"
>> destroyApplicationContextOnStop="true" advisorySupport="false">
>>
>> <destinationPolicy>
>> <policyMap>
>> <policyEntries>
>> <policyEntry topic=">" producerFlowControl="true"
>> memoryLimit="5mb">
>> <pendingSubscriberPolicy>
>> <vmCursor />
>> </pendingSubscriberPolicy>
>> </policyEntry>
>> <policyEntry queue=">" producerFlowControl="false"
>> optimizedDispatch="true" memoryLimit="40mb">
>> <deadLetterStrategy>
>> <individualDeadLetterStrategy
>> queuePrefix="DLQ." useQueueForQueueMessages="true" />
>> </deadLetterStrategy>
>> </policyEntry>
>> </policyEntries>
>> </policyMap>
>> </destinationPolicy>
>>
>> <managementContext>
>> <managementContext connectorPort="2011"/>
>> </managementContext>
>>
>> <persistenceAdapter>
>> <kahaDB directory="${activemq.base}/data/kahadb"
>> enableJournalDiskSyncs="false" indexWriteBatchSize="10000"
>> indexCacheSize="1000"/>
>> </persistenceAdapter>
>>
>> <systemUsage>
>> <systemUsage>
>> <memoryUsage>
>> <memoryUsage limit="50 mb"/>
>> </memoryUsage>
>> <storeUsage>
>> <storeUsage limit="1 gb" name="foo"/>
>> </storeUsage>
>> <tempUsage>
>> <tempUsage limit="3 gb"/>
>> </tempUsage>
>> </systemUsage>
>> </systemUsage>
>>
>> <transportConnectors>
>> <transportConnector name="openwire" uri="tcp://0.0.0.0:61616
>> "/>
>> <transportConnector name="stomp" uri="stomp://0.0.0.0:61613"/>
>> </transportConnectors>
>>
>> </broker>
>>
>> <import resource="jetty.xml"/>
>>
>> </beans>
>>
>>
>> Using just a simple PHP script with Stomp for feeding the queues (running
>> it twice with different queue name):
>>
>> <?php
>>
>> require_once("Stomp.php");
>>
>> $amq = new Stomp("tcp://localhost:61613");
>> $amq->connect();
>>
>> for($i=1; $i <= 100000; $i++)
>> {
>> if($i%1000 == 0)
>> {
>> echo "\nmsg #: $i";
>> }
>> $amq->send("/queue/test", "this is test message # $i" ,array('persistent'
>> => 'true'));
>> }
>>
>> $amq->disconnect();
>>
>> ?>
>>
>>
>>
>> On Fri, Nov 16, 2012 at 3:47 PM, Torsten Mielke <to...@fusesource.com>wrote:
>>
>>> Hello,
>>>
>>> See in-line response.
>>>
>>> On Nov 16, 2012, at 6:29 PM, Juan Nin wrote:
>>>
>>> > Hi!
>>> >
>>> > After some heavy digging about Producer Flow control and the
>>> systemUsage
>>> > properties a couple of years ago, I thought I quite understood it.
>>> > But yesterday I found that one of my configs was not behaving exactly
>>> as I
>>> > expected, so started doing some tests, and I see certain behaviours
>>> which
>>> > don't seem to match what the docs and posts that I find on the list or
>>> > other forums say.
>>> >
>>> > "storeUsage" is perfectly clear, it's the max space that persistent
>>> > messages can use to be stored in disk.
>>> > "tempUsage"" applies to file cursors on non-persistent messages, so as
>>> to
>>> > flush to disk if memory limits are reached (I don't care much about
>>> this
>>> > one anyway, I always use persistent messages).
>>>
>>> Correct.
>>>
>>> >
>>> > Now, according to most posts, memoryUsage would be the maximum memory
>>> that
>>> > the broker would be available to use.
>>> > On this post:
>>> >
>>> http://stackoverflow.com/questions/7646057/activemq-destinationpolicy-and-systemusage-configurationit
>>> > says that "memoryUsage corresponds to the amount of memory that's
>>> > assigned to the in-memory store".
>>>
>>> Correct.
>>>
>>> >
>>> > For example, on my tests using the following config (only showing
>>> relevant
>>> > parts):
>>> >
>>> > <policyEntry queue=">" producerFlowControl="false"
>>> optimizedDispatch="true">
>>> > <deadLetterStrategy>
>>> > <individualDeadLetterStrategy queuePrefix="DLQ."
>>> > useQueueForQueueMessages="true" />
>>> > </deadLetterStrategy>
>>> > </policyEntry>
>>> >
>>> > <systemUsage>
>>> > <systemUsage>
>>> > <memoryUsage>
>>> > <memoryUsage limit="100 mb"/>
>>> > </memoryUsage>
>>> > <storeUsage>
>>> > <storeUsage limit="1 gb" name="foo"/>
>>> > </storeUsage>
>>> > <tempUsage>
>>> > <tempUsage limit="3 gb"/>
>>> > </tempUsage>
>>> > </systemUsage>
>>> > </systemUsage>
>>> >
>>> > With that config I would expect the broker to use 100 mb of maximum
>>> memory
>>> > among all queues. So it could maybe use 30mb in one queue and 70mb in
>>> > second queue.
>>> >
>>> >
>>> > 1) What I'm seeing is that if I start feeding a queue without
>>> consuming it,
>>> > the "Memory percent used" grows up to 70%, after that it doesn't grow
>>> > anymore.
>>> > What is it doing exactly there? The first 70% is stored in memory
>>> (apart
>>> > from disk since it's persistent), and all the rest that continues
>>> being fed
>>> > goes just to disk?
>>>
>>> This behavior is correct. For queues the default cursor is store cursor.
>>> It keeps any newly arrived msgs in memory as long as it does not reach the
>>> configured memory limit (either configured on the queue per destination or
>>> globally in memoryUsage settings).
>>> Once the cursor reaches 70% of the configured limit (in your case of the
>>> memoryUsage limit since you don't specify a per-destination limit), it will
>>> not keep any more messages in memory.
>>> Instead it will reload these messages from the store when its time to
>>> dispatch them. The broker anyway persists any msgs it receives before
>>> passing on to the cursor.
>>> This limit of 70% can be configured and raised to e..g 100%.
>>> This behavior is kind of an optimization. That way you run less often
>>> into producer-flow-control.
>>> As long as the persistence store is not running full, there is no need
>>> to block producers, since the cursor can also load the messages from the
>>> store and does not necessarily have to keep them in memory.
>>> If you configure the vmQueueCursor, then the behavior is different. This
>>> cursor will not be able to load msgs to the store but needs to keep them
>>> all in memory. The vmQueueCursor used to be the default cursor in older
>>> version of AMQ.
>>>
>>> Also note that topic msgs and non-persistent queue messages are not
>>> handled by the store cursor. These msgs are held in memory and if memory
>>> runs low, get swapped out to temp storage.
>>>
>>> > 2) If then I start feeding a 2nd queue, "Memory percent used" continues
>>> > growing until it reaches 140%. So it looks like memoryUsage does not
>>> apply
>>> > globally, but on a per queue basis?
>>>
>>> What version of AMQ do you use? The sum of the memory usage of all
>>> queues should not go any higher than the configured memoryUsage limit. If
>>> you're not on 5.5.1 or higher releases, then I suggest to upgrade.
>>>
>>> > Using memoryLimit on the queue's policyEntry gives more control over
>>> this,
>>> > but it's just a variation, "Memory percent used" can grow more than
>>> 100%
>>> > anyway.
>>>
>>> With the default store cursor this should not be the case from what I
>>> know.
>>>
>>>
>>> >
>>> > 3) If #2 is true, then how would I prevent the broker from running out
>>> of
>>> > memory in case queues would continue to be created?
>>>
>>> Just like above comment. I would expect the brokers MemoryPercentUsage
>>> won't grow over 100% and the destinations MemoryPercentUsage remains fairly
>>> much at 70%.
>>> Not sure why you would see a different behavior? Using an old version of
>>> AMQ perhaps? Or explicitly configuring for the vmQueueCursor?
>>> Could you perhaps also test with
>>>
>>> >
>>> >
>>> > Maybe I'm misunderstanding and some of these settings make no sense
>>> when
>>> > producerFlowControl is disabled?
>>> >
>>> > Thanks in advance.
>>> >
>>> > Juan
>>>
>>>
>>> Regards,
>>>
>>> Torsten Mielke
>>> torsten@fusesource.com
>>> tmielke.blogspot.com
>>>
>>>
>>>
>>>
>>>
>>
>
Re: Understanding memoryUsage (once again!)
Posted by Juan Nin <ju...@gmail.com>.
On that config there's a 40mb memoryLimit per queue, but also tested it
without it with same results.
On Fri, Nov 16, 2012 at 4:05 PM, Juan Nin <ju...@gmail.com> wrote:
> Hi Torsten!
>
> I'm using ActiveMQ 5.3.2, but also tested it on 5.7.0 with the same
> results...
> This is my 5.3.2 config:
>
> <beans
> xmlns="http://www.springframework.org/schema/beans"
> xmlns:amq="http://activemq.apache.org/schema/core"
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
> xsi:schemaLocation="http://www.springframework.org/schema/beans
> http://www.springframework.org/schema/beans/spring-beans-2.0.xsd
> http://activemq.apache.org/schema/core
> http://activemq.apache.org/schema/core/activemq-core.xsd">
>
> <bean
> class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
> <property name="locations">
>
> <value>file:${activemq.base}/conf/credentials.properties</value>
> </property>
> </bean>
>
> <broker xmlns="http://activemq.apache.org/schema/core"
> brokerName="localhost" dataDirectory="${activemq.base}/data"
> destroyApplicationContextOnStop="true" advisorySupport="false">
>
> <destinationPolicy>
> <policyMap>
> <policyEntries>
> <policyEntry topic=">" producerFlowControl="true"
> memoryLimit="5mb">
> <pendingSubscriberPolicy>
> <vmCursor />
> </pendingSubscriberPolicy>
> </policyEntry>
> <policyEntry queue=">" producerFlowControl="false"
> optimizedDispatch="true" memoryLimit="40mb">
> <deadLetterStrategy>
> <individualDeadLetterStrategy
> queuePrefix="DLQ." useQueueForQueueMessages="true" />
> </deadLetterStrategy>
> </policyEntry>
> </policyEntries>
> </policyMap>
> </destinationPolicy>
>
> <managementContext>
> <managementContext connectorPort="2011"/>
> </managementContext>
>
> <persistenceAdapter>
> <kahaDB directory="${activemq.base}/data/kahadb"
> enableJournalDiskSyncs="false" indexWriteBatchSize="10000"
> indexCacheSize="1000"/>
> </persistenceAdapter>
>
> <systemUsage>
> <systemUsage>
> <memoryUsage>
> <memoryUsage limit="50 mb"/>
> </memoryUsage>
> <storeUsage>
> <storeUsage limit="1 gb" name="foo"/>
> </storeUsage>
> <tempUsage>
> <tempUsage limit="3 gb"/>
> </tempUsage>
> </systemUsage>
> </systemUsage>
>
> <transportConnectors>
> <transportConnector name="openwire" uri="tcp://0.0.0.0:61616
> "/>
> <transportConnector name="stomp" uri="stomp://0.0.0.0:61613"/>
> </transportConnectors>
>
> </broker>
>
> <import resource="jetty.xml"/>
>
> </beans>
>
>
> Using just a simple PHP script with Stomp for feeding the queues (running
> it twice with different queue name):
>
> <?php
>
> require_once("Stomp.php");
>
> $amq = new Stomp("tcp://localhost:61613");
> $amq->connect();
>
> for($i=1; $i <= 100000; $i++)
> {
> if($i%1000 == 0)
> {
> echo "\nmsg #: $i";
> }
> $amq->send("/queue/test", "this is test message # $i" ,array('persistent'
> => 'true'));
> }
>
> $amq->disconnect();
>
> ?>
>
>
>
> On Fri, Nov 16, 2012 at 3:47 PM, Torsten Mielke <to...@fusesource.com>wrote:
>
>> Hello,
>>
>> See in-line response.
>>
>> On Nov 16, 2012, at 6:29 PM, Juan Nin wrote:
>>
>> > Hi!
>> >
>> > After some heavy digging about Producer Flow control and the systemUsage
>> > properties a couple of years ago, I thought I quite understood it.
>> > But yesterday I found that one of my configs was not behaving exactly
>> as I
>> > expected, so started doing some tests, and I see certain behaviours
>> which
>> > don't seem to match what the docs and posts that I find on the list or
>> > other forums say.
>> >
>> > "storeUsage" is perfectly clear, it's the max space that persistent
>> > messages can use to be stored in disk.
>> > "tempUsage"" applies to file cursors on non-persistent messages, so as
>> to
>> > flush to disk if memory limits are reached (I don't care much about this
>> > one anyway, I always use persistent messages).
>>
>> Correct.
>>
>> >
>> > Now, according to most posts, memoryUsage would be the maximum memory
>> that
>> > the broker would be available to use.
>> > On this post:
>> >
>> http://stackoverflow.com/questions/7646057/activemq-destinationpolicy-and-systemusage-configurationit
>> > says that "memoryUsage corresponds to the amount of memory that's
>> > assigned to the in-memory store".
>>
>> Correct.
>>
>> >
>> > For example, on my tests using the following config (only showing
>> relevant
>> > parts):
>> >
>> > <policyEntry queue=">" producerFlowControl="false"
>> optimizedDispatch="true">
>> > <deadLetterStrategy>
>> > <individualDeadLetterStrategy queuePrefix="DLQ."
>> > useQueueForQueueMessages="true" />
>> > </deadLetterStrategy>
>> > </policyEntry>
>> >
>> > <systemUsage>
>> > <systemUsage>
>> > <memoryUsage>
>> > <memoryUsage limit="100 mb"/>
>> > </memoryUsage>
>> > <storeUsage>
>> > <storeUsage limit="1 gb" name="foo"/>
>> > </storeUsage>
>> > <tempUsage>
>> > <tempUsage limit="3 gb"/>
>> > </tempUsage>
>> > </systemUsage>
>> > </systemUsage>
>> >
>> > With that config I would expect the broker to use 100 mb of maximum
>> memory
>> > among all queues. So it could maybe use 30mb in one queue and 70mb in
>> > second queue.
>> >
>> >
>> > 1) What I'm seeing is that if I start feeding a queue without consuming
>> it,
>> > the "Memory percent used" grows up to 70%, after that it doesn't grow
>> > anymore.
>> > What is it doing exactly there? The first 70% is stored in memory (apart
>> > from disk since it's persistent), and all the rest that continues being
>> fed
>> > goes just to disk?
>>
>> This behavior is correct. For queues the default cursor is store cursor.
>> It keeps any newly arrived msgs in memory as long as it does not reach the
>> configured memory limit (either configured on the queue per destination or
>> globally in memoryUsage settings).
>> Once the cursor reaches 70% of the configured limit (in your case of the
>> memoryUsage limit since you don't specify a per-destination limit), it will
>> not keep any more messages in memory.
>> Instead it will reload these messages from the store when its time to
>> dispatch them. The broker anyway persists any msgs it receives before
>> passing on to the cursor.
>> This limit of 70% can be configured and raised to e..g 100%.
>> This behavior is kind of an optimization. That way you run less often
>> into producer-flow-control.
>> As long as the persistence store is not running full, there is no need to
>> block producers, since the cursor can also load the messages from the store
>> and does not necessarily have to keep them in memory.
>> If you configure the vmQueueCursor, then the behavior is different. This
>> cursor will not be able to load msgs to the store but needs to keep them
>> all in memory. The vmQueueCursor used to be the default cursor in older
>> version of AMQ.
>>
>> Also note that topic msgs and non-persistent queue messages are not
>> handled by the store cursor. These msgs are held in memory and if memory
>> runs low, get swapped out to temp storage.
>>
>> > 2) If then I start feeding a 2nd queue, "Memory percent used" continues
>> > growing until it reaches 140%. So it looks like memoryUsage does not
>> apply
>> > globally, but on a per queue basis?
>>
>> What version of AMQ do you use? The sum of the memory usage of all queues
>> should not go any higher than the configured memoryUsage limit. If you're
>> not on 5.5.1 or higher releases, then I suggest to upgrade.
>>
>> > Using memoryLimit on the queue's policyEntry gives more control over
>> this,
>> > but it's just a variation, "Memory percent used" can grow more than 100%
>> > anyway.
>>
>> With the default store cursor this should not be the case from what I
>> know.
>>
>>
>> >
>> > 3) If #2 is true, then how would I prevent the broker from running out
>> of
>> > memory in case queues would continue to be created?
>>
>> Just like above comment. I would expect the brokers MemoryPercentUsage
>> won't grow over 100% and the destinations MemoryPercentUsage remains fairly
>> much at 70%.
>> Not sure why you would see a different behavior? Using an old version of
>> AMQ perhaps? Or explicitly configuring for the vmQueueCursor?
>> Could you perhaps also test with
>>
>> >
>> >
>> > Maybe I'm misunderstanding and some of these settings make no sense when
>> > producerFlowControl is disabled?
>> >
>> > Thanks in advance.
>> >
>> > Juan
>>
>>
>> Regards,
>>
>> Torsten Mielke
>> torsten@fusesource.com
>> tmielke.blogspot.com
>>
>>
>>
>>
>>
>
Re: Understanding memoryUsage (once again!)
Posted by Juan Nin <ju...@gmail.com>.
Hi Torsten!
I'm using ActiveMQ 5.3.2, but also tested it on 5.7.0 with the same
results...
This is my 5.3.2 config:
<beans
xmlns="http://www.springframework.org/schema/beans"
xmlns:amq="http://activemq.apache.org/schema/core"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans-2.0.xsd
http://activemq.apache.org/schema/core
http://activemq.apache.org/schema/core/activemq-core.xsd">
<bean
class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
<property name="locations">
<value>file:${activemq.base}/conf/credentials.properties</value>
</property>
</bean>
<broker xmlns="http://activemq.apache.org/schema/core"
brokerName="localhost" dataDirectory="${activemq.base}/data"
destroyApplicationContextOnStop="true" advisorySupport="false">
<destinationPolicy>
<policyMap>
<policyEntries>
<policyEntry topic=">" producerFlowControl="true"
memoryLimit="5mb">
<pendingSubscriberPolicy>
<vmCursor />
</pendingSubscriberPolicy>
</policyEntry>
<policyEntry queue=">" producerFlowControl="false"
optimizedDispatch="true" memoryLimit="40mb">
<deadLetterStrategy>
<individualDeadLetterStrategy
queuePrefix="DLQ." useQueueForQueueMessages="true" />
</deadLetterStrategy>
</policyEntry>
</policyEntries>
</policyMap>
</destinationPolicy>
<managementContext>
<managementContext connectorPort="2011"/>
</managementContext>
<persistenceAdapter>
<kahaDB directory="${activemq.base}/data/kahadb"
enableJournalDiskSyncs="false" indexWriteBatchSize="10000"
indexCacheSize="1000"/>
</persistenceAdapter>
<systemUsage>
<systemUsage>
<memoryUsage>
<memoryUsage limit="50 mb"/>
</memoryUsage>
<storeUsage>
<storeUsage limit="1 gb" name="foo"/>
</storeUsage>
<tempUsage>
<tempUsage limit="3 gb"/>
</tempUsage>
</systemUsage>
</systemUsage>
<transportConnectors>
<transportConnector name="openwire" uri="tcp://0.0.0.0:61616"/>
<transportConnector name="stomp" uri="stomp://0.0.0.0:61613"/>
</transportConnectors>
</broker>
<import resource="jetty.xml"/>
</beans>
Using just a simple PHP script with Stomp for feeding the queues (running
it twice with different queue name):
<?php
require_once("Stomp.php");
$amq = new Stomp("tcp://localhost:61613");
$amq->connect();
for($i=1; $i <= 100000; $i++)
{
if($i%1000 == 0)
{
echo "\nmsg #: $i";
}
$amq->send("/queue/test", "this is test message # $i" ,array('persistent'
=> 'true'));
}
$amq->disconnect();
?>
On Fri, Nov 16, 2012 at 3:47 PM, Torsten Mielke <to...@fusesource.com>wrote:
> Hello,
>
> See in-line response.
>
> On Nov 16, 2012, at 6:29 PM, Juan Nin wrote:
>
> > Hi!
> >
> > After some heavy digging about Producer Flow control and the systemUsage
> > properties a couple of years ago, I thought I quite understood it.
> > But yesterday I found that one of my configs was not behaving exactly as
> I
> > expected, so started doing some tests, and I see certain behaviours which
> > don't seem to match what the docs and posts that I find on the list or
> > other forums say.
> >
> > "storeUsage" is perfectly clear, it's the max space that persistent
> > messages can use to be stored in disk.
> > "tempUsage"" applies to file cursors on non-persistent messages, so as to
> > flush to disk if memory limits are reached (I don't care much about this
> > one anyway, I always use persistent messages).
>
> Correct.
>
> >
> > Now, according to most posts, memoryUsage would be the maximum memory
> that
> > the broker would be available to use.
> > On this post:
> >
> http://stackoverflow.com/questions/7646057/activemq-destinationpolicy-and-systemusage-configurationit
> > says that "memoryUsage corresponds to the amount of memory that's
> > assigned to the in-memory store".
>
> Correct.
>
> >
> > For example, on my tests using the following config (only showing
> relevant
> > parts):
> >
> > <policyEntry queue=">" producerFlowControl="false"
> optimizedDispatch="true">
> > <deadLetterStrategy>
> > <individualDeadLetterStrategy queuePrefix="DLQ."
> > useQueueForQueueMessages="true" />
> > </deadLetterStrategy>
> > </policyEntry>
> >
> > <systemUsage>
> > <systemUsage>
> > <memoryUsage>
> > <memoryUsage limit="100 mb"/>
> > </memoryUsage>
> > <storeUsage>
> > <storeUsage limit="1 gb" name="foo"/>
> > </storeUsage>
> > <tempUsage>
> > <tempUsage limit="3 gb"/>
> > </tempUsage>
> > </systemUsage>
> > </systemUsage>
> >
> > With that config I would expect the broker to use 100 mb of maximum
> memory
> > among all queues. So it could maybe use 30mb in one queue and 70mb in
> > second queue.
> >
> >
> > 1) What I'm seeing is that if I start feeding a queue without consuming
> it,
> > the "Memory percent used" grows up to 70%, after that it doesn't grow
> > anymore.
> > What is it doing exactly there? The first 70% is stored in memory (apart
> > from disk since it's persistent), and all the rest that continues being
> fed
> > goes just to disk?
>
> This behavior is correct. For queues the default cursor is store cursor.
> It keeps any newly arrived msgs in memory as long as it does not reach the
> configured memory limit (either configured on the queue per destination or
> globally in memoryUsage settings).
> Once the cursor reaches 70% of the configured limit (in your case of the
> memoryUsage limit since you don't specify a per-destination limit), it will
> not keep any more messages in memory.
> Instead it will reload these messages from the store when its time to
> dispatch them. The broker anyway persists any msgs it receives before
> passing on to the cursor.
> This limit of 70% can be configured and raised to e..g 100%.
> This behavior is kind of an optimization. That way you run less often into
> producer-flow-control.
> As long as the persistence store is not running full, there is no need to
> block producers, since the cursor can also load the messages from the store
> and does not necessarily have to keep them in memory.
> If you configure the vmQueueCursor, then the behavior is different. This
> cursor will not be able to load msgs to the store but needs to keep them
> all in memory. The vmQueueCursor used to be the default cursor in older
> version of AMQ.
>
> Also note that topic msgs and non-persistent queue messages are not
> handled by the store cursor. These msgs are held in memory and if memory
> runs low, get swapped out to temp storage.
>
> > 2) If then I start feeding a 2nd queue, "Memory percent used" continues
> > growing until it reaches 140%. So it looks like memoryUsage does not
> apply
> > globally, but on a per queue basis?
>
> What version of AMQ do you use? The sum of the memory usage of all queues
> should not go any higher than the configured memoryUsage limit. If you're
> not on 5.5.1 or higher releases, then I suggest to upgrade.
>
> > Using memoryLimit on the queue's policyEntry gives more control over
> this,
> > but it's just a variation, "Memory percent used" can grow more than 100%
> > anyway.
>
> With the default store cursor this should not be the case from what I know.
>
>
> >
> > 3) If #2 is true, then how would I prevent the broker from running out of
> > memory in case queues would continue to be created?
>
> Just like above comment. I would expect the brokers MemoryPercentUsage
> won't grow over 100% and the destinations MemoryPercentUsage remains fairly
> much at 70%.
> Not sure why you would see a different behavior? Using an old version of
> AMQ perhaps? Or explicitly configuring for the vmQueueCursor?
> Could you perhaps also test with
>
> >
> >
> > Maybe I'm misunderstanding and some of these settings make no sense when
> > producerFlowControl is disabled?
> >
> > Thanks in advance.
> >
> > Juan
>
>
> Regards,
>
> Torsten Mielke
> torsten@fusesource.com
> tmielke.blogspot.com
>
>
>
>
>
Re: Understanding memoryUsage (once again!)
Posted by Torsten Mielke <to...@fusesource.com>.
Hello,
See in-line response.
On Nov 16, 2012, at 6:29 PM, Juan Nin wrote:
> Hi!
>
> After some heavy digging about Producer Flow control and the systemUsage
> properties a couple of years ago, I thought I quite understood it.
> But yesterday I found that one of my configs was not behaving exactly as I
> expected, so started doing some tests, and I see certain behaviours which
> don't seem to match what the docs and posts that I find on the list or
> other forums say.
>
> "storeUsage" is perfectly clear, it's the max space that persistent
> messages can use to be stored in disk.
> "tempUsage"" applies to file cursors on non-persistent messages, so as to
> flush to disk if memory limits are reached (I don't care much about this
> one anyway, I always use persistent messages).
Correct.
>
> Now, according to most posts, memoryUsage would be the maximum memory that
> the broker would be available to use.
> On this post:
> http://stackoverflow.com/questions/7646057/activemq-destinationpolicy-and-systemusage-configurationit
> says that "memoryUsage corresponds to the amount of memory that's
> assigned to the in-memory store".
Correct.
>
> For example, on my tests using the following config (only showing relevant
> parts):
>
> <policyEntry queue=">" producerFlowControl="false" optimizedDispatch="true">
> <deadLetterStrategy>
> <individualDeadLetterStrategy queuePrefix="DLQ."
> useQueueForQueueMessages="true" />
> </deadLetterStrategy>
> </policyEntry>
>
> <systemUsage>
> <systemUsage>
> <memoryUsage>
> <memoryUsage limit="100 mb"/>
> </memoryUsage>
> <storeUsage>
> <storeUsage limit="1 gb" name="foo"/>
> </storeUsage>
> <tempUsage>
> <tempUsage limit="3 gb"/>
> </tempUsage>
> </systemUsage>
> </systemUsage>
>
> With that config I would expect the broker to use 100 mb of maximum memory
> among all queues. So it could maybe use 30mb in one queue and 70mb in
> second queue.
>
>
> 1) What I'm seeing is that if I start feeding a queue without consuming it,
> the "Memory percent used" grows up to 70%, after that it doesn't grow
> anymore.
> What is it doing exactly there? The first 70% is stored in memory (apart
> from disk since it's persistent), and all the rest that continues being fed
> goes just to disk?
This behavior is correct. For queues the default cursor is store cursor. It keeps any newly arrived msgs in memory as long as it does not reach the configured memory limit (either configured on the queue per destination or globally in memoryUsage settings).
Once the cursor reaches 70% of the configured limit (in your case of the memoryUsage limit since you don't specify a per-destination limit), it will not keep any more messages in memory.
Instead it will reload these messages from the store when its time to dispatch them. The broker anyway persists any msgs it receives before passing on to the cursor.
This limit of 70% can be configured and raised to e..g 100%.
This behavior is kind of an optimization. That way you run less often into producer-flow-control.
As long as the persistence store is not running full, there is no need to block producers, since the cursor can also load the messages from the store and does not necessarily have to keep them in memory.
If you configure the vmQueueCursor, then the behavior is different. This cursor will not be able to load msgs to the store but needs to keep them all in memory. The vmQueueCursor used to be the default cursor in older version of AMQ.
Also note that topic msgs and non-persistent queue messages are not handled by the store cursor. These msgs are held in memory and if memory runs low, get swapped out to temp storage.
> 2) If then I start feeding a 2nd queue, "Memory percent used" continues
> growing until it reaches 140%. So it looks like memoryUsage does not apply
> globally, but on a per queue basis?
What version of AMQ do you use? The sum of the memory usage of all queues should not go any higher than the configured memoryUsage limit. If you're not on 5.5.1 or higher releases, then I suggest to upgrade.
> Using memoryLimit on the queue's policyEntry gives more control over this,
> but it's just a variation, "Memory percent used" can grow more than 100%
> anyway.
With the default store cursor this should not be the case from what I know.
>
> 3) If #2 is true, then how would I prevent the broker from running out of
> memory in case queues would continue to be created?
Just like above comment. I would expect the brokers MemoryPercentUsage won't grow over 100% and the destinations MemoryPercentUsage remains fairly much at 70%.
Not sure why you would see a different behavior? Using an old version of AMQ perhaps? Or explicitly configuring for the vmQueueCursor?
Could you perhaps also test with
>
>
> Maybe I'm misunderstanding and some of these settings make no sense when
> producerFlowControl is disabled?
>
> Thanks in advance.
>
> Juan
Regards,
Torsten Mielke
torsten@fusesource.com
tmielke.blogspot.com