You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by "Martin C." <ma...@gmx.at> on 2011/11/16 11:26:03 UTC
ActiveMQ needs tempUsage for recovery of persistent messages on startup?
Hi,
I a have a question regarding fileQueueCursor and systemUsage on
ActiveMQ 5.5.0: Below you see my configuration, which limits the
memory to 64MB, store usage to 10GB and tempUsage to 1GB.
I have a KahaDB with about 300000 messages, using 330MB in the kahadb
directory, all messages are persistent. If I start ActiveMQ, I see the
tmp_storage directory grow to 1GB, where it finally blocks and
recovery of the messages starts hanging:
[...]
DEBUG | default:memory: usage change from: 2% of available memory, to:
1% of available memory
DEBUG | default:memory:queue://local.queue.myqueue:memory: usage
change from: 1% of available memory, to: 0% of available memory
DEBUG | default:memory: usage change from: 1% of available memory, to:
0% of available memory
INFO | cursor for queue://local.queue.myqueue has recovered 50000
messages. 15% complete
INFO | cursor for queue://local.queue.myqueue has recovered 100000
messages. 31% complete
INFO | cursor for queue://local.queue.myqueue has recovered 150000
messages. 47% complete
INFO | cursor for queue://local.queue.myqueue has recovered 200000
messages. 62% complete
The memory usage goes up to around 75%, where it starts dropping down
to the 0% from the logfile above. This is where it keeps hanging, with
the tmp_storage directory at 1GB diskspace.
If I increase the tempUsage to 10GB, the startup works fine
(tmp_storage becomes about 1.4GB in this case). Can you explain to me,
why I need to increase tempUsage in order to restart my broker with a
lot of persistent messages left? As far as I understood the tempUsage
is for storing non-persistent messages, persistent messages should be
affected by storeUsage?
Thanks in advance!
Best regards,
Martin
PS: the relevant parts from my configuration
<broker xmlns="http://activemq.apache.org/schema/core"
brokerName="localhost" dataDirectory="${activemq.base}/data"
populateJMSXUserID="true" useJmx="true">
<destinationPolicy>
<policyMap>
<policyEntries>
<policyEntry topic=">" producerFlowControl="false"
gcInactiveDestinations="false" inactiveTimoutBeforeGC="60000">
<vmCursor />
</policyEntry>
<policyEntry queue=">" producerFlowControl="false"
gcInactiveDestinations="false" inactiveTimoutBeforeGC="86400000">
<deadLetterStrategy>
<individualDeadLetterStrategy queuePrefix="DLQ."
useQueueForQueueMessages="true"
processExpired="true" processNonPersistent="false" />
</deadLetterStrategy>
<pendingQueuePolicy>
<fileQueueCursor />
</pendingQueuePolicy>
</policyEntry>
</policyEntries>
</policyMap>
</destinationPolicy>
<persistenceAdapter>
<kahaDB directory="${activemq.base}/data/kahadb"
checkForCorruptJournalFiles="true" checksumJournalFiles="true"
concurrentStoreAndDispatchTopics="false"
concurrentStoreAndDispatchQueues="false"
/>
</persistenceAdapter>
<systemUsage>
<systemUsage sendFailIfNoSpace="true">
<memoryUsage>
<memoryUsage limit="64 mb" />
</memoryUsage>
<storeUsage>
<storeUsage limit="10 gb" />
</storeUsage>
<tempUsage>
<tempUsage limit="1 gb" />
</tempUsage>
</systemUsage>
</systemUsage>
</broker>
Re: ActiveMQ needs tempUsage for recovery of persistent messages on startup?
Posted by Gary Tully <ga...@gmail.com>.
not at the moment, a temp queue always keeps messages in memory. I
know this has come up before and i think it is a sensible enhancement
to allow the pending message cursor to be configured for a temp
destination such that a temp destination need not be limited by
memory.
Can you raise a jira issue to track this enhancement so that it can
get scheduled.
On the temp store cleanup, that depends a bit in the journal size used
in the temp store, until a journal is fully unreferenced it can't be
removed so that may be why it is not immediate in your case. But a
test case should be able to demonstrate if there is a bug there.
On 27 August 2012 13:36, Martin C. <ma...@gmx.at> wrote:
> Hi,
>
> after some experiments, it seems that maybe one part of the problem is that
> temp-queue messages are not spooled to disk. Is there a possibility to tell
> ActiveMQ 5.5 to spool them to the temp store as well?
>
> Another issue I noticed, after purging a queue with non-persistent
> messages, the space in the temp-store seems to not be freed.
>
> Best regards,
> Martin
>
> On Mon, Aug 27, 2012 at 1:05 PM, Martin C. <ma...@gmx.at> wrote:
>
>> Hi,
>>
>> in one of the instances I didn't find anything in the broker log, just in
>> the client log, in the second log I found the same log entry in both client
>> and server:
>>
>> javax.jms.ResourceAllocationException: Usage Manager Memory Limit reached.
>> Stopping producer (ID:hostname:26:33:1) to prevent flooding
>> temp-queue://e1413fd6-2e46-76d5-dc62-df3260914cac:0
>>
>> What would you suggest to use as a persistent <pendingQueuePolicy/> in my
>> case, if I'd just want messages to go to disk if the memory-usage is
>> approached?
>>
>> Thanks in advance!
>>
>> Best regards,
>> Martin
>>
>> On Mon, Aug 27, 2012 at 12:25 PM, Gary Tully <ga...@gmail.com> wrote:
>>
>>> what is out put to the broker log when it stops processing? It should
>>> report the limit that is reached.
>>>
>>> The pendingQueueCursor will stop caching messages when the systemusage
>>> limit reaches 70%, but it won't spool to disk. The persistent messages
>>> will just remain in the store till they are needed.
>>>
>>> On 27 August 2012 10:59, Martin C. <ma...@gmx.at> wrote:
>>> > Hi Gary,
>>> >
>>> > sorry for getting back to this rather old issue, but I just got an
>>> issue in
>>> > the field (twice) with the configuration below. The broker stopped
>>> > accepting messages after it reached the 64MB memory limit, instead of
>>> > spooling them out to disk. It basically is the previously discussed
>>> > configuration with the <pendingQueuePolicy /> section removed.
>>> >
>>> > After increasing the MemoryLimit of the broker via JMX, the messages got
>>> > processed again. The temp-store usage was at 0, the persistent store
>>> usage
>>> > at 11%. So, from my understanding, the broker should not have stopped
>>> > accepting messages. Any hints? Or should I fall back to using
>>> > <fileQueueCursor /> again?
>>> >
>>> > I am still on 5.5.0 and cannot upgrade that easily.
>>> >
>>> > Thanks in advance!
>>> >
>>> > Best regards,
>>> > Martin
>>> >
>>> > The relevant parts of the configuration again.
>>> >
>>> > <broker xmlns="http://activemq.apache.org/schema/core"
>>> > brokerName="localhost" dataDirectory="${activemq.base}/data"
>>> > populateJMSXUserID="true" useJmx="true">
>>> >
>>> > <destinationPolicy>
>>> > <policyMap>
>>> > <policyEntries>
>>> > <policyEntry topic=">" producerFlowControl="false"
>>> > gcInactiveDestinations="false" inactiveTimoutBeforeGC="60000">
>>> > <vmCursor />
>>> > </policyEntry>
>>> > <policyEntry queue=">" producerFlowControl="false"
>>> > gcInactiveDestinations="false" inactiveTimoutBeforeGC="86400000">
>>> > <deadLetterStrategy>
>>> > <individualDeadLetterStrategy queuePrefix="DLQ."
>>> > useQueueForQueueMessages="true"
>>> > processExpired="true" processNonPersistent="false" />
>>> > </deadLetterStrategy>
>>> > </policyEntry>
>>> > </policyEntries>
>>> > </policyMap>
>>> > </destinationPolicy>
>>> >
>>> > <persistenceAdapter>
>>> > <kahaDB directory="${activemq.base}/data/kahadb"
>>> > checkForCorruptJournalFiles="true" checksumJournalFiles="true"
>>> > concurrentStoreAndDispatchTopics="false"
>>> > concurrentStoreAndDispatchQueues="false"
>>> > />
>>> > </persistenceAdapter>
>>> >
>>> > <systemUsage>
>>> > <systemUsage sendFailIfNoSpace="true">
>>> > <memoryUsage>
>>> > <memoryUsage limit="64 mb" />
>>> > </memoryUsage>
>>> > <storeUsage>
>>> > <storeUsage limit="10 gb" />
>>> > </storeUsage>
>>> > <tempUsage>
>>> > <tempUsage limit="1 gb" />
>>> > </tempUsage>
>>> > </systemUsage>
>>> > </systemUsage>
>>> > </broker>
>>> >
>>> > On Wed, Nov 16, 2011 at 1:50 PM, Gary Tully <ga...@gmail.com>
>>> wrote:
>>> >
>>> >> yes.
>>> >>
>>> >> On 16 November 2011 12:01, Martin C. <ma...@gmx.at> wrote:
>>> >> > Hi Gary,
>>> >> >
>>> >> > thanks for the very fast response. I will change the configuration
>>> >> > accordingly. So basically you say I should remove the
>>> >> > <pendingQueuePolicy> as a whole?
>>> >> >
>>> >> > Best regards,
>>> >> > Martin
>>> >> >
>>> >> > On Wed, Nov 16, 2011 at 11:34 AM, Gary Tully <ga...@gmail.com>
>>> >> wrote:
>>> >> >> You should not be using the file pending message cursor in this
>>> case.
>>> >> >> The default store cursor is best, it will use a file cursor for non
>>> >> >> persistent messages and leave excess (exceed the cache) persistent
>>> >> >> messages in the store.
>>> >> >> When it is specified as the default cursor in configuration, it is
>>> >> >> used for both persistent and non persistent messages.
>>> >> >>
>>> >> >> The file pending message cursor replays all messages on startup such
>>> >> >> that it can do an append to the cursor on the next add but it is
>>> only
>>> >> >> intended for really slow stores, where reading from the local file
>>> >> >> system is faster than reading from the store. This is not the case
>>> >> >> with kahaDB.
>>> >> >>
>>> >> >> On 16 November 2011 10:26, Martin C. <ma...@gmx.at> wrote:
>>> >> >>> Hi,
>>> >> >>>
>>> >> >>> I a have a question regarding fileQueueCursor and systemUsage on
>>> >> >>> ActiveMQ 5.5.0: Below you see my configuration, which limits the
>>> >> >>> memory to 64MB, store usage to 10GB and tempUsage to 1GB.
>>> >> >>>
>>> >> >>> I have a KahaDB with about 300000 messages, using 330MB in the
>>> kahadb
>>> >> >>> directory, all messages are persistent. If I start ActiveMQ, I see
>>> the
>>> >> >>> tmp_storage directory grow to 1GB, where it finally blocks and
>>> >> >>> recovery of the messages starts hanging:
>>> >> >>>
>>> >> >>> [...]
>>> >> >>> DEBUG | default:memory: usage change from: 2% of available memory,
>>> to:
>>> >> >>> 1% of available memory
>>> >> >>> DEBUG | default:memory:queue://local.queue.myqueue:memory: usage
>>> >> >>> change from: 1% of available memory, to: 0% of available memory
>>> >> >>> DEBUG | default:memory: usage change from: 1% of available memory,
>>> to:
>>> >> >>> 0% of available memory
>>> >> >>> INFO | cursor for queue://local.queue.myqueue has recovered 50000
>>> >> >>> messages. 15% complete
>>> >> >>> INFO | cursor for queue://local.queue.myqueue has recovered 100000
>>> >> >>> messages. 31% complete
>>> >> >>> INFO | cursor for queue://local.queue.myqueue has recovered 150000
>>> >> >>> messages. 47% complete
>>> >> >>> INFO | cursor for queue://local.queue.myqueue has recovered 200000
>>> >> >>> messages. 62% complete
>>> >> >>>
>>> >> >>> The memory usage goes up to around 75%, where it starts dropping
>>> down
>>> >> >>> to the 0% from the logfile above. This is where it keeps hanging,
>>> with
>>> >> >>> the tmp_storage directory at 1GB diskspace.
>>> >> >>>
>>> >> >>> If I increase the tempUsage to 10GB, the startup works fine
>>> >> >>> (tmp_storage becomes about 1.4GB in this case). Can you explain to
>>> me,
>>> >> >>> why I need to increase tempUsage in order to restart my broker
>>> with a
>>> >> >>> lot of persistent messages left? As far as I understood the
>>> tempUsage
>>> >> >>> is for storing non-persistent messages, persistent messages should
>>> be
>>> >> >>> affected by storeUsage?
>>> >> >>>
>>> >> >>> Thanks in advance!
>>> >> >>>
>>> >> >>> Best regards,
>>> >> >>> Martin
>>> >> >>>
>>> >> >>> PS: the relevant parts from my configuration
>>> >> >>>
>>> >> >>> <broker xmlns="http://activemq.apache.org/schema/core"
>>> >> >>> brokerName="localhost" dataDirectory="${activemq.base}/data"
>>> >> >>> populateJMSXUserID="true" useJmx="true">
>>> >> >>>
>>> >> >>> <destinationPolicy>
>>> >> >>> <policyMap>
>>> >> >>> <policyEntries>
>>> >> >>> <policyEntry topic=">" producerFlowControl="false"
>>> >> >>> gcInactiveDestinations="false" inactiveTimoutBeforeGC="60000">
>>> >> >>> <vmCursor />
>>> >> >>> </policyEntry>
>>> >> >>> <policyEntry queue=">" producerFlowControl="false"
>>> >> >>> gcInactiveDestinations="false" inactiveTimoutBeforeGC="86400000">
>>> >> >>> <deadLetterStrategy>
>>> >> >>> <individualDeadLetterStrategy queuePrefix="DLQ."
>>> >> >>> useQueueForQueueMessages="true"
>>> >> >>> processExpired="true" processNonPersistent="false"
>>> />
>>> >> >>> </deadLetterStrategy>
>>> >> >>> <pendingQueuePolicy>
>>> >> >>> <fileQueueCursor />
>>> >> >>> </pendingQueuePolicy>
>>> >> >>> </policyEntry>
>>> >> >>> </policyEntries>
>>> >> >>> </policyMap>
>>> >> >>> </destinationPolicy>
>>> >> >>>
>>> >> >>> <persistenceAdapter>
>>> >> >>> <kahaDB directory="${activemq.base}/data/kahadb"
>>> >> >>> checkForCorruptJournalFiles="true"
>>> checksumJournalFiles="true"
>>> >> >>> concurrentStoreAndDispatchTopics="false"
>>> >> >>> concurrentStoreAndDispatchQueues="false"
>>> >> >>> />
>>> >> >>> </persistenceAdapter>
>>> >> >>>
>>> >> >>> <systemUsage>
>>> >> >>> <systemUsage sendFailIfNoSpace="true">
>>> >> >>> <memoryUsage>
>>> >> >>> <memoryUsage limit="64 mb" />
>>> >> >>> </memoryUsage>
>>> >> >>> <storeUsage>
>>> >> >>> <storeUsage limit="10 gb" />
>>> >> >>> </storeUsage>
>>> >> >>> <tempUsage>
>>> >> >>> <tempUsage limit="1 gb" />
>>> >> >>> </tempUsage>
>>> >> >>> </systemUsage>
>>> >> >>> </systemUsage>
>>> >> >>> </broker>
>>> >> >>>
>>> >> >>
>>> >> >>
>>> >> >>
>>> >> >> --
>>> >> >> http://fusesource.com
>>> >> >> http://blog.garytully.com
>>> >> >>
>>> >> >
>>> >>
>>> >>
>>> >>
>>> >> --
>>> >> http://fusesource.com
>>> >> http://blog.garytully.com
>>> >>
>>>
>>>
>>>
>>> --
>>> http://fusesource.com
>>> http://blog.garytully.com
>>>
>>
>>
--
http://fusesource.com
http://blog.garytully.com
Re: ActiveMQ needs tempUsage for recovery of persistent messages on startup?
Posted by "Martin C." <ma...@gmx.at>.
Hi,
after some experiments, it seems that maybe one part of the problem is that
temp-queue messages are not spooled to disk. Is there a possibility to tell
ActiveMQ 5.5 to spool them to the temp store as well?
Another issue I noticed, after purging a queue with non-persistent
messages, the space in the temp-store seems to not be freed.
Best regards,
Martin
On Mon, Aug 27, 2012 at 1:05 PM, Martin C. <ma...@gmx.at> wrote:
> Hi,
>
> in one of the instances I didn't find anything in the broker log, just in
> the client log, in the second log I found the same log entry in both client
> and server:
>
> javax.jms.ResourceAllocationException: Usage Manager Memory Limit reached.
> Stopping producer (ID:hostname:26:33:1) to prevent flooding
> temp-queue://e1413fd6-2e46-76d5-dc62-df3260914cac:0
>
> What would you suggest to use as a persistent <pendingQueuePolicy/> in my
> case, if I'd just want messages to go to disk if the memory-usage is
> approached?
>
> Thanks in advance!
>
> Best regards,
> Martin
>
> On Mon, Aug 27, 2012 at 12:25 PM, Gary Tully <ga...@gmail.com> wrote:
>
>> what is out put to the broker log when it stops processing? It should
>> report the limit that is reached.
>>
>> The pendingQueueCursor will stop caching messages when the systemusage
>> limit reaches 70%, but it won't spool to disk. The persistent messages
>> will just remain in the store till they are needed.
>>
>> On 27 August 2012 10:59, Martin C. <ma...@gmx.at> wrote:
>> > Hi Gary,
>> >
>> > sorry for getting back to this rather old issue, but I just got an
>> issue in
>> > the field (twice) with the configuration below. The broker stopped
>> > accepting messages after it reached the 64MB memory limit, instead of
>> > spooling them out to disk. It basically is the previously discussed
>> > configuration with the <pendingQueuePolicy /> section removed.
>> >
>> > After increasing the MemoryLimit of the broker via JMX, the messages got
>> > processed again. The temp-store usage was at 0, the persistent store
>> usage
>> > at 11%. So, from my understanding, the broker should not have stopped
>> > accepting messages. Any hints? Or should I fall back to using
>> > <fileQueueCursor /> again?
>> >
>> > I am still on 5.5.0 and cannot upgrade that easily.
>> >
>> > Thanks in advance!
>> >
>> > Best regards,
>> > Martin
>> >
>> > The relevant parts of the configuration again.
>> >
>> > <broker xmlns="http://activemq.apache.org/schema/core"
>> > brokerName="localhost" dataDirectory="${activemq.base}/data"
>> > populateJMSXUserID="true" useJmx="true">
>> >
>> > <destinationPolicy>
>> > <policyMap>
>> > <policyEntries>
>> > <policyEntry topic=">" producerFlowControl="false"
>> > gcInactiveDestinations="false" inactiveTimoutBeforeGC="60000">
>> > <vmCursor />
>> > </policyEntry>
>> > <policyEntry queue=">" producerFlowControl="false"
>> > gcInactiveDestinations="false" inactiveTimoutBeforeGC="86400000">
>> > <deadLetterStrategy>
>> > <individualDeadLetterStrategy queuePrefix="DLQ."
>> > useQueueForQueueMessages="true"
>> > processExpired="true" processNonPersistent="false" />
>> > </deadLetterStrategy>
>> > </policyEntry>
>> > </policyEntries>
>> > </policyMap>
>> > </destinationPolicy>
>> >
>> > <persistenceAdapter>
>> > <kahaDB directory="${activemq.base}/data/kahadb"
>> > checkForCorruptJournalFiles="true" checksumJournalFiles="true"
>> > concurrentStoreAndDispatchTopics="false"
>> > concurrentStoreAndDispatchQueues="false"
>> > />
>> > </persistenceAdapter>
>> >
>> > <systemUsage>
>> > <systemUsage sendFailIfNoSpace="true">
>> > <memoryUsage>
>> > <memoryUsage limit="64 mb" />
>> > </memoryUsage>
>> > <storeUsage>
>> > <storeUsage limit="10 gb" />
>> > </storeUsage>
>> > <tempUsage>
>> > <tempUsage limit="1 gb" />
>> > </tempUsage>
>> > </systemUsage>
>> > </systemUsage>
>> > </broker>
>> >
>> > On Wed, Nov 16, 2011 at 1:50 PM, Gary Tully <ga...@gmail.com>
>> wrote:
>> >
>> >> yes.
>> >>
>> >> On 16 November 2011 12:01, Martin C. <ma...@gmx.at> wrote:
>> >> > Hi Gary,
>> >> >
>> >> > thanks for the very fast response. I will change the configuration
>> >> > accordingly. So basically you say I should remove the
>> >> > <pendingQueuePolicy> as a whole?
>> >> >
>> >> > Best regards,
>> >> > Martin
>> >> >
>> >> > On Wed, Nov 16, 2011 at 11:34 AM, Gary Tully <ga...@gmail.com>
>> >> wrote:
>> >> >> You should not be using the file pending message cursor in this
>> case.
>> >> >> The default store cursor is best, it will use a file cursor for non
>> >> >> persistent messages and leave excess (exceed the cache) persistent
>> >> >> messages in the store.
>> >> >> When it is specified as the default cursor in configuration, it is
>> >> >> used for both persistent and non persistent messages.
>> >> >>
>> >> >> The file pending message cursor replays all messages on startup such
>> >> >> that it can do an append to the cursor on the next add but it is
>> only
>> >> >> intended for really slow stores, where reading from the local file
>> >> >> system is faster than reading from the store. This is not the case
>> >> >> with kahaDB.
>> >> >>
>> >> >> On 16 November 2011 10:26, Martin C. <ma...@gmx.at> wrote:
>> >> >>> Hi,
>> >> >>>
>> >> >>> I a have a question regarding fileQueueCursor and systemUsage on
>> >> >>> ActiveMQ 5.5.0: Below you see my configuration, which limits the
>> >> >>> memory to 64MB, store usage to 10GB and tempUsage to 1GB.
>> >> >>>
>> >> >>> I have a KahaDB with about 300000 messages, using 330MB in the
>> kahadb
>> >> >>> directory, all messages are persistent. If I start ActiveMQ, I see
>> the
>> >> >>> tmp_storage directory grow to 1GB, where it finally blocks and
>> >> >>> recovery of the messages starts hanging:
>> >> >>>
>> >> >>> [...]
>> >> >>> DEBUG | default:memory: usage change from: 2% of available memory,
>> to:
>> >> >>> 1% of available memory
>> >> >>> DEBUG | default:memory:queue://local.queue.myqueue:memory: usage
>> >> >>> change from: 1% of available memory, to: 0% of available memory
>> >> >>> DEBUG | default:memory: usage change from: 1% of available memory,
>> to:
>> >> >>> 0% of available memory
>> >> >>> INFO | cursor for queue://local.queue.myqueue has recovered 50000
>> >> >>> messages. 15% complete
>> >> >>> INFO | cursor for queue://local.queue.myqueue has recovered 100000
>> >> >>> messages. 31% complete
>> >> >>> INFO | cursor for queue://local.queue.myqueue has recovered 150000
>> >> >>> messages. 47% complete
>> >> >>> INFO | cursor for queue://local.queue.myqueue has recovered 200000
>> >> >>> messages. 62% complete
>> >> >>>
>> >> >>> The memory usage goes up to around 75%, where it starts dropping
>> down
>> >> >>> to the 0% from the logfile above. This is where it keeps hanging,
>> with
>> >> >>> the tmp_storage directory at 1GB diskspace.
>> >> >>>
>> >> >>> If I increase the tempUsage to 10GB, the startup works fine
>> >> >>> (tmp_storage becomes about 1.4GB in this case). Can you explain to
>> me,
>> >> >>> why I need to increase tempUsage in order to restart my broker
>> with a
>> >> >>> lot of persistent messages left? As far as I understood the
>> tempUsage
>> >> >>> is for storing non-persistent messages, persistent messages should
>> be
>> >> >>> affected by storeUsage?
>> >> >>>
>> >> >>> Thanks in advance!
>> >> >>>
>> >> >>> Best regards,
>> >> >>> Martin
>> >> >>>
>> >> >>> PS: the relevant parts from my configuration
>> >> >>>
>> >> >>> <broker xmlns="http://activemq.apache.org/schema/core"
>> >> >>> brokerName="localhost" dataDirectory="${activemq.base}/data"
>> >> >>> populateJMSXUserID="true" useJmx="true">
>> >> >>>
>> >> >>> <destinationPolicy>
>> >> >>> <policyMap>
>> >> >>> <policyEntries>
>> >> >>> <policyEntry topic=">" producerFlowControl="false"
>> >> >>> gcInactiveDestinations="false" inactiveTimoutBeforeGC="60000">
>> >> >>> <vmCursor />
>> >> >>> </policyEntry>
>> >> >>> <policyEntry queue=">" producerFlowControl="false"
>> >> >>> gcInactiveDestinations="false" inactiveTimoutBeforeGC="86400000">
>> >> >>> <deadLetterStrategy>
>> >> >>> <individualDeadLetterStrategy queuePrefix="DLQ."
>> >> >>> useQueueForQueueMessages="true"
>> >> >>> processExpired="true" processNonPersistent="false"
>> />
>> >> >>> </deadLetterStrategy>
>> >> >>> <pendingQueuePolicy>
>> >> >>> <fileQueueCursor />
>> >> >>> </pendingQueuePolicy>
>> >> >>> </policyEntry>
>> >> >>> </policyEntries>
>> >> >>> </policyMap>
>> >> >>> </destinationPolicy>
>> >> >>>
>> >> >>> <persistenceAdapter>
>> >> >>> <kahaDB directory="${activemq.base}/data/kahadb"
>> >> >>> checkForCorruptJournalFiles="true"
>> checksumJournalFiles="true"
>> >> >>> concurrentStoreAndDispatchTopics="false"
>> >> >>> concurrentStoreAndDispatchQueues="false"
>> >> >>> />
>> >> >>> </persistenceAdapter>
>> >> >>>
>> >> >>> <systemUsage>
>> >> >>> <systemUsage sendFailIfNoSpace="true">
>> >> >>> <memoryUsage>
>> >> >>> <memoryUsage limit="64 mb" />
>> >> >>> </memoryUsage>
>> >> >>> <storeUsage>
>> >> >>> <storeUsage limit="10 gb" />
>> >> >>> </storeUsage>
>> >> >>> <tempUsage>
>> >> >>> <tempUsage limit="1 gb" />
>> >> >>> </tempUsage>
>> >> >>> </systemUsage>
>> >> >>> </systemUsage>
>> >> >>> </broker>
>> >> >>>
>> >> >>
>> >> >>
>> >> >>
>> >> >> --
>> >> >> http://fusesource.com
>> >> >> http://blog.garytully.com
>> >> >>
>> >> >
>> >>
>> >>
>> >>
>> >> --
>> >> http://fusesource.com
>> >> http://blog.garytully.com
>> >>
>>
>>
>>
>> --
>> http://fusesource.com
>> http://blog.garytully.com
>>
>
>
Re: ActiveMQ needs tempUsage for recovery of persistent messages on startup?
Posted by "Martin C." <ma...@gmx.at>.
Hi,
in one of the instances I didn't find anything in the broker log, just in
the client log, in the second log I found the same log entry in both client
and server:
javax.jms.ResourceAllocationException: Usage Manager Memory Limit reached.
Stopping producer (ID:hostname:26:33:1) to prevent flooding
temp-queue://e1413fd6-2e46-76d5-dc62-df3260914cac:0
What would you suggest to use as a persistent <pendingQueuePolicy/> in my
case, if I'd just want messages to go to disk if the memory-usage is
approached?
Thanks in advance!
Best regards,
Martin
On Mon, Aug 27, 2012 at 12:25 PM, Gary Tully <ga...@gmail.com> wrote:
> what is out put to the broker log when it stops processing? It should
> report the limit that is reached.
>
> The pendingQueueCursor will stop caching messages when the systemusage
> limit reaches 70%, but it won't spool to disk. The persistent messages
> will just remain in the store till they are needed.
>
> On 27 August 2012 10:59, Martin C. <ma...@gmx.at> wrote:
> > Hi Gary,
> >
> > sorry for getting back to this rather old issue, but I just got an issue
> in
> > the field (twice) with the configuration below. The broker stopped
> > accepting messages after it reached the 64MB memory limit, instead of
> > spooling them out to disk. It basically is the previously discussed
> > configuration with the <pendingQueuePolicy /> section removed.
> >
> > After increasing the MemoryLimit of the broker via JMX, the messages got
> > processed again. The temp-store usage was at 0, the persistent store
> usage
> > at 11%. So, from my understanding, the broker should not have stopped
> > accepting messages. Any hints? Or should I fall back to using
> > <fileQueueCursor /> again?
> >
> > I am still on 5.5.0 and cannot upgrade that easily.
> >
> > Thanks in advance!
> >
> > Best regards,
> > Martin
> >
> > The relevant parts of the configuration again.
> >
> > <broker xmlns="http://activemq.apache.org/schema/core"
> > brokerName="localhost" dataDirectory="${activemq.base}/data"
> > populateJMSXUserID="true" useJmx="true">
> >
> > <destinationPolicy>
> > <policyMap>
> > <policyEntries>
> > <policyEntry topic=">" producerFlowControl="false"
> > gcInactiveDestinations="false" inactiveTimoutBeforeGC="60000">
> > <vmCursor />
> > </policyEntry>
> > <policyEntry queue=">" producerFlowControl="false"
> > gcInactiveDestinations="false" inactiveTimoutBeforeGC="86400000">
> > <deadLetterStrategy>
> > <individualDeadLetterStrategy queuePrefix="DLQ."
> > useQueueForQueueMessages="true"
> > processExpired="true" processNonPersistent="false" />
> > </deadLetterStrategy>
> > </policyEntry>
> > </policyEntries>
> > </policyMap>
> > </destinationPolicy>
> >
> > <persistenceAdapter>
> > <kahaDB directory="${activemq.base}/data/kahadb"
> > checkForCorruptJournalFiles="true" checksumJournalFiles="true"
> > concurrentStoreAndDispatchTopics="false"
> > concurrentStoreAndDispatchQueues="false"
> > />
> > </persistenceAdapter>
> >
> > <systemUsage>
> > <systemUsage sendFailIfNoSpace="true">
> > <memoryUsage>
> > <memoryUsage limit="64 mb" />
> > </memoryUsage>
> > <storeUsage>
> > <storeUsage limit="10 gb" />
> > </storeUsage>
> > <tempUsage>
> > <tempUsage limit="1 gb" />
> > </tempUsage>
> > </systemUsage>
> > </systemUsage>
> > </broker>
> >
> > On Wed, Nov 16, 2011 at 1:50 PM, Gary Tully <ga...@gmail.com>
> wrote:
> >
> >> yes.
> >>
> >> On 16 November 2011 12:01, Martin C. <ma...@gmx.at> wrote:
> >> > Hi Gary,
> >> >
> >> > thanks for the very fast response. I will change the configuration
> >> > accordingly. So basically you say I should remove the
> >> > <pendingQueuePolicy> as a whole?
> >> >
> >> > Best regards,
> >> > Martin
> >> >
> >> > On Wed, Nov 16, 2011 at 11:34 AM, Gary Tully <ga...@gmail.com>
> >> wrote:
> >> >> You should not be using the file pending message cursor in this case.
> >> >> The default store cursor is best, it will use a file cursor for non
> >> >> persistent messages and leave excess (exceed the cache) persistent
> >> >> messages in the store.
> >> >> When it is specified as the default cursor in configuration, it is
> >> >> used for both persistent and non persistent messages.
> >> >>
> >> >> The file pending message cursor replays all messages on startup such
> >> >> that it can do an append to the cursor on the next add but it is only
> >> >> intended for really slow stores, where reading from the local file
> >> >> system is faster than reading from the store. This is not the case
> >> >> with kahaDB.
> >> >>
> >> >> On 16 November 2011 10:26, Martin C. <ma...@gmx.at> wrote:
> >> >>> Hi,
> >> >>>
> >> >>> I a have a question regarding fileQueueCursor and systemUsage on
> >> >>> ActiveMQ 5.5.0: Below you see my configuration, which limits the
> >> >>> memory to 64MB, store usage to 10GB and tempUsage to 1GB.
> >> >>>
> >> >>> I have a KahaDB with about 300000 messages, using 330MB in the
> kahadb
> >> >>> directory, all messages are persistent. If I start ActiveMQ, I see
> the
> >> >>> tmp_storage directory grow to 1GB, where it finally blocks and
> >> >>> recovery of the messages starts hanging:
> >> >>>
> >> >>> [...]
> >> >>> DEBUG | default:memory: usage change from: 2% of available memory,
> to:
> >> >>> 1% of available memory
> >> >>> DEBUG | default:memory:queue://local.queue.myqueue:memory: usage
> >> >>> change from: 1% of available memory, to: 0% of available memory
> >> >>> DEBUG | default:memory: usage change from: 1% of available memory,
> to:
> >> >>> 0% of available memory
> >> >>> INFO | cursor for queue://local.queue.myqueue has recovered 50000
> >> >>> messages. 15% complete
> >> >>> INFO | cursor for queue://local.queue.myqueue has recovered 100000
> >> >>> messages. 31% complete
> >> >>> INFO | cursor for queue://local.queue.myqueue has recovered 150000
> >> >>> messages. 47% complete
> >> >>> INFO | cursor for queue://local.queue.myqueue has recovered 200000
> >> >>> messages. 62% complete
> >> >>>
> >> >>> The memory usage goes up to around 75%, where it starts dropping
> down
> >> >>> to the 0% from the logfile above. This is where it keeps hanging,
> with
> >> >>> the tmp_storage directory at 1GB diskspace.
> >> >>>
> >> >>> If I increase the tempUsage to 10GB, the startup works fine
> >> >>> (tmp_storage becomes about 1.4GB in this case). Can you explain to
> me,
> >> >>> why I need to increase tempUsage in order to restart my broker with
> a
> >> >>> lot of persistent messages left? As far as I understood the
> tempUsage
> >> >>> is for storing non-persistent messages, persistent messages should
> be
> >> >>> affected by storeUsage?
> >> >>>
> >> >>> Thanks in advance!
> >> >>>
> >> >>> Best regards,
> >> >>> Martin
> >> >>>
> >> >>> PS: the relevant parts from my configuration
> >> >>>
> >> >>> <broker xmlns="http://activemq.apache.org/schema/core"
> >> >>> brokerName="localhost" dataDirectory="${activemq.base}/data"
> >> >>> populateJMSXUserID="true" useJmx="true">
> >> >>>
> >> >>> <destinationPolicy>
> >> >>> <policyMap>
> >> >>> <policyEntries>
> >> >>> <policyEntry topic=">" producerFlowControl="false"
> >> >>> gcInactiveDestinations="false" inactiveTimoutBeforeGC="60000">
> >> >>> <vmCursor />
> >> >>> </policyEntry>
> >> >>> <policyEntry queue=">" producerFlowControl="false"
> >> >>> gcInactiveDestinations="false" inactiveTimoutBeforeGC="86400000">
> >> >>> <deadLetterStrategy>
> >> >>> <individualDeadLetterStrategy queuePrefix="DLQ."
> >> >>> useQueueForQueueMessages="true"
> >> >>> processExpired="true" processNonPersistent="false" />
> >> >>> </deadLetterStrategy>
> >> >>> <pendingQueuePolicy>
> >> >>> <fileQueueCursor />
> >> >>> </pendingQueuePolicy>
> >> >>> </policyEntry>
> >> >>> </policyEntries>
> >> >>> </policyMap>
> >> >>> </destinationPolicy>
> >> >>>
> >> >>> <persistenceAdapter>
> >> >>> <kahaDB directory="${activemq.base}/data/kahadb"
> >> >>> checkForCorruptJournalFiles="true"
> checksumJournalFiles="true"
> >> >>> concurrentStoreAndDispatchTopics="false"
> >> >>> concurrentStoreAndDispatchQueues="false"
> >> >>> />
> >> >>> </persistenceAdapter>
> >> >>>
> >> >>> <systemUsage>
> >> >>> <systemUsage sendFailIfNoSpace="true">
> >> >>> <memoryUsage>
> >> >>> <memoryUsage limit="64 mb" />
> >> >>> </memoryUsage>
> >> >>> <storeUsage>
> >> >>> <storeUsage limit="10 gb" />
> >> >>> </storeUsage>
> >> >>> <tempUsage>
> >> >>> <tempUsage limit="1 gb" />
> >> >>> </tempUsage>
> >> >>> </systemUsage>
> >> >>> </systemUsage>
> >> >>> </broker>
> >> >>>
> >> >>
> >> >>
> >> >>
> >> >> --
> >> >> http://fusesource.com
> >> >> http://blog.garytully.com
> >> >>
> >> >
> >>
> >>
> >>
> >> --
> >> http://fusesource.com
> >> http://blog.garytully.com
> >>
>
>
>
> --
> http://fusesource.com
> http://blog.garytully.com
>
Re: ActiveMQ needs tempUsage for recovery of persistent messages on startup?
Posted by Gary Tully <ga...@gmail.com>.
what is out put to the broker log when it stops processing? It should
report the limit that is reached.
The pendingQueueCursor will stop caching messages when the systemusage
limit reaches 70%, but it won't spool to disk. The persistent messages
will just remain in the store till they are needed.
On 27 August 2012 10:59, Martin C. <ma...@gmx.at> wrote:
> Hi Gary,
>
> sorry for getting back to this rather old issue, but I just got an issue in
> the field (twice) with the configuration below. The broker stopped
> accepting messages after it reached the 64MB memory limit, instead of
> spooling them out to disk. It basically is the previously discussed
> configuration with the <pendingQueuePolicy /> section removed.
>
> After increasing the MemoryLimit of the broker via JMX, the messages got
> processed again. The temp-store usage was at 0, the persistent store usage
> at 11%. So, from my understanding, the broker should not have stopped
> accepting messages. Any hints? Or should I fall back to using
> <fileQueueCursor /> again?
>
> I am still on 5.5.0 and cannot upgrade that easily.
>
> Thanks in advance!
>
> Best regards,
> Martin
>
> The relevant parts of the configuration again.
>
> <broker xmlns="http://activemq.apache.org/schema/core"
> brokerName="localhost" dataDirectory="${activemq.base}/data"
> populateJMSXUserID="true" useJmx="true">
>
> <destinationPolicy>
> <policyMap>
> <policyEntries>
> <policyEntry topic=">" producerFlowControl="false"
> gcInactiveDestinations="false" inactiveTimoutBeforeGC="60000">
> <vmCursor />
> </policyEntry>
> <policyEntry queue=">" producerFlowControl="false"
> gcInactiveDestinations="false" inactiveTimoutBeforeGC="86400000">
> <deadLetterStrategy>
> <individualDeadLetterStrategy queuePrefix="DLQ."
> useQueueForQueueMessages="true"
> processExpired="true" processNonPersistent="false" />
> </deadLetterStrategy>
> </policyEntry>
> </policyEntries>
> </policyMap>
> </destinationPolicy>
>
> <persistenceAdapter>
> <kahaDB directory="${activemq.base}/data/kahadb"
> checkForCorruptJournalFiles="true" checksumJournalFiles="true"
> concurrentStoreAndDispatchTopics="false"
> concurrentStoreAndDispatchQueues="false"
> />
> </persistenceAdapter>
>
> <systemUsage>
> <systemUsage sendFailIfNoSpace="true">
> <memoryUsage>
> <memoryUsage limit="64 mb" />
> </memoryUsage>
> <storeUsage>
> <storeUsage limit="10 gb" />
> </storeUsage>
> <tempUsage>
> <tempUsage limit="1 gb" />
> </tempUsage>
> </systemUsage>
> </systemUsage>
> </broker>
>
> On Wed, Nov 16, 2011 at 1:50 PM, Gary Tully <ga...@gmail.com> wrote:
>
>> yes.
>>
>> On 16 November 2011 12:01, Martin C. <ma...@gmx.at> wrote:
>> > Hi Gary,
>> >
>> > thanks for the very fast response. I will change the configuration
>> > accordingly. So basically you say I should remove the
>> > <pendingQueuePolicy> as a whole?
>> >
>> > Best regards,
>> > Martin
>> >
>> > On Wed, Nov 16, 2011 at 11:34 AM, Gary Tully <ga...@gmail.com>
>> wrote:
>> >> You should not be using the file pending message cursor in this case.
>> >> The default store cursor is best, it will use a file cursor for non
>> >> persistent messages and leave excess (exceed the cache) persistent
>> >> messages in the store.
>> >> When it is specified as the default cursor in configuration, it is
>> >> used for both persistent and non persistent messages.
>> >>
>> >> The file pending message cursor replays all messages on startup such
>> >> that it can do an append to the cursor on the next add but it is only
>> >> intended for really slow stores, where reading from the local file
>> >> system is faster than reading from the store. This is not the case
>> >> with kahaDB.
>> >>
>> >> On 16 November 2011 10:26, Martin C. <ma...@gmx.at> wrote:
>> >>> Hi,
>> >>>
>> >>> I a have a question regarding fileQueueCursor and systemUsage on
>> >>> ActiveMQ 5.5.0: Below you see my configuration, which limits the
>> >>> memory to 64MB, store usage to 10GB and tempUsage to 1GB.
>> >>>
>> >>> I have a KahaDB with about 300000 messages, using 330MB in the kahadb
>> >>> directory, all messages are persistent. If I start ActiveMQ, I see the
>> >>> tmp_storage directory grow to 1GB, where it finally blocks and
>> >>> recovery of the messages starts hanging:
>> >>>
>> >>> [...]
>> >>> DEBUG | default:memory: usage change from: 2% of available memory, to:
>> >>> 1% of available memory
>> >>> DEBUG | default:memory:queue://local.queue.myqueue:memory: usage
>> >>> change from: 1% of available memory, to: 0% of available memory
>> >>> DEBUG | default:memory: usage change from: 1% of available memory, to:
>> >>> 0% of available memory
>> >>> INFO | cursor for queue://local.queue.myqueue has recovered 50000
>> >>> messages. 15% complete
>> >>> INFO | cursor for queue://local.queue.myqueue has recovered 100000
>> >>> messages. 31% complete
>> >>> INFO | cursor for queue://local.queue.myqueue has recovered 150000
>> >>> messages. 47% complete
>> >>> INFO | cursor for queue://local.queue.myqueue has recovered 200000
>> >>> messages. 62% complete
>> >>>
>> >>> The memory usage goes up to around 75%, where it starts dropping down
>> >>> to the 0% from the logfile above. This is where it keeps hanging, with
>> >>> the tmp_storage directory at 1GB diskspace.
>> >>>
>> >>> If I increase the tempUsage to 10GB, the startup works fine
>> >>> (tmp_storage becomes about 1.4GB in this case). Can you explain to me,
>> >>> why I need to increase tempUsage in order to restart my broker with a
>> >>> lot of persistent messages left? As far as I understood the tempUsage
>> >>> is for storing non-persistent messages, persistent messages should be
>> >>> affected by storeUsage?
>> >>>
>> >>> Thanks in advance!
>> >>>
>> >>> Best regards,
>> >>> Martin
>> >>>
>> >>> PS: the relevant parts from my configuration
>> >>>
>> >>> <broker xmlns="http://activemq.apache.org/schema/core"
>> >>> brokerName="localhost" dataDirectory="${activemq.base}/data"
>> >>> populateJMSXUserID="true" useJmx="true">
>> >>>
>> >>> <destinationPolicy>
>> >>> <policyMap>
>> >>> <policyEntries>
>> >>> <policyEntry topic=">" producerFlowControl="false"
>> >>> gcInactiveDestinations="false" inactiveTimoutBeforeGC="60000">
>> >>> <vmCursor />
>> >>> </policyEntry>
>> >>> <policyEntry queue=">" producerFlowControl="false"
>> >>> gcInactiveDestinations="false" inactiveTimoutBeforeGC="86400000">
>> >>> <deadLetterStrategy>
>> >>> <individualDeadLetterStrategy queuePrefix="DLQ."
>> >>> useQueueForQueueMessages="true"
>> >>> processExpired="true" processNonPersistent="false" />
>> >>> </deadLetterStrategy>
>> >>> <pendingQueuePolicy>
>> >>> <fileQueueCursor />
>> >>> </pendingQueuePolicy>
>> >>> </policyEntry>
>> >>> </policyEntries>
>> >>> </policyMap>
>> >>> </destinationPolicy>
>> >>>
>> >>> <persistenceAdapter>
>> >>> <kahaDB directory="${activemq.base}/data/kahadb"
>> >>> checkForCorruptJournalFiles="true" checksumJournalFiles="true"
>> >>> concurrentStoreAndDispatchTopics="false"
>> >>> concurrentStoreAndDispatchQueues="false"
>> >>> />
>> >>> </persistenceAdapter>
>> >>>
>> >>> <systemUsage>
>> >>> <systemUsage sendFailIfNoSpace="true">
>> >>> <memoryUsage>
>> >>> <memoryUsage limit="64 mb" />
>> >>> </memoryUsage>
>> >>> <storeUsage>
>> >>> <storeUsage limit="10 gb" />
>> >>> </storeUsage>
>> >>> <tempUsage>
>> >>> <tempUsage limit="1 gb" />
>> >>> </tempUsage>
>> >>> </systemUsage>
>> >>> </systemUsage>
>> >>> </broker>
>> >>>
>> >>
>> >>
>> >>
>> >> --
>> >> http://fusesource.com
>> >> http://blog.garytully.com
>> >>
>> >
>>
>>
>>
>> --
>> http://fusesource.com
>> http://blog.garytully.com
>>
--
http://fusesource.com
http://blog.garytully.com
Re: ActiveMQ needs tempUsage for recovery of persistent messages on startup?
Posted by "Martin C." <ma...@gmx.at>.
Hi Gary,
sorry for getting back to this rather old issue, but I just got an issue in
the field (twice) with the configuration below. The broker stopped
accepting messages after it reached the 64MB memory limit, instead of
spooling them out to disk. It basically is the previously discussed
configuration with the <pendingQueuePolicy /> section removed.
After increasing the MemoryLimit of the broker via JMX, the messages got
processed again. The temp-store usage was at 0, the persistent store usage
at 11%. So, from my understanding, the broker should not have stopped
accepting messages. Any hints? Or should I fall back to using
<fileQueueCursor /> again?
I am still on 5.5.0 and cannot upgrade that easily.
Thanks in advance!
Best regards,
Martin
The relevant parts of the configuration again.
<broker xmlns="http://activemq.apache.org/schema/core"
brokerName="localhost" dataDirectory="${activemq.base}/data"
populateJMSXUserID="true" useJmx="true">
<destinationPolicy>
<policyMap>
<policyEntries>
<policyEntry topic=">" producerFlowControl="false"
gcInactiveDestinations="false" inactiveTimoutBeforeGC="60000">
<vmCursor />
</policyEntry>
<policyEntry queue=">" producerFlowControl="false"
gcInactiveDestinations="false" inactiveTimoutBeforeGC="86400000">
<deadLetterStrategy>
<individualDeadLetterStrategy queuePrefix="DLQ."
useQueueForQueueMessages="true"
processExpired="true" processNonPersistent="false" />
</deadLetterStrategy>
</policyEntry>
</policyEntries>
</policyMap>
</destinationPolicy>
<persistenceAdapter>
<kahaDB directory="${activemq.base}/data/kahadb"
checkForCorruptJournalFiles="true" checksumJournalFiles="true"
concurrentStoreAndDispatchTopics="false"
concurrentStoreAndDispatchQueues="false"
/>
</persistenceAdapter>
<systemUsage>
<systemUsage sendFailIfNoSpace="true">
<memoryUsage>
<memoryUsage limit="64 mb" />
</memoryUsage>
<storeUsage>
<storeUsage limit="10 gb" />
</storeUsage>
<tempUsage>
<tempUsage limit="1 gb" />
</tempUsage>
</systemUsage>
</systemUsage>
</broker>
On Wed, Nov 16, 2011 at 1:50 PM, Gary Tully <ga...@gmail.com> wrote:
> yes.
>
> On 16 November 2011 12:01, Martin C. <ma...@gmx.at> wrote:
> > Hi Gary,
> >
> > thanks for the very fast response. I will change the configuration
> > accordingly. So basically you say I should remove the
> > <pendingQueuePolicy> as a whole?
> >
> > Best regards,
> > Martin
> >
> > On Wed, Nov 16, 2011 at 11:34 AM, Gary Tully <ga...@gmail.com>
> wrote:
> >> You should not be using the file pending message cursor in this case.
> >> The default store cursor is best, it will use a file cursor for non
> >> persistent messages and leave excess (exceed the cache) persistent
> >> messages in the store.
> >> When it is specified as the default cursor in configuration, it is
> >> used for both persistent and non persistent messages.
> >>
> >> The file pending message cursor replays all messages on startup such
> >> that it can do an append to the cursor on the next add but it is only
> >> intended for really slow stores, where reading from the local file
> >> system is faster than reading from the store. This is not the case
> >> with kahaDB.
> >>
> >> On 16 November 2011 10:26, Martin C. <ma...@gmx.at> wrote:
> >>> Hi,
> >>>
> >>> I a have a question regarding fileQueueCursor and systemUsage on
> >>> ActiveMQ 5.5.0: Below you see my configuration, which limits the
> >>> memory to 64MB, store usage to 10GB and tempUsage to 1GB.
> >>>
> >>> I have a KahaDB with about 300000 messages, using 330MB in the kahadb
> >>> directory, all messages are persistent. If I start ActiveMQ, I see the
> >>> tmp_storage directory grow to 1GB, where it finally blocks and
> >>> recovery of the messages starts hanging:
> >>>
> >>> [...]
> >>> DEBUG | default:memory: usage change from: 2% of available memory, to:
> >>> 1% of available memory
> >>> DEBUG | default:memory:queue://local.queue.myqueue:memory: usage
> >>> change from: 1% of available memory, to: 0% of available memory
> >>> DEBUG | default:memory: usage change from: 1% of available memory, to:
> >>> 0% of available memory
> >>> INFO | cursor for queue://local.queue.myqueue has recovered 50000
> >>> messages. 15% complete
> >>> INFO | cursor for queue://local.queue.myqueue has recovered 100000
> >>> messages. 31% complete
> >>> INFO | cursor for queue://local.queue.myqueue has recovered 150000
> >>> messages. 47% complete
> >>> INFO | cursor for queue://local.queue.myqueue has recovered 200000
> >>> messages. 62% complete
> >>>
> >>> The memory usage goes up to around 75%, where it starts dropping down
> >>> to the 0% from the logfile above. This is where it keeps hanging, with
> >>> the tmp_storage directory at 1GB diskspace.
> >>>
> >>> If I increase the tempUsage to 10GB, the startup works fine
> >>> (tmp_storage becomes about 1.4GB in this case). Can you explain to me,
> >>> why I need to increase tempUsage in order to restart my broker with a
> >>> lot of persistent messages left? As far as I understood the tempUsage
> >>> is for storing non-persistent messages, persistent messages should be
> >>> affected by storeUsage?
> >>>
> >>> Thanks in advance!
> >>>
> >>> Best regards,
> >>> Martin
> >>>
> >>> PS: the relevant parts from my configuration
> >>>
> >>> <broker xmlns="http://activemq.apache.org/schema/core"
> >>> brokerName="localhost" dataDirectory="${activemq.base}/data"
> >>> populateJMSXUserID="true" useJmx="true">
> >>>
> >>> <destinationPolicy>
> >>> <policyMap>
> >>> <policyEntries>
> >>> <policyEntry topic=">" producerFlowControl="false"
> >>> gcInactiveDestinations="false" inactiveTimoutBeforeGC="60000">
> >>> <vmCursor />
> >>> </policyEntry>
> >>> <policyEntry queue=">" producerFlowControl="false"
> >>> gcInactiveDestinations="false" inactiveTimoutBeforeGC="86400000">
> >>> <deadLetterStrategy>
> >>> <individualDeadLetterStrategy queuePrefix="DLQ."
> >>> useQueueForQueueMessages="true"
> >>> processExpired="true" processNonPersistent="false" />
> >>> </deadLetterStrategy>
> >>> <pendingQueuePolicy>
> >>> <fileQueueCursor />
> >>> </pendingQueuePolicy>
> >>> </policyEntry>
> >>> </policyEntries>
> >>> </policyMap>
> >>> </destinationPolicy>
> >>>
> >>> <persistenceAdapter>
> >>> <kahaDB directory="${activemq.base}/data/kahadb"
> >>> checkForCorruptJournalFiles="true" checksumJournalFiles="true"
> >>> concurrentStoreAndDispatchTopics="false"
> >>> concurrentStoreAndDispatchQueues="false"
> >>> />
> >>> </persistenceAdapter>
> >>>
> >>> <systemUsage>
> >>> <systemUsage sendFailIfNoSpace="true">
> >>> <memoryUsage>
> >>> <memoryUsage limit="64 mb" />
> >>> </memoryUsage>
> >>> <storeUsage>
> >>> <storeUsage limit="10 gb" />
> >>> </storeUsage>
> >>> <tempUsage>
> >>> <tempUsage limit="1 gb" />
> >>> </tempUsage>
> >>> </systemUsage>
> >>> </systemUsage>
> >>> </broker>
> >>>
> >>
> >>
> >>
> >> --
> >> http://fusesource.com
> >> http://blog.garytully.com
> >>
> >
>
>
>
> --
> http://fusesource.com
> http://blog.garytully.com
>
Re: ActiveMQ needs tempUsage for recovery of persistent messages on startup?
Posted by Gary Tully <ga...@gmail.com>.
yes.
On 16 November 2011 12:01, Martin C. <ma...@gmx.at> wrote:
> Hi Gary,
>
> thanks for the very fast response. I will change the configuration
> accordingly. So basically you say I should remove the
> <pendingQueuePolicy> as a whole?
>
> Best regards,
> Martin
>
> On Wed, Nov 16, 2011 at 11:34 AM, Gary Tully <ga...@gmail.com> wrote:
>> You should not be using the file pending message cursor in this case.
>> The default store cursor is best, it will use a file cursor for non
>> persistent messages and leave excess (exceed the cache) persistent
>> messages in the store.
>> When it is specified as the default cursor in configuration, it is
>> used for both persistent and non persistent messages.
>>
>> The file pending message cursor replays all messages on startup such
>> that it can do an append to the cursor on the next add but it is only
>> intended for really slow stores, where reading from the local file
>> system is faster than reading from the store. This is not the case
>> with kahaDB.
>>
>> On 16 November 2011 10:26, Martin C. <ma...@gmx.at> wrote:
>>> Hi,
>>>
>>> I a have a question regarding fileQueueCursor and systemUsage on
>>> ActiveMQ 5.5.0: Below you see my configuration, which limits the
>>> memory to 64MB, store usage to 10GB and tempUsage to 1GB.
>>>
>>> I have a KahaDB with about 300000 messages, using 330MB in the kahadb
>>> directory, all messages are persistent. If I start ActiveMQ, I see the
>>> tmp_storage directory grow to 1GB, where it finally blocks and
>>> recovery of the messages starts hanging:
>>>
>>> [...]
>>> DEBUG | default:memory: usage change from: 2% of available memory, to:
>>> 1% of available memory
>>> DEBUG | default:memory:queue://local.queue.myqueue:memory: usage
>>> change from: 1% of available memory, to: 0% of available memory
>>> DEBUG | default:memory: usage change from: 1% of available memory, to:
>>> 0% of available memory
>>> INFO | cursor for queue://local.queue.myqueue has recovered 50000
>>> messages. 15% complete
>>> INFO | cursor for queue://local.queue.myqueue has recovered 100000
>>> messages. 31% complete
>>> INFO | cursor for queue://local.queue.myqueue has recovered 150000
>>> messages. 47% complete
>>> INFO | cursor for queue://local.queue.myqueue has recovered 200000
>>> messages. 62% complete
>>>
>>> The memory usage goes up to around 75%, where it starts dropping down
>>> to the 0% from the logfile above. This is where it keeps hanging, with
>>> the tmp_storage directory at 1GB diskspace.
>>>
>>> If I increase the tempUsage to 10GB, the startup works fine
>>> (tmp_storage becomes about 1.4GB in this case). Can you explain to me,
>>> why I need to increase tempUsage in order to restart my broker with a
>>> lot of persistent messages left? As far as I understood the tempUsage
>>> is for storing non-persistent messages, persistent messages should be
>>> affected by storeUsage?
>>>
>>> Thanks in advance!
>>>
>>> Best regards,
>>> Martin
>>>
>>> PS: the relevant parts from my configuration
>>>
>>> <broker xmlns="http://activemq.apache.org/schema/core"
>>> brokerName="localhost" dataDirectory="${activemq.base}/data"
>>> populateJMSXUserID="true" useJmx="true">
>>>
>>> <destinationPolicy>
>>> <policyMap>
>>> <policyEntries>
>>> <policyEntry topic=">" producerFlowControl="false"
>>> gcInactiveDestinations="false" inactiveTimoutBeforeGC="60000">
>>> <vmCursor />
>>> </policyEntry>
>>> <policyEntry queue=">" producerFlowControl="false"
>>> gcInactiveDestinations="false" inactiveTimoutBeforeGC="86400000">
>>> <deadLetterStrategy>
>>> <individualDeadLetterStrategy queuePrefix="DLQ."
>>> useQueueForQueueMessages="true"
>>> processExpired="true" processNonPersistent="false" />
>>> </deadLetterStrategy>
>>> <pendingQueuePolicy>
>>> <fileQueueCursor />
>>> </pendingQueuePolicy>
>>> </policyEntry>
>>> </policyEntries>
>>> </policyMap>
>>> </destinationPolicy>
>>>
>>> <persistenceAdapter>
>>> <kahaDB directory="${activemq.base}/data/kahadb"
>>> checkForCorruptJournalFiles="true" checksumJournalFiles="true"
>>> concurrentStoreAndDispatchTopics="false"
>>> concurrentStoreAndDispatchQueues="false"
>>> />
>>> </persistenceAdapter>
>>>
>>> <systemUsage>
>>> <systemUsage sendFailIfNoSpace="true">
>>> <memoryUsage>
>>> <memoryUsage limit="64 mb" />
>>> </memoryUsage>
>>> <storeUsage>
>>> <storeUsage limit="10 gb" />
>>> </storeUsage>
>>> <tempUsage>
>>> <tempUsage limit="1 gb" />
>>> </tempUsage>
>>> </systemUsage>
>>> </systemUsage>
>>> </broker>
>>>
>>
>>
>>
>> --
>> http://fusesource.com
>> http://blog.garytully.com
>>
>
--
http://fusesource.com
http://blog.garytully.com
Re: ActiveMQ needs tempUsage for recovery of persistent messages on startup?
Posted by "Martin C." <ma...@gmx.at>.
Hi Gary,
thanks for the very fast response. I will change the configuration
accordingly. So basically you say I should remove the
<pendingQueuePolicy> as a whole?
Best regards,
Martin
On Wed, Nov 16, 2011 at 11:34 AM, Gary Tully <ga...@gmail.com> wrote:
> You should not be using the file pending message cursor in this case.
> The default store cursor is best, it will use a file cursor for non
> persistent messages and leave excess (exceed the cache) persistent
> messages in the store.
> When it is specified as the default cursor in configuration, it is
> used for both persistent and non persistent messages.
>
> The file pending message cursor replays all messages on startup such
> that it can do an append to the cursor on the next add but it is only
> intended for really slow stores, where reading from the local file
> system is faster than reading from the store. This is not the case
> with kahaDB.
>
> On 16 November 2011 10:26, Martin C. <ma...@gmx.at> wrote:
>> Hi,
>>
>> I a have a question regarding fileQueueCursor and systemUsage on
>> ActiveMQ 5.5.0: Below you see my configuration, which limits the
>> memory to 64MB, store usage to 10GB and tempUsage to 1GB.
>>
>> I have a KahaDB with about 300000 messages, using 330MB in the kahadb
>> directory, all messages are persistent. If I start ActiveMQ, I see the
>> tmp_storage directory grow to 1GB, where it finally blocks and
>> recovery of the messages starts hanging:
>>
>> [...]
>> DEBUG | default:memory: usage change from: 2% of available memory, to:
>> 1% of available memory
>> DEBUG | default:memory:queue://local.queue.myqueue:memory: usage
>> change from: 1% of available memory, to: 0% of available memory
>> DEBUG | default:memory: usage change from: 1% of available memory, to:
>> 0% of available memory
>> INFO | cursor for queue://local.queue.myqueue has recovered 50000
>> messages. 15% complete
>> INFO | cursor for queue://local.queue.myqueue has recovered 100000
>> messages. 31% complete
>> INFO | cursor for queue://local.queue.myqueue has recovered 150000
>> messages. 47% complete
>> INFO | cursor for queue://local.queue.myqueue has recovered 200000
>> messages. 62% complete
>>
>> The memory usage goes up to around 75%, where it starts dropping down
>> to the 0% from the logfile above. This is where it keeps hanging, with
>> the tmp_storage directory at 1GB diskspace.
>>
>> If I increase the tempUsage to 10GB, the startup works fine
>> (tmp_storage becomes about 1.4GB in this case). Can you explain to me,
>> why I need to increase tempUsage in order to restart my broker with a
>> lot of persistent messages left? As far as I understood the tempUsage
>> is for storing non-persistent messages, persistent messages should be
>> affected by storeUsage?
>>
>> Thanks in advance!
>>
>> Best regards,
>> Martin
>>
>> PS: the relevant parts from my configuration
>>
>> <broker xmlns="http://activemq.apache.org/schema/core"
>> brokerName="localhost" dataDirectory="${activemq.base}/data"
>> populateJMSXUserID="true" useJmx="true">
>>
>> <destinationPolicy>
>> <policyMap>
>> <policyEntries>
>> <policyEntry topic=">" producerFlowControl="false"
>> gcInactiveDestinations="false" inactiveTimoutBeforeGC="60000">
>> <vmCursor />
>> </policyEntry>
>> <policyEntry queue=">" producerFlowControl="false"
>> gcInactiveDestinations="false" inactiveTimoutBeforeGC="86400000">
>> <deadLetterStrategy>
>> <individualDeadLetterStrategy queuePrefix="DLQ."
>> useQueueForQueueMessages="true"
>> processExpired="true" processNonPersistent="false" />
>> </deadLetterStrategy>
>> <pendingQueuePolicy>
>> <fileQueueCursor />
>> </pendingQueuePolicy>
>> </policyEntry>
>> </policyEntries>
>> </policyMap>
>> </destinationPolicy>
>>
>> <persistenceAdapter>
>> <kahaDB directory="${activemq.base}/data/kahadb"
>> checkForCorruptJournalFiles="true" checksumJournalFiles="true"
>> concurrentStoreAndDispatchTopics="false"
>> concurrentStoreAndDispatchQueues="false"
>> />
>> </persistenceAdapter>
>>
>> <systemUsage>
>> <systemUsage sendFailIfNoSpace="true">
>> <memoryUsage>
>> <memoryUsage limit="64 mb" />
>> </memoryUsage>
>> <storeUsage>
>> <storeUsage limit="10 gb" />
>> </storeUsage>
>> <tempUsage>
>> <tempUsage limit="1 gb" />
>> </tempUsage>
>> </systemUsage>
>> </systemUsage>
>> </broker>
>>
>
>
>
> --
> http://fusesource.com
> http://blog.garytully.com
>
Re: ActiveMQ needs tempUsage for recovery of persistent messages on startup?
Posted by Gary Tully <ga...@gmail.com>.
You should not be using the file pending message cursor in this case.
The default store cursor is best, it will use a file cursor for non
persistent messages and leave excess (exceed the cache) persistent
messages in the store.
When it is specified as the default cursor in configuration, it is
used for both persistent and non persistent messages.
The file pending message cursor replays all messages on startup such
that it can do an append to the cursor on the next add but it is only
intended for really slow stores, where reading from the local file
system is faster than reading from the store. This is not the case
with kahaDB.
On 16 November 2011 10:26, Martin C. <ma...@gmx.at> wrote:
> Hi,
>
> I a have a question regarding fileQueueCursor and systemUsage on
> ActiveMQ 5.5.0: Below you see my configuration, which limits the
> memory to 64MB, store usage to 10GB and tempUsage to 1GB.
>
> I have a KahaDB with about 300000 messages, using 330MB in the kahadb
> directory, all messages are persistent. If I start ActiveMQ, I see the
> tmp_storage directory grow to 1GB, where it finally blocks and
> recovery of the messages starts hanging:
>
> [...]
> DEBUG | default:memory: usage change from: 2% of available memory, to:
> 1% of available memory
> DEBUG | default:memory:queue://local.queue.myqueue:memory: usage
> change from: 1% of available memory, to: 0% of available memory
> DEBUG | default:memory: usage change from: 1% of available memory, to:
> 0% of available memory
> INFO | cursor for queue://local.queue.myqueue has recovered 50000
> messages. 15% complete
> INFO | cursor for queue://local.queue.myqueue has recovered 100000
> messages. 31% complete
> INFO | cursor for queue://local.queue.myqueue has recovered 150000
> messages. 47% complete
> INFO | cursor for queue://local.queue.myqueue has recovered 200000
> messages. 62% complete
>
> The memory usage goes up to around 75%, where it starts dropping down
> to the 0% from the logfile above. This is where it keeps hanging, with
> the tmp_storage directory at 1GB diskspace.
>
> If I increase the tempUsage to 10GB, the startup works fine
> (tmp_storage becomes about 1.4GB in this case). Can you explain to me,
> why I need to increase tempUsage in order to restart my broker with a
> lot of persistent messages left? As far as I understood the tempUsage
> is for storing non-persistent messages, persistent messages should be
> affected by storeUsage?
>
> Thanks in advance!
>
> Best regards,
> Martin
>
> PS: the relevant parts from my configuration
>
> <broker xmlns="http://activemq.apache.org/schema/core"
> brokerName="localhost" dataDirectory="${activemq.base}/data"
> populateJMSXUserID="true" useJmx="true">
>
> <destinationPolicy>
> <policyMap>
> <policyEntries>
> <policyEntry topic=">" producerFlowControl="false"
> gcInactiveDestinations="false" inactiveTimoutBeforeGC="60000">
> <vmCursor />
> </policyEntry>
> <policyEntry queue=">" producerFlowControl="false"
> gcInactiveDestinations="false" inactiveTimoutBeforeGC="86400000">
> <deadLetterStrategy>
> <individualDeadLetterStrategy queuePrefix="DLQ."
> useQueueForQueueMessages="true"
> processExpired="true" processNonPersistent="false" />
> </deadLetterStrategy>
> <pendingQueuePolicy>
> <fileQueueCursor />
> </pendingQueuePolicy>
> </policyEntry>
> </policyEntries>
> </policyMap>
> </destinationPolicy>
>
> <persistenceAdapter>
> <kahaDB directory="${activemq.base}/data/kahadb"
> checkForCorruptJournalFiles="true" checksumJournalFiles="true"
> concurrentStoreAndDispatchTopics="false"
> concurrentStoreAndDispatchQueues="false"
> />
> </persistenceAdapter>
>
> <systemUsage>
> <systemUsage sendFailIfNoSpace="true">
> <memoryUsage>
> <memoryUsage limit="64 mb" />
> </memoryUsage>
> <storeUsage>
> <storeUsage limit="10 gb" />
> </storeUsage>
> <tempUsage>
> <tempUsage limit="1 gb" />
> </tempUsage>
> </systemUsage>
> </systemUsage>
> </broker>
>
--
http://fusesource.com
http://blog.garytully.com