You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by Elliot Barlas <el...@gmail.com> on 2009/06/11 02:48:06 UTC
Why do persistent messages exhaust memory and lock out consumers?
Scenario:
-Minimal AMQ message broker with persistence enabled and default usage
values of 64 MB memory limit, 10 GB temp store limit, and 100 GB persistent
store limit
-AMQ producer client sending 100 KB persistent messages to queue X in a loop
with a short sleep (roughly 10 msgs/second)
Observations:
-MemoryPercentUsage reaches 70% after a few minutes and remains there
-StorePercentUsage remains at 0 (since persistent store limit is so high)
-Disk usage under persistent data directory increases as expected
Why is the memory percent usage increasing so rapidly? Shouldn't messages
go directly to the persistent store? If memory is used, shouldn't it be
flushed once it reaches a certain threshold?
Scenario continued:
-A second AMQ producer client sending 100 KB persistent messages to queue Y
in a loop
-AMQ consumer client consuming messages from queue Y in a loop
Observations continued:
-MemoryPercentUsage remains at 70%
-StorePercentUsage remains at 0
-Disk usage under persistent data directory continues to increase
-NO messages dequeued from queue Y!
Shockingly, messages are NOT consumed from queue Y in the scenario described
above. Once the 70% memory usage threshold is reached, messages cannot be
consumed from queue Y. After further testing, I have seen the messages can
be consumed from queue X, which decreased the memory percent used, and
restores the system (messages can then be consumed from other queues).
Why is this happening? Has anyone else seen this?
The same test with non-persistent messages is not problematic. The memory
percent usage increases, but once it reaches 70% the messages are flushed to
the temp store and the percent usage is reduce to 0%
Thanks,
Elliot
--
View this message in context: http://www.nabble.com/Why-do-persistent-messages-exhaust-memory-and-lock-out-consumers--tp23973571p23973571.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Why do persistent messages exhaust memory and lock out consumers?
Posted by Gary Tully <ga...@gmail.com>.
For scenario 1, messages are cached in memory up to a limit, when caching
stops, message are comsumed from the cache till it is exhausted and then
dequeues resume from the store cursors. You can enable the caching with
cache=true/false in a PolicyEntry.
Can you try scenario 2 with 5.3-SNAPSHOT. I recall an interaction with the
usage manager and listeners that could account for the behavior you are
seeing.
If the behavior persists with 5.3-SNAPSHOT, please submit your test case and
activemq,xml in a JIRA issue so it can be investigated.
2009/6/11 Elliot Barlas <el...@gmail.com>
>
> AMQ version 5.2.0
>
>
> rajdavies wrote:
> >
> > which version are you using ?
> > On 11 Jun 2009, at 01:48, Elliot Barlas wrote:
> >
> >>
> >> Scenario:
> >> -Minimal AMQ message broker with persistence enabled and default usage
> >> values of 64 MB memory limit, 10 GB temp store limit, and 100 GB
> >> persistent
> >> store limit
> >> -AMQ producer client sending 100 KB persistent messages to queue X
> >> in a loop
> >> with a short sleep (roughly 10 msgs/second)
> >>
> >> Observations:
> >> -MemoryPercentUsage reaches 70% after a few minutes and remains there
> >> -StorePercentUsage remains at 0 (since persistent store limit is so
> >> high)
> >> -Disk usage under persistent data directory increases as expected
> >>
> >>
> >> Why is the memory percent usage increasing so rapidly? Shouldn't
> >> messages
> >> go directly to the persistent store? If memory is used, shouldn't
> >> it be
> >> flushed once it reaches a certain threshold?
> >>
> >>
> >> Scenario continued:
> >> -A second AMQ producer client sending 100 KB persistent messages to
> >> queue Y
> >> in a loop
> >> -AMQ consumer client consuming messages from queue Y in a loop
> >>
> >> Observations continued:
> >> -MemoryPercentUsage remains at 70%
> >> -StorePercentUsage remains at 0
> >> -Disk usage under persistent data directory continues to increase
> >> -NO messages dequeued from queue Y!
> >>
> >>
> >> Shockingly, messages are NOT consumed from queue Y in the scenario
> >> described
> >> above. Once the 70% memory usage threshold is reached, messages
> >> cannot be
> >> consumed from queue Y. After further testing, I have seen the
> >> messages can
> >> be consumed from queue X, which decreased the memory percent used, and
> >> restores the system (messages can then be consumed from other queues).
> >>
> >> Why is this happening? Has anyone else seen this?
> >>
> >> The same test with non-persistent messages is not problematic. The
> >> memory
> >> percent usage increases, but once it reaches 70% the messages are
> >> flushed to
> >> the temp store and the percent usage is reduce to 0%
> >>
> >> Thanks,
> >> Elliot
> >> --
> >> View this message in context:
> >>
> http://www.nabble.com/Why-do-persistent-messages-exhaust-memory-and-lock-out-consumers--tp23973571p23973571.html
> >> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
> >>
> >
> >
> >
>
> --
> View this message in context:
> http://www.nabble.com/Why-do-persistent-messages-exhaust-memory-and-lock-out-consumers--tp23973571p23984660.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
>
--
http://blog.garytully.com
Open Source Integration
http://fusesource.com
Re: Why do persistent messages exhaust memory and lock out
consumers?
Posted by Elliot Barlas <el...@gmail.com>.
AMQ version 5.2.0
rajdavies wrote:
>
> which version are you using ?
> On 11 Jun 2009, at 01:48, Elliot Barlas wrote:
>
>>
>> Scenario:
>> -Minimal AMQ message broker with persistence enabled and default usage
>> values of 64 MB memory limit, 10 GB temp store limit, and 100 GB
>> persistent
>> store limit
>> -AMQ producer client sending 100 KB persistent messages to queue X
>> in a loop
>> with a short sleep (roughly 10 msgs/second)
>>
>> Observations:
>> -MemoryPercentUsage reaches 70% after a few minutes and remains there
>> -StorePercentUsage remains at 0 (since persistent store limit is so
>> high)
>> -Disk usage under persistent data directory increases as expected
>>
>>
>> Why is the memory percent usage increasing so rapidly? Shouldn't
>> messages
>> go directly to the persistent store? If memory is used, shouldn't
>> it be
>> flushed once it reaches a certain threshold?
>>
>>
>> Scenario continued:
>> -A second AMQ producer client sending 100 KB persistent messages to
>> queue Y
>> in a loop
>> -AMQ consumer client consuming messages from queue Y in a loop
>>
>> Observations continued:
>> -MemoryPercentUsage remains at 70%
>> -StorePercentUsage remains at 0
>> -Disk usage under persistent data directory continues to increase
>> -NO messages dequeued from queue Y!
>>
>>
>> Shockingly, messages are NOT consumed from queue Y in the scenario
>> described
>> above. Once the 70% memory usage threshold is reached, messages
>> cannot be
>> consumed from queue Y. After further testing, I have seen the
>> messages can
>> be consumed from queue X, which decreased the memory percent used, and
>> restores the system (messages can then be consumed from other queues).
>>
>> Why is this happening? Has anyone else seen this?
>>
>> The same test with non-persistent messages is not problematic. The
>> memory
>> percent usage increases, but once it reaches 70% the messages are
>> flushed to
>> the temp store and the percent usage is reduce to 0%
>>
>> Thanks,
>> Elliot
>> --
>> View this message in context:
>> http://www.nabble.com/Why-do-persistent-messages-exhaust-memory-and-lock-out-consumers--tp23973571p23973571.html
>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>
>
>
>
--
View this message in context: http://www.nabble.com/Why-do-persistent-messages-exhaust-memory-and-lock-out-consumers--tp23973571p23984660.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Why do persistent messages exhaust memory and lock out consumers?
Posted by Rob Davies <ra...@gmail.com>.
which version are you using ?
On 11 Jun 2009, at 01:48, Elliot Barlas wrote:
>
> Scenario:
> -Minimal AMQ message broker with persistence enabled and default usage
> values of 64 MB memory limit, 10 GB temp store limit, and 100 GB
> persistent
> store limit
> -AMQ producer client sending 100 KB persistent messages to queue X
> in a loop
> with a short sleep (roughly 10 msgs/second)
>
> Observations:
> -MemoryPercentUsage reaches 70% after a few minutes and remains there
> -StorePercentUsage remains at 0 (since persistent store limit is so
> high)
> -Disk usage under persistent data directory increases as expected
>
>
> Why is the memory percent usage increasing so rapidly? Shouldn't
> messages
> go directly to the persistent store? If memory is used, shouldn't
> it be
> flushed once it reaches a certain threshold?
>
>
> Scenario continued:
> -A second AMQ producer client sending 100 KB persistent messages to
> queue Y
> in a loop
> -AMQ consumer client consuming messages from queue Y in a loop
>
> Observations continued:
> -MemoryPercentUsage remains at 70%
> -StorePercentUsage remains at 0
> -Disk usage under persistent data directory continues to increase
> -NO messages dequeued from queue Y!
>
>
> Shockingly, messages are NOT consumed from queue Y in the scenario
> described
> above. Once the 70% memory usage threshold is reached, messages
> cannot be
> consumed from queue Y. After further testing, I have seen the
> messages can
> be consumed from queue X, which decreased the memory percent used, and
> restores the system (messages can then be consumed from other queues).
>
> Why is this happening? Has anyone else seen this?
>
> The same test with non-persistent messages is not problematic. The
> memory
> percent usage increases, but once it reaches 70% the messages are
> flushed to
> the temp store and the percent usage is reduce to 0%
>
> Thanks,
> Elliot
> --
> View this message in context: http://www.nabble.com/Why-do-persistent-messages-exhaust-memory-and-lock-out-consumers--tp23973571p23973571.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>