You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by "Frugia, Kirby A" <ki...@hp.com> on 2008/12/02 16:37:55 UTC
RE: Broker not releasing memory
Yes, this is exactly what we are seeing. Any ideas?
-----Original Message-----
From: Will van der Leij [mailto:will@stonethree.com]
Sent: Thursday, November 27, 2008 11:42 PM
To: users@activemq.apache.org
Subject: Re: Broker not releasing memory
We see similar behaviour in a slightly different setup so I'd like to
elaborate a bit in the hope of finding out if this is either intended
behaviour or a memory leak.
Essentially, with a single or multiple producers and a single or multiple
consumers on a single topic:
- We send messages faster than the consumer can read asynchronously
- When the send queue is larger than the consumer's prefetch buffer then
we correctly see messages filling up the broker memory
(memoryPercentUSage in JMX)
- Killing the consumer (cleanly through a close or not-cleanly with a
terminate signal)
results in the memoryPercentUsage not being released
- even though there are no more consumers on the Topic
- this memory is never released until the broker is restarted
I've tried it with persistence on and off, all the retroactive recovery
options on and off, flow control on and off, caching on and off,
asynchronous dispatching on and off, noLocal flag on and off etc.
This is a very typical case of a consumer dying with inflight messages on
the broker. Is the behaviour of the messages not being cleaned up or
recovered intended (i.e. they are there for a reason) or is there soething
wrnog with the setup.
Many thanks
Will van der Leij
Frugia, Kirby A wrote:
>
> Hi All,
>
> Sorry I dual-posted in the dev list. I think I sent to the wrong one...
>
> We are seeing an issue with our broker not releasing memory on topics.
>
> Setup:
> * Active MQ 5.1 (out of the box)
> * Persistent messages sent by publishers
> * Non-durable topics
>
> We were seeing an issue in production with slow consumers causing the
> broker to run out of memory.
>
> We wrote a test app, which has one publisher and one subscriber to the
> same topic. The publisher sends messages on this topic frequently. The
> subscriber, upon receiving a message, goes to sleep for 10 minutes.
>
> Very quickly, the broker runs out of memory. When this happens, our
> publishers can no longer send messages; the send is blocked. This is
> expected. However, if we kill our application (which cleanly disconnects
> from the broker), the broker's memory usage (MemoryPercentUsage) does not
> go back down. Also, any new apps that startup will have their publishers
> blocked.
>
> Shouldn't the broker release the memory associated with these messages?
>
> Thanks,
> Kirby
>
>
>
--
View this message in context: http://www.nabble.com/Broker-not-releasing-memory-tp20662078p20730116.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Broker not releasing memory
Posted by Gary Tully <ga...@gmail.com>.
If you can construct a little failing junit test case and attach it to
a jira issue it would be fantastic.
2008/12/9 Will van der Leij <wi...@stonethree.com>:
>
> I've experienced this on 5.1.0, 5.2.0 and a 5.3.0 snapshot from 2 weeks back.
>
>
> Frugia, Kirby A wrote:
>>
>> 5.1.0. I believe it's happening with 5.2.0, too, but have not thoroughly
>> tested.
>>
>> The problem is that the broker runs out of memory. At the point in time
>> of the failure, no more messages could be sent anyway because the broker
>> is out of memory.
>>
>> So we're less worried about blocking producers and more worried about how
>> to maintain the health of the broker. We set the message queue limits to
>> prevent more than X messages on a topic for each consumer. The queue
>> limits are only set to prevent the broker from running out of memory if a
>> particular consumer starts blocking. We are hoping that if a consumer
>> dies, the queue limits will at least keep any message queues for the dead
>> consumer from going out of control.
>>
>> -----Original Message-----
>> From: Bruce Snyder [mailto:bruce.snyder@gmail.com]
>> Sent: Monday, December 08, 2008 10:17 AM
>> To: users@activemq.apache.org
>> Subject: Re: Broker not releasing memory
>>
>> On Mon, Dec 8, 2008 at 7:28 AM, Frugia, Kirby A <ki...@hp.com>
>> wrote:
>>> I can reproduce this with the non-embedded, out-of-the-box broker. I
>>> also think it's an under-the-hood problem.
>>>
>>> Until we get a fix, we've set pending message queue limits on our busy
>>> topics. So, for example, the broker will only queue up 500 messages on a
>>> particular topic per consumer. This will prevent things from getting too
>>> out of hand on a single client crash, but is not the solution to the
>>> problem.
>>
>> What version of ActiveMQ are you using? Also, have you disabled
>> producer flow control to allow the producer to continue to send even
>> when the consumer is going very slow?
>>
>> Bruce
>> --
>> perl -e 'print
>> unpack("u30","D0G)U8V4\@4VYY9&5R\"F)R=6-E+G-N>61E<D\!G;6%I;\"YC;VT*"
>> );'
>>
>> Apache ActiveMQ - http://activemq.org/
>> Apache Camel - http://activemq.org/camel/
>> Apache ServiceMix - http://servicemix.org/
>>
>> Blog: http://bruceblog.org/
>>
>>
>
> --
> View this message in context: http://www.nabble.com/Broker-not-releasing-memory-tp20662078p20910288.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
>
--
http://blog.garytully.com
Open Source SOA
http://FUSESource.com
RE: Broker not releasing memory
Posted by Will van der Leij <wi...@stonethree.com>.
I've experienced this on 5.1.0, 5.2.0 and a 5.3.0 snapshot from 2 weeks back.
Frugia, Kirby A wrote:
>
> 5.1.0. I believe it's happening with 5.2.0, too, but have not thoroughly
> tested.
>
> The problem is that the broker runs out of memory. At the point in time
> of the failure, no more messages could be sent anyway because the broker
> is out of memory.
>
> So we're less worried about blocking producers and more worried about how
> to maintain the health of the broker. We set the message queue limits to
> prevent more than X messages on a topic for each consumer. The queue
> limits are only set to prevent the broker from running out of memory if a
> particular consumer starts blocking. We are hoping that if a consumer
> dies, the queue limits will at least keep any message queues for the dead
> consumer from going out of control.
>
> -----Original Message-----
> From: Bruce Snyder [mailto:bruce.snyder@gmail.com]
> Sent: Monday, December 08, 2008 10:17 AM
> To: users@activemq.apache.org
> Subject: Re: Broker not releasing memory
>
> On Mon, Dec 8, 2008 at 7:28 AM, Frugia, Kirby A <ki...@hp.com>
> wrote:
>> I can reproduce this with the non-embedded, out-of-the-box broker. I
>> also think it's an under-the-hood problem.
>>
>> Until we get a fix, we've set pending message queue limits on our busy
>> topics. So, for example, the broker will only queue up 500 messages on a
>> particular topic per consumer. This will prevent things from getting too
>> out of hand on a single client crash, but is not the solution to the
>> problem.
>
> What version of ActiveMQ are you using? Also, have you disabled
> producer flow control to allow the producer to continue to send even
> when the consumer is going very slow?
>
> Bruce
> --
> perl -e 'print
> unpack("u30","D0G)U8V4\@4VYY9&5R\"F)R=6-E+G-N>61E<D\!G;6%I;\"YC;VT*"
> );'
>
> Apache ActiveMQ - http://activemq.org/
> Apache Camel - http://activemq.org/camel/
> Apache ServiceMix - http://servicemix.org/
>
> Blog: http://bruceblog.org/
>
>
--
View this message in context: http://www.nabble.com/Broker-not-releasing-memory-tp20662078p20910288.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
RE: Broker not releasing memory
Posted by "Frugia, Kirby A" <ki...@hp.com>.
5.1.0. I believe it's happening with 5.2.0, too, but have not thoroughly tested.
The problem is that the broker runs out of memory. At the point in time of the failure, no more messages could be sent anyway because the broker is out of memory.
So we're less worried about blocking producers and more worried about how to maintain the health of the broker. We set the message queue limits to prevent more than X messages on a topic for each consumer. The queue limits are only set to prevent the broker from running out of memory if a particular consumer starts blocking. We are hoping that if a consumer dies, the queue limits will at least keep any message queues for the dead consumer from going out of control.
-----Original Message-----
From: Bruce Snyder [mailto:bruce.snyder@gmail.com]
Sent: Monday, December 08, 2008 10:17 AM
To: users@activemq.apache.org
Subject: Re: Broker not releasing memory
On Mon, Dec 8, 2008 at 7:28 AM, Frugia, Kirby A <ki...@hp.com> wrote:
> I can reproduce this with the non-embedded, out-of-the-box broker. I also think it's an under-the-hood problem.
>
> Until we get a fix, we've set pending message queue limits on our busy topics. So, for example, the broker will only queue up 500 messages on a particular topic per consumer. This will prevent things from getting too out of hand on a single client crash, but is not the solution to the problem.
What version of ActiveMQ are you using? Also, have you disabled
producer flow control to allow the producer to continue to send even
when the consumer is going very slow?
Bruce
--
perl -e 'print unpack("u30","D0G)U8V4\@4VYY9&5R\"F)R=6-E+G-N>61E<D\!G;6%I;\"YC;VT*"
);'
Apache ActiveMQ - http://activemq.org/
Apache Camel - http://activemq.org/camel/
Apache ServiceMix - http://servicemix.org/
Blog: http://bruceblog.org/
Re: Broker not releasing memory
Posted by Bruce Snyder <br...@gmail.com>.
On Mon, Dec 8, 2008 at 7:28 AM, Frugia, Kirby A <ki...@hp.com> wrote:
> I can reproduce this with the non-embedded, out-of-the-box broker. I also think it's an under-the-hood problem.
>
> Until we get a fix, we've set pending message queue limits on our busy topics. So, for example, the broker will only queue up 500 messages on a particular topic per consumer. This will prevent things from getting too out of hand on a single client crash, but is not the solution to the problem.
What version of ActiveMQ are you using? Also, have you disabled
producer flow control to allow the producer to continue to send even
when the consumer is going very slow?
Bruce
--
perl -e 'print unpack("u30","D0G)U8V4\@4VYY9&5R\"F)R=6-E+G-N>61E<D\!G;6%I;\"YC;VT*"
);'
Apache ActiveMQ - http://activemq.org/
Apache Camel - http://activemq.org/camel/
Apache ServiceMix - http://servicemix.org/
Blog: http://bruceblog.org/
RE: Broker not releasing memory
Posted by "Frugia, Kirby A" <ki...@hp.com>.
I can reproduce this with the non-embedded, out-of-the-box broker. I also think it's an under-the-hood problem.
Until we get a fix, we've set pending message queue limits on our busy topics. So, for example, the broker will only queue up 500 messages on a particular topic per consumer. This will prevent things from getting too out of hand on a single client crash, but is not the solution to the problem.
-Kirby
-----Original Message-----
From: Will van der Leij [mailto:will@stonethree.com]
Sent: Monday, December 08, 2008 12:57 AM
To: users@activemq.apache.org
Subject: Re: Broker not releasing memory
I use an embedded broker too and have gone through all the possible
iterations of config options for threading, acknowledgements and message
storage.
Alas, an apparent "memory leak" still persists with slow consumers that
close/die with messages still in-flight/pending. I've had several other
people report the same problem with different clients leading me to believe
that there must be something odd under the hood.
Rick Blair wrote:
>
> Hi Will,
>
> We use an embedded broker Here is the url.
>
> tcp://{0}:{1}?jms.useAsyncSend=false&jms.optimizeAcknowledge=false&jms.prefe
> tchPolicy.all=1000";
>
> The problem started when optimizeAcknowledge=true was used. Also I cannt
> get old messages in the topics to timeout. Some do, most don¹t.
>
>
> Hope this helps.
>
> Rick
> --
> Tìoraidh!
>
> Rick Blair
> Associate Technical Fellow
> Boeing Phantom Works
> Network Systems Technology
> Information Management Program
> M/S: 42-50
> Voice: (206) 544-1610
>
>
>
>> From: Will van der Leij <wi...@stonethree.com>
>> Reply-To: <us...@activemq.apache.org>
>> Date: Wed, 3 Dec 2008 23:32:31 -0800 (PST)
>> To: <us...@activemq.apache.org>
>> Subject: Re: Broker not releasing memory
>>
>>
>> Hi Rick,
>> As I understand optimizeAcknowledge is disabled by default.
>> Unfortunately,
>> setting it explicitely has not had an affect on our memory usage.
>> Could you perhaps briefly describe the rest of your configuration? I'm
>> presuming that it is a combination of options that would do the trick.
>>
>> Regards,
>> Will
>>
>>
>>
>> Rick Blair wrote:
>>>
>>> Hi Kirby,
>>>
>>> We have seen exactly the same behaviour. As a work around we set
>>> optimizeAcknowdlege to false. I did not run down the root cause,
>>> however.
>>>
>>> --
>>> Cheers!
>>>
>>> Rick Blair
>>>
>>>
>>>> From: "Frugia, Kirby A" <ki...@hp.com>
>>>> Reply-To: <us...@activemq.apache.org>
>>>> Date: Wed, 3 Dec 2008 15:43:30 +0000
>>>> To: "users@activemq.apache.org" <us...@activemq.apache.org>
>>>> Conversation: Broker not releasing memory
>>>> Subject: RE: Broker not releasing memory
>>>>
>>>> This is fairly easy to reproduce if you create one app which sends
>>>> messages
>>>> quickly and another which consumes the messages and sleeps before
>>>> ack'ing.
>>>> Once you see broker memory usage going up (through activemq-admin),
>>>> kill
>>>> the
>>>> consumer application. Then run activemq-admin again some time later
>>>> and
>>>> you'll see the same memorypercentusage.
>>>>
>>>> Or, let the consumer run the broker out of memory and you'll see the
>>>> producer
>>>> get blocked. Kill both apps and you'll never see memory go back down
>>>> again
>>>> and you'll never be able to send messages again.
>>>>
>>>> Thanks,
>>>> Kirby
>>>>
>>>> -----Original Message-----
>>>> From: Will van der Leij [mailto:will@stonethree.com]
>>>> Sent: Wednesday, December 03, 2008 12:17 AM
>>>> To: users@activemq.apache.org
>>>> Subject: RE: Broker not releasing memory
>>>>
>>>>
>>>> Well, I'm reticent to suggest something going on under the hood without
>>>> taking a good look at it, however, if there is something wrong in the
>>>> broker
>>>> then I presume it has to do with a "sweeper"-like thread that
>>>> periodically
>>>> cleans up queues and topics that are left pending and consumerless
>>>> (similar
>>>> to how sweeper threads would clean up expired messages etc.
>>>>
>>>> Then again, we are the only two that I know of describing this
>>>> behaviour.
>>>> This suggests that either there is something bizarrely odd about our
>>>> configurations or we are, indeed, correct :)
>>>>
>>>>
>>>>
>>>> Frugia, Kirby A wrote:
>>>>>
>>>>> Yes, this is exactly what we are seeing. Any ideas?
>>>>>
>>>>> -----Original Message-----
>>>>> From: Will van der Leij [mailto:will@stonethree.com]
>>>>> Sent: Thursday, November 27, 2008 11:42 PM
>>>>> To: users@activemq.apache.org
>>>>> Subject: Re: Broker not releasing memory
>>>>>
>>>>>
>>>>> We see similar behaviour in a slightly different setup so I'd like to
>>>>> elaborate a bit in the hope of finding out if this is either intended
>>>>> behaviour or a memory leak.
>>>>>
>>>>> Essentially, with a single or multiple producers and a single or
>>>>> multiple
>>>>> consumers on a single topic:
>>>>> - We send messages faster than the consumer can read asynchronously
>>>>> - When the send queue is larger than the consumer's prefetch buffer
>>>>> then
>>>>> we correctly see messages filling up the broker memory
>>>>> (memoryPercentUSage in JMX)
>>>>> - Killing the consumer (cleanly through a close or not-cleanly with a
>>>>> terminate signal)
>>>>> results in the memoryPercentUsage not being released
>>>>> - even though there are no more consumers on the Topic
>>>>> - this memory is never released until the broker is restarted
>>>>>
>>>>> I've tried it with persistence on and off, all the retroactive
>>>>> recovery
>>>>> options on and off, flow control on and off, caching on and off,
>>>>> asynchronous dispatching on and off, noLocal flag on and off etc.
>>>>>
>>>>> This is a very typical case of a consumer dying with inflight messages
>>>>> on
>>>>> the broker. Is the behaviour of the messages not being cleaned up or
>>>>> recovered intended (i.e. they are there for a reason) or is there
>>>>> soething
>>>>> wrnog with the setup.
>>>>>
>>>>> Many thanks
>>>>> Will van der Leij
>>>>>
>>>>>
>>>>>
>>>>> Frugia, Kirby A wrote:
>>>>>>
>>>>>> Hi All,
>>>>>>
>>>>>> Sorry I dual-posted in the dev list. I think I sent to the wrong
>>>>>> one...
>>>>>>
>>>>>> We are seeing an issue with our broker not releasing memory on
>>>>>> topics.
>>>>>>
>>>>>> Setup:
>>>>>> * Active MQ 5.1 (out of the box)
>>>>>> * Persistent messages sent by publishers
>>>>>> * Non-durable topics
>>>>>>
>>>>>> We were seeing an issue in production with slow consumers causing the
>>>>>> broker to run out of memory.
>>>>>>
>>>>>> We wrote a test app, which has one publisher and one subscriber to
>>>>>> the
>>>>>> same topic. The publisher sends messages on this topic frequently.
>>>>>> The
>>>>>> subscriber, upon receiving a message, goes to sleep for 10 minutes.
>>>>>>
>>>>>> Very quickly, the broker runs out of memory. When this happens, our
>>>>>> publishers can no longer send messages; the send is blocked. This is
>>>>>> expected. However, if we kill our application (which cleanly
>>>>>> disconnects
>>>>>> from the broker), the broker's memory usage (MemoryPercentUsage) does
>>>>>> not
>>>>>> go back down. Also, any new apps that startup will have their
>>>>>> publishers
>>>>>> blocked.
>>>>>>
>>>>>> Shouldn't the broker release the memory associated with these
>>>>>> messages?
>>>>>>
>>>>>> Thanks,
>>>>>> Kirby
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> View this message in context:
>>>>> http://www.nabble.com/Broker-not-releasing-memory-tp20662078p20730116.html
>>>>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>>>>
>>>>>
>>>>>
>>>>
>>>> --
>>>> View this message in context:
>>>> http://www.nabble.com/Broker-not-releasing-memory-tp20662078p20808585.html
>>>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>>>
>>>
>>>
>>>
>>
>> --
>> View this message in context:
>> http://www.nabble.com/Broker-not-releasing-memory-tp20662078p20828589.html
>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>
>
>
>
--
View this message in context: http://www.nabble.com/Broker-not-releasing-memory-tp20662078p20890719.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Broker not releasing memory
Posted by Will van der Leij <wi...@stonethree.com>.
I use an embedded broker too and have gone through all the possible
iterations of config options for threading, acknowledgements and message
storage.
Alas, an apparent "memory leak" still persists with slow consumers that
close/die with messages still in-flight/pending. I've had several other
people report the same problem with different clients leading me to believe
that there must be something odd under the hood.
Rick Blair wrote:
>
> Hi Will,
>
> We use an embedded broker Here is the url.
>
> tcp://{0}:{1}?jms.useAsyncSend=false&jms.optimizeAcknowledge=false&jms.prefe
> tchPolicy.all=1000";
>
> The problem started when optimizeAcknowledge=true was used. Also I cannt
> get old messages in the topics to timeout. Some do, most don¹t.
>
>
> Hope this helps.
>
> Rick
> --
> Tìoraidh!
>
> Rick Blair
> Associate Technical Fellow
> Boeing Phantom Works
> Network Systems Technology
> Information Management Program
> M/S: 42-50
> Voice: (206) 544-1610
>
>
>
>> From: Will van der Leij <wi...@stonethree.com>
>> Reply-To: <us...@activemq.apache.org>
>> Date: Wed, 3 Dec 2008 23:32:31 -0800 (PST)
>> To: <us...@activemq.apache.org>
>> Subject: Re: Broker not releasing memory
>>
>>
>> Hi Rick,
>> As I understand optimizeAcknowledge is disabled by default.
>> Unfortunately,
>> setting it explicitely has not had an affect on our memory usage.
>> Could you perhaps briefly describe the rest of your configuration? I'm
>> presuming that it is a combination of options that would do the trick.
>>
>> Regards,
>> Will
>>
>>
>>
>> Rick Blair wrote:
>>>
>>> Hi Kirby,
>>>
>>> We have seen exactly the same behaviour. As a work around we set
>>> optimizeAcknowdlege to false. I did not run down the root cause,
>>> however.
>>>
>>> --
>>> Cheers!
>>>
>>> Rick Blair
>>>
>>>
>>>> From: "Frugia, Kirby A" <ki...@hp.com>
>>>> Reply-To: <us...@activemq.apache.org>
>>>> Date: Wed, 3 Dec 2008 15:43:30 +0000
>>>> To: "users@activemq.apache.org" <us...@activemq.apache.org>
>>>> Conversation: Broker not releasing memory
>>>> Subject: RE: Broker not releasing memory
>>>>
>>>> This is fairly easy to reproduce if you create one app which sends
>>>> messages
>>>> quickly and another which consumes the messages and sleeps before
>>>> ack'ing.
>>>> Once you see broker memory usage going up (through activemq-admin),
>>>> kill
>>>> the
>>>> consumer application. Then run activemq-admin again some time later
>>>> and
>>>> you'll see the same memorypercentusage.
>>>>
>>>> Or, let the consumer run the broker out of memory and you'll see the
>>>> producer
>>>> get blocked. Kill both apps and you'll never see memory go back down
>>>> again
>>>> and you'll never be able to send messages again.
>>>>
>>>> Thanks,
>>>> Kirby
>>>>
>>>> -----Original Message-----
>>>> From: Will van der Leij [mailto:will@stonethree.com]
>>>> Sent: Wednesday, December 03, 2008 12:17 AM
>>>> To: users@activemq.apache.org
>>>> Subject: RE: Broker not releasing memory
>>>>
>>>>
>>>> Well, I'm reticent to suggest something going on under the hood without
>>>> taking a good look at it, however, if there is something wrong in the
>>>> broker
>>>> then I presume it has to do with a "sweeper"-like thread that
>>>> periodically
>>>> cleans up queues and topics that are left pending and consumerless
>>>> (similar
>>>> to how sweeper threads would clean up expired messages etc.
>>>>
>>>> Then again, we are the only two that I know of describing this
>>>> behaviour.
>>>> This suggests that either there is something bizarrely odd about our
>>>> configurations or we are, indeed, correct :)
>>>>
>>>>
>>>>
>>>> Frugia, Kirby A wrote:
>>>>>
>>>>> Yes, this is exactly what we are seeing. Any ideas?
>>>>>
>>>>> -----Original Message-----
>>>>> From: Will van der Leij [mailto:will@stonethree.com]
>>>>> Sent: Thursday, November 27, 2008 11:42 PM
>>>>> To: users@activemq.apache.org
>>>>> Subject: Re: Broker not releasing memory
>>>>>
>>>>>
>>>>> We see similar behaviour in a slightly different setup so I'd like to
>>>>> elaborate a bit in the hope of finding out if this is either intended
>>>>> behaviour or a memory leak.
>>>>>
>>>>> Essentially, with a single or multiple producers and a single or
>>>>> multiple
>>>>> consumers on a single topic:
>>>>> - We send messages faster than the consumer can read asynchronously
>>>>> - When the send queue is larger than the consumer's prefetch buffer
>>>>> then
>>>>> we correctly see messages filling up the broker memory
>>>>> (memoryPercentUSage in JMX)
>>>>> - Killing the consumer (cleanly through a close or not-cleanly with a
>>>>> terminate signal)
>>>>> results in the memoryPercentUsage not being released
>>>>> - even though there are no more consumers on the Topic
>>>>> - this memory is never released until the broker is restarted
>>>>>
>>>>> I've tried it with persistence on and off, all the retroactive
>>>>> recovery
>>>>> options on and off, flow control on and off, caching on and off,
>>>>> asynchronous dispatching on and off, noLocal flag on and off etc.
>>>>>
>>>>> This is a very typical case of a consumer dying with inflight messages
>>>>> on
>>>>> the broker. Is the behaviour of the messages not being cleaned up or
>>>>> recovered intended (i.e. they are there for a reason) or is there
>>>>> soething
>>>>> wrnog with the setup.
>>>>>
>>>>> Many thanks
>>>>> Will van der Leij
>>>>>
>>>>>
>>>>>
>>>>> Frugia, Kirby A wrote:
>>>>>>
>>>>>> Hi All,
>>>>>>
>>>>>> Sorry I dual-posted in the dev list. I think I sent to the wrong
>>>>>> one...
>>>>>>
>>>>>> We are seeing an issue with our broker not releasing memory on
>>>>>> topics.
>>>>>>
>>>>>> Setup:
>>>>>> * Active MQ 5.1 (out of the box)
>>>>>> * Persistent messages sent by publishers
>>>>>> * Non-durable topics
>>>>>>
>>>>>> We were seeing an issue in production with slow consumers causing the
>>>>>> broker to run out of memory.
>>>>>>
>>>>>> We wrote a test app, which has one publisher and one subscriber to
>>>>>> the
>>>>>> same topic. The publisher sends messages on this topic frequently.
>>>>>> The
>>>>>> subscriber, upon receiving a message, goes to sleep for 10 minutes.
>>>>>>
>>>>>> Very quickly, the broker runs out of memory. When this happens, our
>>>>>> publishers can no longer send messages; the send is blocked. This is
>>>>>> expected. However, if we kill our application (which cleanly
>>>>>> disconnects
>>>>>> from the broker), the broker's memory usage (MemoryPercentUsage) does
>>>>>> not
>>>>>> go back down. Also, any new apps that startup will have their
>>>>>> publishers
>>>>>> blocked.
>>>>>>
>>>>>> Shouldn't the broker release the memory associated with these
>>>>>> messages?
>>>>>>
>>>>>> Thanks,
>>>>>> Kirby
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> View this message in context:
>>>>> http://www.nabble.com/Broker-not-releasing-memory-tp20662078p20730116.html
>>>>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>>>>
>>>>>
>>>>>
>>>>
>>>> --
>>>> View this message in context:
>>>> http://www.nabble.com/Broker-not-releasing-memory-tp20662078p20808585.html
>>>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>>>
>>>
>>>
>>>
>>
>> --
>> View this message in context:
>> http://www.nabble.com/Broker-not-releasing-memory-tp20662078p20828589.html
>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>
>
>
>
--
View this message in context: http://www.nabble.com/Broker-not-releasing-memory-tp20662078p20890719.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Broker not releasing memory
Posted by Rick Blair <ri...@boeing.com>.
Hi Will,
We use an embedded broker Here is the url.
tcp://{0}:{1}?jms.useAsyncSend=false&jms.optimizeAcknowledge=false&jms.prefe
tchPolicy.all=1000";
The problem started when optimizeAcknowledge=true was used. Also I cannt
get old messages in the topics to timeout. Some do, most don¹t.
Hope this helps.
Rick
--
Tìoraidh!
Rick Blair
Associate Technical Fellow
Boeing Phantom Works
Network Systems Technology
Information Management Program
M/S: 42-50
Voice: (206) 544-1610
> From: Will van der Leij <wi...@stonethree.com>
> Reply-To: <us...@activemq.apache.org>
> Date: Wed, 3 Dec 2008 23:32:31 -0800 (PST)
> To: <us...@activemq.apache.org>
> Subject: Re: Broker not releasing memory
>
>
> Hi Rick,
> As I understand optimizeAcknowledge is disabled by default. Unfortunately,
> setting it explicitely has not had an affect on our memory usage.
> Could you perhaps briefly describe the rest of your configuration? I'm
> presuming that it is a combination of options that would do the trick.
>
> Regards,
> Will
>
>
>
> Rick Blair wrote:
>>
>> Hi Kirby,
>>
>> We have seen exactly the same behaviour. As a work around we set
>> optimizeAcknowdlege to false. I did not run down the root cause, however.
>>
>> --
>> Cheers!
>>
>> Rick Blair
>>
>>
>>> From: "Frugia, Kirby A" <ki...@hp.com>
>>> Reply-To: <us...@activemq.apache.org>
>>> Date: Wed, 3 Dec 2008 15:43:30 +0000
>>> To: "users@activemq.apache.org" <us...@activemq.apache.org>
>>> Conversation: Broker not releasing memory
>>> Subject: RE: Broker not releasing memory
>>>
>>> This is fairly easy to reproduce if you create one app which sends
>>> messages
>>> quickly and another which consumes the messages and sleeps before
>>> ack'ing.
>>> Once you see broker memory usage going up (through activemq-admin), kill
>>> the
>>> consumer application. Then run activemq-admin again some time later and
>>> you'll see the same memorypercentusage.
>>>
>>> Or, let the consumer run the broker out of memory and you'll see the
>>> producer
>>> get blocked. Kill both apps and you'll never see memory go back down
>>> again
>>> and you'll never be able to send messages again.
>>>
>>> Thanks,
>>> Kirby
>>>
>>> -----Original Message-----
>>> From: Will van der Leij [mailto:will@stonethree.com]
>>> Sent: Wednesday, December 03, 2008 12:17 AM
>>> To: users@activemq.apache.org
>>> Subject: RE: Broker not releasing memory
>>>
>>>
>>> Well, I'm reticent to suggest something going on under the hood without
>>> taking a good look at it, however, if there is something wrong in the
>>> broker
>>> then I presume it has to do with a "sweeper"-like thread that
>>> periodically
>>> cleans up queues and topics that are left pending and consumerless
>>> (similar
>>> to how sweeper threads would clean up expired messages etc.
>>>
>>> Then again, we are the only two that I know of describing this behaviour.
>>> This suggests that either there is something bizarrely odd about our
>>> configurations or we are, indeed, correct :)
>>>
>>>
>>>
>>> Frugia, Kirby A wrote:
>>>>
>>>> Yes, this is exactly what we are seeing. Any ideas?
>>>>
>>>> -----Original Message-----
>>>> From: Will van der Leij [mailto:will@stonethree.com]
>>>> Sent: Thursday, November 27, 2008 11:42 PM
>>>> To: users@activemq.apache.org
>>>> Subject: Re: Broker not releasing memory
>>>>
>>>>
>>>> We see similar behaviour in a slightly different setup so I'd like to
>>>> elaborate a bit in the hope of finding out if this is either intended
>>>> behaviour or a memory leak.
>>>>
>>>> Essentially, with a single or multiple producers and a single or
>>>> multiple
>>>> consumers on a single topic:
>>>> - We send messages faster than the consumer can read asynchronously
>>>> - When the send queue is larger than the consumer's prefetch buffer
>>>> then
>>>> we correctly see messages filling up the broker memory
>>>> (memoryPercentUSage in JMX)
>>>> - Killing the consumer (cleanly through a close or not-cleanly with a
>>>> terminate signal)
>>>> results in the memoryPercentUsage not being released
>>>> - even though there are no more consumers on the Topic
>>>> - this memory is never released until the broker is restarted
>>>>
>>>> I've tried it with persistence on and off, all the retroactive recovery
>>>> options on and off, flow control on and off, caching on and off,
>>>> asynchronous dispatching on and off, noLocal flag on and off etc.
>>>>
>>>> This is a very typical case of a consumer dying with inflight messages
>>>> on
>>>> the broker. Is the behaviour of the messages not being cleaned up or
>>>> recovered intended (i.e. they are there for a reason) or is there
>>>> soething
>>>> wrnog with the setup.
>>>>
>>>> Many thanks
>>>> Will van der Leij
>>>>
>>>>
>>>>
>>>> Frugia, Kirby A wrote:
>>>>>
>>>>> Hi All,
>>>>>
>>>>> Sorry I dual-posted in the dev list. I think I sent to the wrong
>>>>> one...
>>>>>
>>>>> We are seeing an issue with our broker not releasing memory on topics.
>>>>>
>>>>> Setup:
>>>>> * Active MQ 5.1 (out of the box)
>>>>> * Persistent messages sent by publishers
>>>>> * Non-durable topics
>>>>>
>>>>> We were seeing an issue in production with slow consumers causing the
>>>>> broker to run out of memory.
>>>>>
>>>>> We wrote a test app, which has one publisher and one subscriber to the
>>>>> same topic. The publisher sends messages on this topic frequently.
>>>>> The
>>>>> subscriber, upon receiving a message, goes to sleep for 10 minutes.
>>>>>
>>>>> Very quickly, the broker runs out of memory. When this happens, our
>>>>> publishers can no longer send messages; the send is blocked. This is
>>>>> expected. However, if we kill our application (which cleanly
>>>>> disconnects
>>>>> from the broker), the broker's memory usage (MemoryPercentUsage) does
>>>>> not
>>>>> go back down. Also, any new apps that startup will have their
>>>>> publishers
>>>>> blocked.
>>>>>
>>>>> Shouldn't the broker release the memory associated with these messages?
>>>>>
>>>>> Thanks,
>>>>> Kirby
>>>>>
>>>>>
>>>>>
>>>>
>>>> --
>>>> View this message in context:
>>>> http://www.nabble.com/Broker-not-releasing-memory-tp20662078p20730116.html
>>>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>>>
>>>>
>>>>
>>>
>>> --
>>> View this message in context:
>>> http://www.nabble.com/Broker-not-releasing-memory-tp20662078p20808585.html
>>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>>
>>
>>
>>
>
> --
> View this message in context:
> http://www.nabble.com/Broker-not-releasing-memory-tp20662078p20828589.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Re: Broker not releasing memory
Posted by Will van der Leij <wi...@stonethree.com>.
Hi Rick,
As I understand optimizeAcknowledge is disabled by default. Unfortunately,
setting it explicitely has not had an affect on our memory usage.
Could you perhaps briefly describe the rest of your configuration? I'm
presuming that it is a combination of options that would do the trick.
Regards,
Will
Rick Blair wrote:
>
> Hi Kirby,
>
> We have seen exactly the same behaviour. As a work around we set
> optimizeAcknowdlege to false. I did not run down the root cause, however.
>
> --
> Cheers!
>
> Rick Blair
>
>
>> From: "Frugia, Kirby A" <ki...@hp.com>
>> Reply-To: <us...@activemq.apache.org>
>> Date: Wed, 3 Dec 2008 15:43:30 +0000
>> To: "users@activemq.apache.org" <us...@activemq.apache.org>
>> Conversation: Broker not releasing memory
>> Subject: RE: Broker not releasing memory
>>
>> This is fairly easy to reproduce if you create one app which sends
>> messages
>> quickly and another which consumes the messages and sleeps before
>> ack'ing.
>> Once you see broker memory usage going up (through activemq-admin), kill
>> the
>> consumer application. Then run activemq-admin again some time later and
>> you'll see the same memorypercentusage.
>>
>> Or, let the consumer run the broker out of memory and you'll see the
>> producer
>> get blocked. Kill both apps and you'll never see memory go back down
>> again
>> and you'll never be able to send messages again.
>>
>> Thanks,
>> Kirby
>>
>> -----Original Message-----
>> From: Will van der Leij [mailto:will@stonethree.com]
>> Sent: Wednesday, December 03, 2008 12:17 AM
>> To: users@activemq.apache.org
>> Subject: RE: Broker not releasing memory
>>
>>
>> Well, I'm reticent to suggest something going on under the hood without
>> taking a good look at it, however, if there is something wrong in the
>> broker
>> then I presume it has to do with a "sweeper"-like thread that
>> periodically
>> cleans up queues and topics that are left pending and consumerless
>> (similar
>> to how sweeper threads would clean up expired messages etc.
>>
>> Then again, we are the only two that I know of describing this behaviour.
>> This suggests that either there is something bizarrely odd about our
>> configurations or we are, indeed, correct :)
>>
>>
>>
>> Frugia, Kirby A wrote:
>>>
>>> Yes, this is exactly what we are seeing. Any ideas?
>>>
>>> -----Original Message-----
>>> From: Will van der Leij [mailto:will@stonethree.com]
>>> Sent: Thursday, November 27, 2008 11:42 PM
>>> To: users@activemq.apache.org
>>> Subject: Re: Broker not releasing memory
>>>
>>>
>>> We see similar behaviour in a slightly different setup so I'd like to
>>> elaborate a bit in the hope of finding out if this is either intended
>>> behaviour or a memory leak.
>>>
>>> Essentially, with a single or multiple producers and a single or
>>> multiple
>>> consumers on a single topic:
>>> - We send messages faster than the consumer can read asynchronously
>>> - When the send queue is larger than the consumer's prefetch buffer
>>> then
>>> we correctly see messages filling up the broker memory
>>> (memoryPercentUSage in JMX)
>>> - Killing the consumer (cleanly through a close or not-cleanly with a
>>> terminate signal)
>>> results in the memoryPercentUsage not being released
>>> - even though there are no more consumers on the Topic
>>> - this memory is never released until the broker is restarted
>>>
>>> I've tried it with persistence on and off, all the retroactive recovery
>>> options on and off, flow control on and off, caching on and off,
>>> asynchronous dispatching on and off, noLocal flag on and off etc.
>>>
>>> This is a very typical case of a consumer dying with inflight messages
>>> on
>>> the broker. Is the behaviour of the messages not being cleaned up or
>>> recovered intended (i.e. they are there for a reason) or is there
>>> soething
>>> wrnog with the setup.
>>>
>>> Many thanks
>>> Will van der Leij
>>>
>>>
>>>
>>> Frugia, Kirby A wrote:
>>>>
>>>> Hi All,
>>>>
>>>> Sorry I dual-posted in the dev list. I think I sent to the wrong
>>>> one...
>>>>
>>>> We are seeing an issue with our broker not releasing memory on topics.
>>>>
>>>> Setup:
>>>> * Active MQ 5.1 (out of the box)
>>>> * Persistent messages sent by publishers
>>>> * Non-durable topics
>>>>
>>>> We were seeing an issue in production with slow consumers causing the
>>>> broker to run out of memory.
>>>>
>>>> We wrote a test app, which has one publisher and one subscriber to the
>>>> same topic. The publisher sends messages on this topic frequently.
>>>> The
>>>> subscriber, upon receiving a message, goes to sleep for 10 minutes.
>>>>
>>>> Very quickly, the broker runs out of memory. When this happens, our
>>>> publishers can no longer send messages; the send is blocked. This is
>>>> expected. However, if we kill our application (which cleanly
>>>> disconnects
>>>> from the broker), the broker's memory usage (MemoryPercentUsage) does
>>>> not
>>>> go back down. Also, any new apps that startup will have their
>>>> publishers
>>>> blocked.
>>>>
>>>> Shouldn't the broker release the memory associated with these messages?
>>>>
>>>> Thanks,
>>>> Kirby
>>>>
>>>>
>>>>
>>>
>>> --
>>> View this message in context:
>>> http://www.nabble.com/Broker-not-releasing-memory-tp20662078p20730116.html
>>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>>
>>>
>>>
>>
>> --
>> View this message in context:
>> http://www.nabble.com/Broker-not-releasing-memory-tp20662078p20808585.html
>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>
>
>
>
--
View this message in context: http://www.nabble.com/Broker-not-releasing-memory-tp20662078p20828589.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Broker not releasing memory
Posted by Rick Blair <ri...@boeing.com>.
Hi Kirby,
We have seen exactly the same behaviour. As a work around we set
optimizeAcknowdlege to false. I did not run down the root cause, however.
--
Cheers!
Rick Blair
> From: "Frugia, Kirby A" <ki...@hp.com>
> Reply-To: <us...@activemq.apache.org>
> Date: Wed, 3 Dec 2008 15:43:30 +0000
> To: "users@activemq.apache.org" <us...@activemq.apache.org>
> Conversation: Broker not releasing memory
> Subject: RE: Broker not releasing memory
>
> This is fairly easy to reproduce if you create one app which sends messages
> quickly and another which consumes the messages and sleeps before ack'ing.
> Once you see broker memory usage going up (through activemq-admin), kill the
> consumer application. Then run activemq-admin again some time later and
> you'll see the same memorypercentusage.
>
> Or, let the consumer run the broker out of memory and you'll see the producer
> get blocked. Kill both apps and you'll never see memory go back down again
> and you'll never be able to send messages again.
>
> Thanks,
> Kirby
>
> -----Original Message-----
> From: Will van der Leij [mailto:will@stonethree.com]
> Sent: Wednesday, December 03, 2008 12:17 AM
> To: users@activemq.apache.org
> Subject: RE: Broker not releasing memory
>
>
> Well, I'm reticent to suggest something going on under the hood without
> taking a good look at it, however, if there is something wrong in the broker
> then I presume it has to do with a "sweeper"-like thread that periodically
> cleans up queues and topics that are left pending and consumerless (similar
> to how sweeper threads would clean up expired messages etc.
>
> Then again, we are the only two that I know of describing this behaviour.
> This suggests that either there is something bizarrely odd about our
> configurations or we are, indeed, correct :)
>
>
>
> Frugia, Kirby A wrote:
>>
>> Yes, this is exactly what we are seeing. Any ideas?
>>
>> -----Original Message-----
>> From: Will van der Leij [mailto:will@stonethree.com]
>> Sent: Thursday, November 27, 2008 11:42 PM
>> To: users@activemq.apache.org
>> Subject: Re: Broker not releasing memory
>>
>>
>> We see similar behaviour in a slightly different setup so I'd like to
>> elaborate a bit in the hope of finding out if this is either intended
>> behaviour or a memory leak.
>>
>> Essentially, with a single or multiple producers and a single or multiple
>> consumers on a single topic:
>> - We send messages faster than the consumer can read asynchronously
>> - When the send queue is larger than the consumer's prefetch buffer then
>> we correctly see messages filling up the broker memory
>> (memoryPercentUSage in JMX)
>> - Killing the consumer (cleanly through a close or not-cleanly with a
>> terminate signal)
>> results in the memoryPercentUsage not being released
>> - even though there are no more consumers on the Topic
>> - this memory is never released until the broker is restarted
>>
>> I've tried it with persistence on and off, all the retroactive recovery
>> options on and off, flow control on and off, caching on and off,
>> asynchronous dispatching on and off, noLocal flag on and off etc.
>>
>> This is a very typical case of a consumer dying with inflight messages on
>> the broker. Is the behaviour of the messages not being cleaned up or
>> recovered intended (i.e. they are there for a reason) or is there soething
>> wrnog with the setup.
>>
>> Many thanks
>> Will van der Leij
>>
>>
>>
>> Frugia, Kirby A wrote:
>>>
>>> Hi All,
>>>
>>> Sorry I dual-posted in the dev list. I think I sent to the wrong one...
>>>
>>> We are seeing an issue with our broker not releasing memory on topics.
>>>
>>> Setup:
>>> * Active MQ 5.1 (out of the box)
>>> * Persistent messages sent by publishers
>>> * Non-durable topics
>>>
>>> We were seeing an issue in production with slow consumers causing the
>>> broker to run out of memory.
>>>
>>> We wrote a test app, which has one publisher and one subscriber to the
>>> same topic. The publisher sends messages on this topic frequently. The
>>> subscriber, upon receiving a message, goes to sleep for 10 minutes.
>>>
>>> Very quickly, the broker runs out of memory. When this happens, our
>>> publishers can no longer send messages; the send is blocked. This is
>>> expected. However, if we kill our application (which cleanly disconnects
>>> from the broker), the broker's memory usage (MemoryPercentUsage) does not
>>> go back down. Also, any new apps that startup will have their publishers
>>> blocked.
>>>
>>> Shouldn't the broker release the memory associated with these messages?
>>>
>>> Thanks,
>>> Kirby
>>>
>>>
>>>
>>
>> --
>> View this message in context:
>> http://www.nabble.com/Broker-not-releasing-memory-tp20662078p20730116.html
>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>
>>
>>
>
> --
> View this message in context:
> http://www.nabble.com/Broker-not-releasing-memory-tp20662078p20808585.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
RE: Broker not releasing memory
Posted by "Frugia, Kirby A" <ki...@hp.com>.
This is fairly easy to reproduce if you create one app which sends messages quickly and another which consumes the messages and sleeps before ack'ing. Once you see broker memory usage going up (through activemq-admin), kill the consumer application. Then run activemq-admin again some time later and you'll see the same memorypercentusage.
Or, let the consumer run the broker out of memory and you'll see the producer get blocked. Kill both apps and you'll never see memory go back down again and you'll never be able to send messages again.
Thanks,
Kirby
-----Original Message-----
From: Will van der Leij [mailto:will@stonethree.com]
Sent: Wednesday, December 03, 2008 12:17 AM
To: users@activemq.apache.org
Subject: RE: Broker not releasing memory
Well, I'm reticent to suggest something going on under the hood without
taking a good look at it, however, if there is something wrong in the broker
then I presume it has to do with a "sweeper"-like thread that periodically
cleans up queues and topics that are left pending and consumerless (similar
to how sweeper threads would clean up expired messages etc.
Then again, we are the only two that I know of describing this behaviour.
This suggests that either there is something bizarrely odd about our
configurations or we are, indeed, correct :)
Frugia, Kirby A wrote:
>
> Yes, this is exactly what we are seeing. Any ideas?
>
> -----Original Message-----
> From: Will van der Leij [mailto:will@stonethree.com]
> Sent: Thursday, November 27, 2008 11:42 PM
> To: users@activemq.apache.org
> Subject: Re: Broker not releasing memory
>
>
> We see similar behaviour in a slightly different setup so I'd like to
> elaborate a bit in the hope of finding out if this is either intended
> behaviour or a memory leak.
>
> Essentially, with a single or multiple producers and a single or multiple
> consumers on a single topic:
> - We send messages faster than the consumer can read asynchronously
> - When the send queue is larger than the consumer's prefetch buffer then
> we correctly see messages filling up the broker memory
> (memoryPercentUSage in JMX)
> - Killing the consumer (cleanly through a close or not-cleanly with a
> terminate signal)
> results in the memoryPercentUsage not being released
> - even though there are no more consumers on the Topic
> - this memory is never released until the broker is restarted
>
> I've tried it with persistence on and off, all the retroactive recovery
> options on and off, flow control on and off, caching on and off,
> asynchronous dispatching on and off, noLocal flag on and off etc.
>
> This is a very typical case of a consumer dying with inflight messages on
> the broker. Is the behaviour of the messages not being cleaned up or
> recovered intended (i.e. they are there for a reason) or is there soething
> wrnog with the setup.
>
> Many thanks
> Will van der Leij
>
>
>
> Frugia, Kirby A wrote:
>>
>> Hi All,
>>
>> Sorry I dual-posted in the dev list. I think I sent to the wrong one...
>>
>> We are seeing an issue with our broker not releasing memory on topics.
>>
>> Setup:
>> * Active MQ 5.1 (out of the box)
>> * Persistent messages sent by publishers
>> * Non-durable topics
>>
>> We were seeing an issue in production with slow consumers causing the
>> broker to run out of memory.
>>
>> We wrote a test app, which has one publisher and one subscriber to the
>> same topic. The publisher sends messages on this topic frequently. The
>> subscriber, upon receiving a message, goes to sleep for 10 minutes.
>>
>> Very quickly, the broker runs out of memory. When this happens, our
>> publishers can no longer send messages; the send is blocked. This is
>> expected. However, if we kill our application (which cleanly disconnects
>> from the broker), the broker's memory usage (MemoryPercentUsage) does not
>> go back down. Also, any new apps that startup will have their publishers
>> blocked.
>>
>> Shouldn't the broker release the memory associated with these messages?
>>
>> Thanks,
>> Kirby
>>
>>
>>
>
> --
> View this message in context:
> http://www.nabble.com/Broker-not-releasing-memory-tp20662078p20730116.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
>
>
--
View this message in context: http://www.nabble.com/Broker-not-releasing-memory-tp20662078p20808585.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
RE: Broker not releasing memory
Posted by Will van der Leij <wi...@stonethree.com>.
Well, I'm reticent to suggest something going on under the hood without
taking a good look at it, however, if there is something wrong in the broker
then I presume it has to do with a "sweeper"-like thread that periodically
cleans up queues and topics that are left pending and consumerless (similar
to how sweeper threads would clean up expired messages etc.
Then again, we are the only two that I know of describing this behaviour.
This suggests that either there is something bizarrely odd about our
configurations or we are, indeed, correct :)
Frugia, Kirby A wrote:
>
> Yes, this is exactly what we are seeing. Any ideas?
>
> -----Original Message-----
> From: Will van der Leij [mailto:will@stonethree.com]
> Sent: Thursday, November 27, 2008 11:42 PM
> To: users@activemq.apache.org
> Subject: Re: Broker not releasing memory
>
>
> We see similar behaviour in a slightly different setup so I'd like to
> elaborate a bit in the hope of finding out if this is either intended
> behaviour or a memory leak.
>
> Essentially, with a single or multiple producers and a single or multiple
> consumers on a single topic:
> - We send messages faster than the consumer can read asynchronously
> - When the send queue is larger than the consumer's prefetch buffer then
> we correctly see messages filling up the broker memory
> (memoryPercentUSage in JMX)
> - Killing the consumer (cleanly through a close or not-cleanly with a
> terminate signal)
> results in the memoryPercentUsage not being released
> - even though there are no more consumers on the Topic
> - this memory is never released until the broker is restarted
>
> I've tried it with persistence on and off, all the retroactive recovery
> options on and off, flow control on and off, caching on and off,
> asynchronous dispatching on and off, noLocal flag on and off etc.
>
> This is a very typical case of a consumer dying with inflight messages on
> the broker. Is the behaviour of the messages not being cleaned up or
> recovered intended (i.e. they are there for a reason) or is there soething
> wrnog with the setup.
>
> Many thanks
> Will van der Leij
>
>
>
> Frugia, Kirby A wrote:
>>
>> Hi All,
>>
>> Sorry I dual-posted in the dev list. I think I sent to the wrong one...
>>
>> We are seeing an issue with our broker not releasing memory on topics.
>>
>> Setup:
>> * Active MQ 5.1 (out of the box)
>> * Persistent messages sent by publishers
>> * Non-durable topics
>>
>> We were seeing an issue in production with slow consumers causing the
>> broker to run out of memory.
>>
>> We wrote a test app, which has one publisher and one subscriber to the
>> same topic. The publisher sends messages on this topic frequently. The
>> subscriber, upon receiving a message, goes to sleep for 10 minutes.
>>
>> Very quickly, the broker runs out of memory. When this happens, our
>> publishers can no longer send messages; the send is blocked. This is
>> expected. However, if we kill our application (which cleanly disconnects
>> from the broker), the broker's memory usage (MemoryPercentUsage) does not
>> go back down. Also, any new apps that startup will have their publishers
>> blocked.
>>
>> Shouldn't the broker release the memory associated with these messages?
>>
>> Thanks,
>> Kirby
>>
>>
>>
>
> --
> View this message in context:
> http://www.nabble.com/Broker-not-releasing-memory-tp20662078p20730116.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
>
>
--
View this message in context: http://www.nabble.com/Broker-not-releasing-memory-tp20662078p20808585.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.