You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@qpid.apache.org by "xiaodan.wang" <xi...@salesforce.com> on 2014/10/17 01:59:53 UTC
Possible for unconsumed messages to prevent GC from reclaiming
memory held by prior messages?
We encountered an interesting behavior on the Qpid Java broker with respect
to memory consumption. Specifically, we reserve certain queues for
"suspending" messages. Messages on these suspend queues remain unconsumed
and no consumers are setup to listen on them. For the incident in question,
we enqueued a handful of messages to the suspend queue. Then, over the
course of 2 days, broker heap gradually trended up to 80%. At this point our
flow control mechanism kicked in and we stopped enqueuing to the broker.
When we inspected the broker, we found a couple dozen messages on the
suspend queue and not much else on the other queues. We then manually
triggered GC on the broker but heap utilization remained consistent at 80%.
Next, we manually deleted each message from the suspend queue and forced
another GC. At this point, heap utilization dropped back to normal levels.
This led us to believe that if an unconsumed message M1 is enqueued on queue
A, and a message M2 is subsequently enqueued on queue B, then even if M2 is
consumed, the memory referenced by M2 will not be reclaimed until M1 is
consumed. Unfortunately, we tried to reproduce such a scenario at scale and
was unsuccessful. Specifically, GC appears to reclaim memory in the presence
of unconsumed messages.
We are scratching our heads at the moment and looking for suggestions from
Qpid gurus. Is there any condition in which an unconsumed message can
prevent GC from reclaiming memory?
Our setup is as follows: non-persistent messages, asynchronous onMessage
delivery, transaction sessions.
Thanks in advance! Dan
--
View this message in context: http://qpid.2158936.n2.nabble.com/Possible-for-unconsumed-messages-to-prevent-GC-from-reclaiming-memory-held-by-prior-messages-tp7615368.html
Sent from the Apache Qpid users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: Possible for unconsumed messages to prevent GC from reclaiming
memory held by prior messages?
Posted by "xiaodan.wang" <xi...@salesforce.com>.
I think the companion code from v0.16 broker is sitting under the send method
in
java/broker/src/main/java/org/apache/qpid/server/subscription/Subscription_0_10.java
_session.sendMessage(xfr, _postIdSettingAction);
entry.incrementDeliveryCount();
_deliveredCount.incrementAndGet();
if(_acceptMode == MessageAcceptMode.NONE && _acquireMode ==
MessageAcquireMode.PRE_ACQUIRED)
{
Hopefully the same fix can be applied here.
Thanks again!
--
View this message in context: http://qpid.2158936.n2.nabble.com/Possible-for-unconsumed-messages-to-prevent-GC-from-reclaiming-memory-held-by-prior-messages-tp7615368p7615941.html
Sent from the Apache Qpid users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: Possible for unconsumed messages to prevent GC from reclaiming
memory held by prior messages?
Posted by Helen Kwong <he...@gmail.com>.
Hi Rob, just got your fix in trunk and ran our test again, and confirmed
that the bug is fixed. Thank you again for your help and responding so
quickly!
On Thu, Oct 30, 2014 at 5:03 AM, Rob Godfrey <ro...@gmail.com>
wrote:
> Thank you for those kind words.
>
> It's always gratifying to be able know that our collective efforts in Qpid
> are being used as widely as possible. The users of Qpid form a vital part
> of our extended community.
>
> I would be remiss, however, if I didn't mention that the efforts of both
> Dan and Helen in creating a clear reproducible test cases, and providing
> detailed analysis of the symptoms (both in this case and in previous
> investigations) makes the job of diagnosing and solving issues a *much*
> easier task. I'd certainly like to thank them for all their work which has
> helped make Qpid a better experience for everyone.
>
> Finally we'd always like to hear the opinion of our users on features and
> changes they'd like to see in Qpid in future releases... so all feedback
> and suggestions you can give would be most welcome.
>
> Thanks again,
> Rob
>
> On 30 October 2014 12:51, John Buisson <jb...@salesforce.com> wrote:
>
> > Wanted to give a huge thank you to the community! We use QPID outside of
> > what I'd consider the "normal" use cases and we've had nothing but
> amazing
> > responses and help once we can reproduce problems we have. Community
> > involvement was one of the many reasons we went with QPID and it has
> been a
> > great experience! Bugs happen, and the exteremely fast response time on
> an
> > open-source project is *much* appreciated!
> >
> > On Wed, Oct 29, 2014 at 8:05 PM, Rob Godfrey <ro...@gmail.com>
> > wrote:
> >
> > > Well... we'll see if there is anything else going on, but that's
> > definitely
> > > a bug which would cause a memory leak.
> > >
> > > I've checked in a change onto trunk for that issue (for 0.16 the code
> > will
> > > be in a slightly different place ... I'll dig that out tmr (actually
> > later
> > > today for me, 1am here now) after I've had some sleep :-).
> > >
> > > Apologies that you've run into this bug,
> > >
> > > -- Rob
> > >
> > > On 30 October 2014 01:01, xiaodan.wang <xi...@salesforce.com>
> > > wrote:
> > >
> > > > Well I'm speechless, literally an hour after our post, you provide
> the
> > > fix
> > > > :)
> > > > Thank you so much! We will test this out with the fix.
> > > >
> > > > Cheers! Dan
> > > >
> > > >
> > > >
> > > > --
> > > > View this message in context:
> > > >
> > >
> >
> http://qpid.2158936.n2.nabble.com/Possible-for-unconsumed-messages-to-prevent-GC-from-reclaiming-memory-held-by-prior-messages-tp7615368p7615937.html
> > > > Sent from the Apache Qpid users mailing list archive at Nabble.com.
> > > >
> > > > ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> > > > For additional commands, e-mail: users-help@qpid.apache.org
> > > >
> > > >
> > >
> >
>
Re: Possible for unconsumed messages to prevent GC from reclaiming
memory held by prior messages?
Posted by Rob Godfrey <ro...@gmail.com>.
Thank you for those kind words.
It's always gratifying to be able know that our collective efforts in Qpid
are being used as widely as possible. The users of Qpid form a vital part
of our extended community.
I would be remiss, however, if I didn't mention that the efforts of both
Dan and Helen in creating a clear reproducible test cases, and providing
detailed analysis of the symptoms (both in this case and in previous
investigations) makes the job of diagnosing and solving issues a *much*
easier task. I'd certainly like to thank them for all their work which has
helped make Qpid a better experience for everyone.
Finally we'd always like to hear the opinion of our users on features and
changes they'd like to see in Qpid in future releases... so all feedback
and suggestions you can give would be most welcome.
Thanks again,
Rob
On 30 October 2014 12:51, John Buisson <jb...@salesforce.com> wrote:
> Wanted to give a huge thank you to the community! We use QPID outside of
> what I'd consider the "normal" use cases and we've had nothing but amazing
> responses and help once we can reproduce problems we have. Community
> involvement was one of the many reasons we went with QPID and it has been a
> great experience! Bugs happen, and the exteremely fast response time on an
> open-source project is *much* appreciated!
>
> On Wed, Oct 29, 2014 at 8:05 PM, Rob Godfrey <ro...@gmail.com>
> wrote:
>
> > Well... we'll see if there is anything else going on, but that's
> definitely
> > a bug which would cause a memory leak.
> >
> > I've checked in a change onto trunk for that issue (for 0.16 the code
> will
> > be in a slightly different place ... I'll dig that out tmr (actually
> later
> > today for me, 1am here now) after I've had some sleep :-).
> >
> > Apologies that you've run into this bug,
> >
> > -- Rob
> >
> > On 30 October 2014 01:01, xiaodan.wang <xi...@salesforce.com>
> > wrote:
> >
> > > Well I'm speechless, literally an hour after our post, you provide the
> > fix
> > > :)
> > > Thank you so much! We will test this out with the fix.
> > >
> > > Cheers! Dan
> > >
> > >
> > >
> > > --
> > > View this message in context:
> > >
> >
> http://qpid.2158936.n2.nabble.com/Possible-for-unconsumed-messages-to-prevent-GC-from-reclaiming-memory-held-by-prior-messages-tp7615368p7615937.html
> > > Sent from the Apache Qpid users mailing list archive at Nabble.com.
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> > > For additional commands, e-mail: users-help@qpid.apache.org
> > >
> > >
> >
>
Re: Possible for unconsumed messages to prevent GC from reclaiming
memory held by prior messages?
Posted by John Buisson <jb...@salesforce.com>.
Wanted to give a huge thank you to the community! We use QPID outside of
what I'd consider the "normal" use cases and we've had nothing but amazing
responses and help once we can reproduce problems we have. Community
involvement was one of the many reasons we went with QPID and it has been a
great experience! Bugs happen, and the exteremely fast response time on an
open-source project is *much* appreciated!
On Wed, Oct 29, 2014 at 8:05 PM, Rob Godfrey <ro...@gmail.com>
wrote:
> Well... we'll see if there is anything else going on, but that's definitely
> a bug which would cause a memory leak.
>
> I've checked in a change onto trunk for that issue (for 0.16 the code will
> be in a slightly different place ... I'll dig that out tmr (actually later
> today for me, 1am here now) after I've had some sleep :-).
>
> Apologies that you've run into this bug,
>
> -- Rob
>
> On 30 October 2014 01:01, xiaodan.wang <xi...@salesforce.com>
> wrote:
>
> > Well I'm speechless, literally an hour after our post, you provide the
> fix
> > :)
> > Thank you so much! We will test this out with the fix.
> >
> > Cheers! Dan
> >
> >
> >
> > --
> > View this message in context:
> >
> http://qpid.2158936.n2.nabble.com/Possible-for-unconsumed-messages-to-prevent-GC-from-reclaiming-memory-held-by-prior-messages-tp7615368p7615937.html
> > Sent from the Apache Qpid users mailing list archive at Nabble.com.
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> > For additional commands, e-mail: users-help@qpid.apache.org
> >
> >
>
Re: Possible for unconsumed messages to prevent GC from reclaiming
memory held by prior messages?
Posted by Rob Godfrey <ro...@gmail.com>.
Well... we'll see if there is anything else going on, but that's definitely
a bug which would cause a memory leak.
I've checked in a change onto trunk for that issue (for 0.16 the code will
be in a slightly different place ... I'll dig that out tmr (actually later
today for me, 1am here now) after I've had some sleep :-).
Apologies that you've run into this bug,
-- Rob
On 30 October 2014 01:01, xiaodan.wang <xi...@salesforce.com> wrote:
> Well I'm speechless, literally an hour after our post, you provide the fix
> :)
> Thank you so much! We will test this out with the fix.
>
> Cheers! Dan
>
>
>
> --
> View this message in context:
> http://qpid.2158936.n2.nabble.com/Possible-for-unconsumed-messages-to-prevent-GC-from-reclaiming-memory-held-by-prior-messages-tp7615368p7615937.html
> Sent from the Apache Qpid users mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>
>
Re: Possible for unconsumed messages to prevent GC from reclaiming
memory held by prior messages?
Posted by "xiaodan.wang" <xi...@salesforce.com>.
Well I'm speechless, literally an hour after our post, you provide the fix :)
Thank you so much! We will test this out with the fix.
Cheers! Dan
--
View this message in context: http://qpid.2158936.n2.nabble.com/Possible-for-unconsumed-messages-to-prevent-GC-from-reclaiming-memory-held-by-prior-messages-tp7615368p7615937.html
Sent from the Apache Qpid users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: Possible for unconsumed messages to prevent GC from reclaiming
memory held by prior messages?
Posted by Rob Godfrey <ro...@gmail.com>.
Ah - well, at least that is easy to fix... I think on trunk the following
change should fix it:
Index:
broker-plugins/amqp-0-10-protocol/src/main/java/org/apache/qpid/server/protocol/v0_10/ConsumerTarget_0_10.java
===================================================================
---
broker-plugins/amqp-0-10-protocol/src/main/java/org/apache/qpid/server/protocol/v0_10/ConsumerTarget_0_10.java
(revision 1634888)
+++
broker-plugins/amqp-0-10-protocol/src/main/java/org/apache/qpid/server/protocol/v0_10/ConsumerTarget_0_10.java
(working copy)
@@ -329,6 +329,10 @@
_session.sendMessage(xfr, _postIdSettingAction);
+
+ _postIdSettingAction.setAction(null);
+ _postIdSettingAction.setXfr(null);
+
entry.incrementDeliveryCount();
if(_acceptMode == MessageAcceptMode.NONE && _acquireMode ==
MessageAcquireMode.PRE_ACQUIRED)
{
The issue is the per session postIdSettingAction retaining a reference to
the last message it sent... There's no need for it to retain that
reference, so it can be cleared immediately after sending.
I'll raise a JIRA and fix that on trunk right now.
-- Rob
On 30 October 2014 00:42, xiaodan.wang <xi...@salesforce.com> wrote:
> Attaching the screenshot that Helen referenced:
>
> qpid_broker_heap_queueB_queueAEntries
> <
> http://qpid.2158936.n2.nabble.com/file/n7615934/qpid_broker_heap_queueB_queueAEntries.png
> >
>
> qpid_broker_heap_queueB_referenceToQueueAConsumer
> <
> http://qpid.2158936.n2.nabble.com/file/n7615934/qpid_broker_heap_queueB_referenceToQueueAConsumer.png
> >
>
>
>
> --
> View this message in context:
> http://qpid.2158936.n2.nabble.com/Possible-for-unconsumed-messages-to-prevent-GC-from-reclaiming-memory-held-by-prior-messages-tp7615368p7615934.html
> Sent from the Apache Qpid users mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>
>
Re: Possible for unconsumed messages to prevent GC from reclaiming
memory held by prior messages?
Posted by "xiaodan.wang" <xi...@salesforce.com>.
Attaching the screenshot that Helen referenced:
qpid_broker_heap_queueB_queueAEntries
<http://qpid.2158936.n2.nabble.com/file/n7615934/qpid_broker_heap_queueB_queueAEntries.png>
qpid_broker_heap_queueB_referenceToQueueAConsumer
<http://qpid.2158936.n2.nabble.com/file/n7615934/qpid_broker_heap_queueB_referenceToQueueAConsumer.png>
--
View this message in context: http://qpid.2158936.n2.nabble.com/Possible-for-unconsumed-messages-to-prevent-GC-from-reclaiming-memory-held-by-prior-messages-tp7615368p7615934.html
Sent from the Apache Qpid users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: Possible for unconsumed messages to prevent GC from reclaiming
memory held by prior messages?
Posted by Helen Kwong <he...@gmail.com>.
Hi Rob, I'll describe what I see from a heap dump during a run of the test
Dan described. I'm running the 0.32 broker and client, though I've also
looked at a heap dump with the 0.16 broker and client and have seen similar
patterns.
The "steady state" heap size of my broker is ~200MB. After running the
test, the heap goes up to ~1GB, even though there are no more messages on
queue A and I have manually triggered a GC. What I see is that the
PriorityQueueImpl that represents Queue B from the test has a retained size
that takes up most of the 1GB. It appears that from the steps 3(b)-(e) that
Dan described, objects representing the consumers of both Queue A and Queue
B are still being referenced by the entries of Queue B, and these consumer
objects of Queue A can reference old entries of the queue that have already
been drained in steps 3(f)-(h).
More details and an example of this: I attached 2 screenshots of from the
heap analysis. qpid_broker_heap_queueB_referenceToQueueAConsumer.png shows
the object reference tree of the PriorityQueueImpl representing Queue B. I
actually ran the test with Queue B having 20 messages rather than 1, and
the screenshot is showing the _stateChangeListeners list of the 4th queue
entry. We then see the list holding a QueueConsumerImpl for Queue B, which
eventually references its ServerSession, which references its 2 _consumers.
The 0 index element of this is for Queue A, and the 1 index element is
Queue B.
Then if you look at qpid_broker_heap_queueB_queueAEntries.png, you'll see
what's referenced by the Queue A consumer. Through _target ->
_postIdSettingAction -> _action_entry, we have a reference to a
PriorityQueueEntry. This further more starts a linked list of many more
entries on Queue A. The _state fields of these entries are all
MessageInstance$DeletedState, since we've already drained Queue A.
So it seems like in this way, we can be holding onto many queue entries
that have already been consumed from the queue.
On Wed, Oct 29, 2014 at 3:32 PM, xiaodan.wang <xi...@salesforce.com>
wrote:
> Hi Rob, apologies for taking so long to follow-up. We now have meaningful
> data (both a reproducible test and heap dump) to describe the issue. I will
> first describe the setup of the test we used to reproduce this issue and
> then Helen, on of my teammate, will describe the heap dump analysis.
>
> To recap, we observed that the Qpid (Java) broker heap gradually trended up
> in production when we had a handful of messages on a special "suspend"
> queue
> that were unconsumed. Even though we found no other messages when we
> inspected the broker, attempts to GC and reclaim broke memory failed. That
> is until we manually deleted the handful of suspended/unconsumed messages.
>
> I will now describe the test we used to reproduce this issue. We are using
> non-persistent messages, asynchronous onMessage delivery, and transacted
> sessions. We also ran the test using both the v0.16 Java broker and the
> v0.32/trunk Java broker and verified that the issue occurs on both.
>
> Experiment:
> 1) We setup two queues, A and B.
>
> 2) We first enqueue one message to queue B and do not consume the message
> (this message remains unconsumed for the entire duration of the test)
>
> 3) Next we setup a single test thread that runs iteratively and does:
> a) First enqueues 1000 messages to queue A and commits
> b) Next, checks out a NEW qpid session and creates a consumer for queue
> A. Invokes consumer.receive to fetch one message and then immediately rolls
> back the receive.
> c) Close the consumer for queue A
> d) Using the SAME qpid session from step b), create a consumer for queue
> B. Invoke consumer.receive and to fetch one message and then immediately
> rolls back the receive.
> e) Close the consumer for queue B and close the qpid session.
> f) Create a NEW qpid session, and create a consumer for queue A.
> g) Invoke consumer.receive repeatedly to drain all 1000 messages from
> queue A and then commit.
> h) Close the qpid session, at this point there are no messages on queue
> A
> and a single message on queue B.
>
> 4) Repeat step 3) many times
>
> 5) Before test is finished, validate that queue B has a single message and
> queue A is empty.
>
> At the conclusion of the test, we observed that Qpid broker heap is nearly
> exhausted and memory cannot be reclaimed via GC. Also, we can reproduce the
> same issue if steps 3b) - 3e) is replaced with session.createBrowser (queue
> browse/peeking) instead of consumer.receive on queues A and B.
>
> Helen is in the process of writing up the heap dump analysis and will post
> shortly.
> Thanks! Dan
>
>
>
>
> --
> View this message in context:
> http://qpid.2158936.n2.nabble.com/Possible-for-unconsumed-messages-to-prevent-GC-from-reclaiming-memory-held-by-prior-messages-tp7615368p7615932.html
> Sent from the Apache Qpid users mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>
>
Re: Possible for unconsumed messages to prevent GC from reclaiming
memory held by prior messages?
Posted by "xiaodan.wang" <xi...@salesforce.com>.
Hi Rob, apologies for taking so long to follow-up. We now have meaningful
data (both a reproducible test and heap dump) to describe the issue. I will
first describe the setup of the test we used to reproduce this issue and
then Helen, on of my teammate, will describe the heap dump analysis.
To recap, we observed that the Qpid (Java) broker heap gradually trended up
in production when we had a handful of messages on a special "suspend" queue
that were unconsumed. Even though we found no other messages when we
inspected the broker, attempts to GC and reclaim broke memory failed. That
is until we manually deleted the handful of suspended/unconsumed messages.
I will now describe the test we used to reproduce this issue. We are using
non-persistent messages, asynchronous onMessage delivery, and transacted
sessions. We also ran the test using both the v0.16 Java broker and the
v0.32/trunk Java broker and verified that the issue occurs on both.
Experiment:
1) We setup two queues, A and B.
2) We first enqueue one message to queue B and do not consume the message
(this message remains unconsumed for the entire duration of the test)
3) Next we setup a single test thread that runs iteratively and does:
a) First enqueues 1000 messages to queue A and commits
b) Next, checks out a NEW qpid session and creates a consumer for queue
A. Invokes consumer.receive to fetch one message and then immediately rolls
back the receive.
c) Close the consumer for queue A
d) Using the SAME qpid session from step b), create a consumer for queue
B. Invoke consumer.receive and to fetch one message and then immediately
rolls back the receive.
e) Close the consumer for queue B and close the qpid session.
f) Create a NEW qpid session, and create a consumer for queue A.
g) Invoke consumer.receive repeatedly to drain all 1000 messages from
queue A and then commit.
h) Close the qpid session, at this point there are no messages on queue A
and a single message on queue B.
4) Repeat step 3) many times
5) Before test is finished, validate that queue B has a single message and
queue A is empty.
At the conclusion of the test, we observed that Qpid broker heap is nearly
exhausted and memory cannot be reclaimed via GC. Also, we can reproduce the
same issue if steps 3b) - 3e) is replaced with session.createBrowser (queue
browse/peeking) instead of consumer.receive on queues A and B.
Helen is in the process of writing up the heap dump analysis and will post
shortly.
Thanks! Dan
--
View this message in context: http://qpid.2158936.n2.nabble.com/Possible-for-unconsumed-messages-to-prevent-GC-from-reclaiming-memory-held-by-prior-messages-tp7615368p7615932.html
Sent from the Apache Qpid users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: Possible for unconsumed messages to prevent GC from reclaiming
memory held by prior messages?
Posted by "xiaodan.wang" <xi...@salesforce.com>.
Thanks Rob, it is actually comforting to know that we likely ran into a rare
corner case or an unrelated issue and that our original assumption is
incorrect.
We will try to reproduce this and grab the heap dump. Will report back if
successful :)
Cheers! Dan
--
View this message in context: http://qpid.2158936.n2.nabble.com/Possible-for-unconsumed-messages-to-prevent-GC-from-reclaiming-memory-held-by-prior-messages-tp7615368p7615426.html
Sent from the Apache Qpid users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: Possible for unconsumed messages to prevent GC from reclaiming
memory held by prior messages?
Posted by Rob Godfrey <ro...@gmail.com>.
Hi Dan,
In general there's nothing I can think of with the current broker that I
know of that would explain what you saw, but do I remember correctly that
you guys still using 0.16... ? (Not that I recall of any defect in 0.16
that would cause this behaviour).
Without looking at a heap dump from the scenario you found, it's pretty
difficult to work out what the cause might be.
Certainly, regarding
>
> This led us to believe that if an unconsumed message M1 is enqueued on
> queue
> A, and a message M2 is subsequently enqueued on queue B, then even if M2 is
> consumed, the memory referenced by M2 will not be reclaimed until M1 is
> consumed. Unfortunately, we tried to reproduce such a scenario at scale and
> was unsuccessful. Specifically, GC appears to reclaim memory in the
> presence
> of unconsumed messages.
>
>
Yes - there's nothing in the broker that should link messages from
unrelated queues. So if there is any linkage in 0.16 it's not as simple as
that.
Unfortunately all I can suggest to you is that you try to reproduce the
issue, take a heap dump at that point and see what is taking up memory and
what is referencing those objects causing the issue.
Apologies I can't be of more help,
Rob
On 17 October 2014 01:59, xiaodan.wang <xi...@salesforce.com> wrote:
> We encountered an interesting behavior on the Qpid Java broker with respect
> to memory consumption. Specifically, we reserve certain queues for
> "suspending" messages. Messages on these suspend queues remain unconsumed
> and no consumers are setup to listen on them. For the incident in question,
> we enqueued a handful of messages to the suspend queue. Then, over the
> course of 2 days, broker heap gradually trended up to 80%. At this point
> our
> flow control mechanism kicked in and we stopped enqueuing to the broker.
>
> When we inspected the broker, we found a couple dozen messages on the
> suspend queue and not much else on the other queues. We then manually
> triggered GC on the broker but heap utilization remained consistent at 80%.
> Next, we manually deleted each message from the suspend queue and forced
> another GC. At this point, heap utilization dropped back to normal levels.
We are scratching our heads at the moment and looking for suggestions from
> Qpid gurus. Is there any condition in which an unconsumed message can
> prevent GC from reclaiming memory?
>
> Our setup is as follows: non-persistent messages, asynchronous onMessage
> delivery, transaction sessions.
>
> Thanks in advance! Dan
>
>
>
> --
> View this message in context:
> http://qpid.2158936.n2.nabble.com/Possible-for-unconsumed-messages-to-prevent-GC-from-reclaiming-memory-held-by-prior-messages-tp7615368.html
> Sent from the Apache Qpid users mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>
>