You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@qpid.apache.org by helenkwong <he...@gmail.com> on 2015/10/31 02:26:57 UTC
Unconsumed messages can still prevent GC of consumed messages
Hi Rob,
We've been using the v0.32 Java broker and haven't run into memory leak
problems for a while, but we've just seen something similar to what we saw
before happen again and would appreciate your help. See our previous thread
here (from almost exactly a year ago):
http://qpid.2158936.n2.nabble.com/Possible-for-unconsumed-messages-to-prevent-GC-from-reclaiming-memory-held-by-prior-messages-tp7615368p7615933.html.
The pattern, like before, is that when there's an unconsumed message that
has been sitting in a Queue A for a long time, we can end up being unable to
reclaim old messages in another Queue B, even though these messages have
already been consumed long ago. The link seems to be that at some point in
the past, a consumer of Queue A and a consumer of Queue B shared the same
ServerSession. And although the session and both consumers were closed long
ago, Queue A's head entry still refers to the old Queue A consumer via its
_stateChangeListeners list. Consequently, we're still holding onto the old
messages that were in Queue B, via the common session.
More specifically, the screenshot below shows how we go from a Queue A
consumer referring to old Queue B messages: QueueConsumerImpl -> _target ->
_session -> commands table -> MessageTransfer belonging to a Queue B message
consumed long ago -> completionListener -> old PriorityQueueEntry of Queue B
(highlighted in screenshot) -> many more old queue entries.
<http://qpid.2158936.n2.nabble.com/file/n7633133/qpid_broker_heap_queueAConsumerToQueueBEntry.png>
But the Queue A QueueConsumerImpl was created long ago and is closed, so it
seems like it should have been cleaned up in the first place. The second
screenshot below shows a path from a GC root to the consumer. It looks like
the _stateChangeListeners list of the head of the queue (which has been
sitting in Queue A for a long time) still contains this old
QueueConsumerImpl. We regularly create and close consumers to our queues, so
it seems like these old consumers can then accumulate over time, along with
old consumers and old queue entries they reference. (Your fix made things
much better though when the old consumers accumulate.)
<http://qpid.2158936.n2.nabble.com/file/n7633133/qpid_broker_heap_pathToQueueAConsumer.png>
So, the problem seems to be that even though a consumer has been closed, it
remains in a queue entry's _stateChangeListeners list. I see QueueEntryImpl
has a removeStateChangeListener() method, but can't see where its caller
QueueEntryListener.stateChanged() gets called. So I'm not sure how we can
get into this situation. Would really appreciate your insight on this.
Thanks!
Helen
--
View this message in context: http://qpid.2158936.n2.nabble.com/Unconsumed-messages-can-still-prevent-GC-of-consumed-messages-tp7633133.html
Sent from the Apache Qpid users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: Unconsumed messages can still prevent GC of consumed messages
Posted by Helen Kwong <he...@gmail.com>.
Hi Rob, will the fix be committed to the 0.32 branch as well?
On Wed, Nov 4, 2015 at 3:11 PM, Rob Godfrey <ro...@gmail.com> wrote:
> Hi Helen,
>
> glad to hear the patch worked for you, we'll make sure that fix is in the
> upcoming 6.0 release (hopefully the Beta for that will be out by the end of
> this week),
>
> Cheers,
> Rob
>
> On 4 November 2015 at 21:57, Helen Kwong <he...@gmail.com> wrote:
>
> > Hi Rob,
> >
> > The patch fixed the issue in our tests. It looks like we ran into the
> > problem because we regularly create and close QueueBrowsers sharing the
> > same session to our set of queues, and while we constantly consume
> messages
> > from most queues with MessageConsumers, there can be a small number of
> > queues that we do not create any regular consumer to for a long time.
> >
> > Thanks again for fixing this so quickly (as usual)!
> >
> > Helen
> >
> >
> > On Sun, Nov 1, 2015 at 4:44 PM, Helen Kwong <he...@gmail.com>
> wrote:
> >
> > > Thanks a lot for looking into this and fixing it so quickly! Will try
> the
> > > patch and let you know.
> > >
> > > Helen
> > >
> > > On Sat, Oct 31, 2015 at 4:50 PM, Rob Godfrey <ro...@gmail.com>
> > > wrote:
> > >
> > >> I've raised a jira https://issues.apache.org/jira/browse/QPID-6820
> > >> covering
> > >> the issue with state change listeners not being removed - I've also
> > >> attached a patch which I believe should fix the issue.
> > >>
> > >> The issue will only occur where a consumer runs out of credit while
> > there
> > >> are still messages on the queue, and then that consumer is closed...
> and
> > >> the first (oldest) message it was unable to consume is not itself
> > consumed
> > >> (or even delivered) to any other consumer. The only way to free the
> > >> association is for the message to be acquired by some other party (it
> > >> doesn't need to be dequeued, just acquired - a session that started,
> > >> opened
> > >> a consumer, prefetched all the messages from the queue, and then
> closed
> > >> would be enough to unjam it).
> > >>
> > >> Basically the state listener is there in case the reason the consumer
> > was
> > >> unable to consume it was because it was a particularly large
> message...
> > if
> > >> it is subsequently acquired by a different consumer, the first
> consumer
> > >> needs to be woken up to look at subsequent messages on the queue to
> see
> > if
> > >> it can accept them.
> > >>
> > >> Let me know if you have a chance to test the patch, and if it solves
> > this
> > >> issue for you.
> > >>
> > >> Again, apologies that you've run into another obscure issue - but
> thanks
> > >> for taking the time to work out what was going on - much appreciated!
> > >>
> > >> -- Rob
> > >>
> > >> On 31 October 2015 at 07:55, Rob Godfrey <ro...@gmail.com>
> > wrote:
> > >>
> > >> > Hi Helen,
> > >> >
> > >> > thanks for the detailed analysis - I'll look into this today.
> > >> >
> > >> > -- Rob
> > >> >
> > >> > On 31 October 2015 at 01:26, helenkwong <he...@gmail.com>
> wrote:
> > >> >
> > >> >> Hi Rob,
> > >> >>
> > >> >> We've been using the v0.32 Java broker and haven't run into memory
> > leak
> > >> >> problems for a while, but we've just seen something similar to what
> > we
> > >> saw
> > >> >> before happen again and would appreciate your help. See our
> previous
> > >> >> thread
> > >> >> here (from almost exactly a year ago):
> > >> >>
> > >> >>
> > >>
> >
> http://qpid.2158936.n2.nabble.com/Possible-for-unconsumed-messages-to-prevent-GC-from-reclaiming-memory-held-by-prior-messages-tp7615368p7615933.html
> > >> >> .
> > >> >>
> > >> >> The pattern, like before, is that when there's an unconsumed
> message
> > >> that
> > >> >> has been sitting in a Queue A for a long time, we can end up being
> > >> unable
> > >> >> to
> > >> >> reclaim old messages in another Queue B, even though these messages
> > >> have
> > >> >> already been consumed long ago. The link seems to be that at some
> > >> point in
> > >> >> the past, a consumer of Queue A and a consumer of Queue B shared
> the
> > >> same
> > >> >> ServerSession. And although the session and both consumers were
> > closed
> > >> >> long
> > >> >> ago, Queue A's head entry still refers to the old Queue A consumer
> > via
> > >> its
> > >> >> _stateChangeListeners list. Consequently, we're still holding onto
> > the
> > >> old
> > >> >> messages that were in Queue B, via the common session.
> > >> >>
> > >> >> More specifically, the screenshot below shows how we go from a
> Queue
> > A
> > >> >> consumer referring to old Queue B messages: QueueConsumerImpl ->
> > >> _target
> > >> >> ->
> > >> >> _session -> commands table -> MessageTransfer belonging to a Queue
> B
> > >> >> message
> > >> >> consumed long ago -> completionListener -> old PriorityQueueEntry
> of
> > >> >> Queue B
> > >> >> (highlighted in screenshot) -> many more old queue entries.
> > >> >> <
> > >> >>
> > >>
> >
> http://qpid.2158936.n2.nabble.com/file/n7633133/qpid_broker_heap_queueAConsumerToQueueBEntry.png
> > >> >> >
> > >> >>
> > >> >> But the Queue A QueueConsumerImpl was created long ago and is
> closed,
> > >> so
> > >> >> it
> > >> >> seems like it should have been cleaned up in the first place. The
> > >> second
> > >> >> screenshot below shows a path from a GC root to the consumer. It
> > looks
> > >> >> like
> > >> >> the _stateChangeListeners list of the head of the queue (which has
> > been
> > >> >> sitting in Queue A for a long time) still contains this old
> > >> >> QueueConsumerImpl. We regularly create and close consumers to our
> > >> queues,
> > >> >> so
> > >> >> it seems like these old consumers can then accumulate over time,
> > along
> > >> >> with
> > >> >> old consumers and old queue entries they reference. (Your fix made
> > >> things
> > >> >> much better though when the old consumers accumulate.)
> > >> >> <
> > >> >>
> > >>
> >
> http://qpid.2158936.n2.nabble.com/file/n7633133/qpid_broker_heap_pathToQueueAConsumer.png
> > >> >> >
> > >> >>
> > >> >> So, the problem seems to be that even though a consumer has been
> > >> closed,
> > >> >> it
> > >> >> remains in a queue entry's _stateChangeListeners list. I see
> > >> >> QueueEntryImpl
> > >> >> has a removeStateChangeListener() method, but can't see where its
> > >> caller
> > >> >> QueueEntryListener.stateChanged() gets called. So I'm not sure how
> we
> > >> can
> > >> >> get into this situation. Would really appreciate your insight on
> > this.
> > >> >>
> > >> >> Thanks!
> > >> >>
> > >> >> Helen
> > >> >>
> > >> >>
> > >> >>
> > >> >> --
> > >> >> View this message in context:
> > >> >>
> > >>
> >
> http://qpid.2158936.n2.nabble.com/Unconsumed-messages-can-still-prevent-GC-of-consumed-messages-tp7633133.html
> > >> >> Sent from the Apache Qpid users mailing list archive at Nabble.com.
> > >> >>
> > >> >>
> ---------------------------------------------------------------------
> > >> >> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> > >> >> For additional commands, e-mail: users-help@qpid.apache.org
> > >> >>
> > >> >>
> > >> >
> > >>
> > >
> > >
> >
>
Re: Unconsumed messages can still prevent GC of consumed messages
Posted by Rob Godfrey <ro...@gmail.com>.
Hi Helen,
glad to hear the patch worked for you, we'll make sure that fix is in the
upcoming 6.0 release (hopefully the Beta for that will be out by the end of
this week),
Cheers,
Rob
On 4 November 2015 at 21:57, Helen Kwong <he...@gmail.com> wrote:
> Hi Rob,
>
> The patch fixed the issue in our tests. It looks like we ran into the
> problem because we regularly create and close QueueBrowsers sharing the
> same session to our set of queues, and while we constantly consume messages
> from most queues with MessageConsumers, there can be a small number of
> queues that we do not create any regular consumer to for a long time.
>
> Thanks again for fixing this so quickly (as usual)!
>
> Helen
>
>
> On Sun, Nov 1, 2015 at 4:44 PM, Helen Kwong <he...@gmail.com> wrote:
>
> > Thanks a lot for looking into this and fixing it so quickly! Will try the
> > patch and let you know.
> >
> > Helen
> >
> > On Sat, Oct 31, 2015 at 4:50 PM, Rob Godfrey <ro...@gmail.com>
> > wrote:
> >
> >> I've raised a jira https://issues.apache.org/jira/browse/QPID-6820
> >> covering
> >> the issue with state change listeners not being removed - I've also
> >> attached a patch which I believe should fix the issue.
> >>
> >> The issue will only occur where a consumer runs out of credit while
> there
> >> are still messages on the queue, and then that consumer is closed... and
> >> the first (oldest) message it was unable to consume is not itself
> consumed
> >> (or even delivered) to any other consumer. The only way to free the
> >> association is for the message to be acquired by some other party (it
> >> doesn't need to be dequeued, just acquired - a session that started,
> >> opened
> >> a consumer, prefetched all the messages from the queue, and then closed
> >> would be enough to unjam it).
> >>
> >> Basically the state listener is there in case the reason the consumer
> was
> >> unable to consume it was because it was a particularly large message...
> if
> >> it is subsequently acquired by a different consumer, the first consumer
> >> needs to be woken up to look at subsequent messages on the queue to see
> if
> >> it can accept them.
> >>
> >> Let me know if you have a chance to test the patch, and if it solves
> this
> >> issue for you.
> >>
> >> Again, apologies that you've run into another obscure issue - but thanks
> >> for taking the time to work out what was going on - much appreciated!
> >>
> >> -- Rob
> >>
> >> On 31 October 2015 at 07:55, Rob Godfrey <ro...@gmail.com>
> wrote:
> >>
> >> > Hi Helen,
> >> >
> >> > thanks for the detailed analysis - I'll look into this today.
> >> >
> >> > -- Rob
> >> >
> >> > On 31 October 2015 at 01:26, helenkwong <he...@gmail.com> wrote:
> >> >
> >> >> Hi Rob,
> >> >>
> >> >> We've been using the v0.32 Java broker and haven't run into memory
> leak
> >> >> problems for a while, but we've just seen something similar to what
> we
> >> saw
> >> >> before happen again and would appreciate your help. See our previous
> >> >> thread
> >> >> here (from almost exactly a year ago):
> >> >>
> >> >>
> >>
> http://qpid.2158936.n2.nabble.com/Possible-for-unconsumed-messages-to-prevent-GC-from-reclaiming-memory-held-by-prior-messages-tp7615368p7615933.html
> >> >> .
> >> >>
> >> >> The pattern, like before, is that when there's an unconsumed message
> >> that
> >> >> has been sitting in a Queue A for a long time, we can end up being
> >> unable
> >> >> to
> >> >> reclaim old messages in another Queue B, even though these messages
> >> have
> >> >> already been consumed long ago. The link seems to be that at some
> >> point in
> >> >> the past, a consumer of Queue A and a consumer of Queue B shared the
> >> same
> >> >> ServerSession. And although the session and both consumers were
> closed
> >> >> long
> >> >> ago, Queue A's head entry still refers to the old Queue A consumer
> via
> >> its
> >> >> _stateChangeListeners list. Consequently, we're still holding onto
> the
> >> old
> >> >> messages that were in Queue B, via the common session.
> >> >>
> >> >> More specifically, the screenshot below shows how we go from a Queue
> A
> >> >> consumer referring to old Queue B messages: QueueConsumerImpl ->
> >> _target
> >> >> ->
> >> >> _session -> commands table -> MessageTransfer belonging to a Queue B
> >> >> message
> >> >> consumed long ago -> completionListener -> old PriorityQueueEntry of
> >> >> Queue B
> >> >> (highlighted in screenshot) -> many more old queue entries.
> >> >> <
> >> >>
> >>
> http://qpid.2158936.n2.nabble.com/file/n7633133/qpid_broker_heap_queueAConsumerToQueueBEntry.png
> >> >> >
> >> >>
> >> >> But the Queue A QueueConsumerImpl was created long ago and is closed,
> >> so
> >> >> it
> >> >> seems like it should have been cleaned up in the first place. The
> >> second
> >> >> screenshot below shows a path from a GC root to the consumer. It
> looks
> >> >> like
> >> >> the _stateChangeListeners list of the head of the queue (which has
> been
> >> >> sitting in Queue A for a long time) still contains this old
> >> >> QueueConsumerImpl. We regularly create and close consumers to our
> >> queues,
> >> >> so
> >> >> it seems like these old consumers can then accumulate over time,
> along
> >> >> with
> >> >> old consumers and old queue entries they reference. (Your fix made
> >> things
> >> >> much better though when the old consumers accumulate.)
> >> >> <
> >> >>
> >>
> http://qpid.2158936.n2.nabble.com/file/n7633133/qpid_broker_heap_pathToQueueAConsumer.png
> >> >> >
> >> >>
> >> >> So, the problem seems to be that even though a consumer has been
> >> closed,
> >> >> it
> >> >> remains in a queue entry's _stateChangeListeners list. I see
> >> >> QueueEntryImpl
> >> >> has a removeStateChangeListener() method, but can't see where its
> >> caller
> >> >> QueueEntryListener.stateChanged() gets called. So I'm not sure how we
> >> can
> >> >> get into this situation. Would really appreciate your insight on
> this.
> >> >>
> >> >> Thanks!
> >> >>
> >> >> Helen
> >> >>
> >> >>
> >> >>
> >> >> --
> >> >> View this message in context:
> >> >>
> >>
> http://qpid.2158936.n2.nabble.com/Unconsumed-messages-can-still-prevent-GC-of-consumed-messages-tp7633133.html
> >> >> Sent from the Apache Qpid users mailing list archive at Nabble.com.
> >> >>
> >> >> ---------------------------------------------------------------------
> >> >> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> >> >> For additional commands, e-mail: users-help@qpid.apache.org
> >> >>
> >> >>
> >> >
> >>
> >
> >
>
Re: Unconsumed messages can still prevent GC of consumed messages
Posted by Helen Kwong <he...@gmail.com>.
Hi Rob,
The patch fixed the issue in our tests. It looks like we ran into the
problem because we regularly create and close QueueBrowsers sharing the
same session to our set of queues, and while we constantly consume messages
from most queues with MessageConsumers, there can be a small number of
queues that we do not create any regular consumer to for a long time.
Thanks again for fixing this so quickly (as usual)!
Helen
On Sun, Nov 1, 2015 at 4:44 PM, Helen Kwong <he...@gmail.com> wrote:
> Thanks a lot for looking into this and fixing it so quickly! Will try the
> patch and let you know.
>
> Helen
>
> On Sat, Oct 31, 2015 at 4:50 PM, Rob Godfrey <ro...@gmail.com>
> wrote:
>
>> I've raised a jira https://issues.apache.org/jira/browse/QPID-6820
>> covering
>> the issue with state change listeners not being removed - I've also
>> attached a patch which I believe should fix the issue.
>>
>> The issue will only occur where a consumer runs out of credit while there
>> are still messages on the queue, and then that consumer is closed... and
>> the first (oldest) message it was unable to consume is not itself consumed
>> (or even delivered) to any other consumer. The only way to free the
>> association is for the message to be acquired by some other party (it
>> doesn't need to be dequeued, just acquired - a session that started,
>> opened
>> a consumer, prefetched all the messages from the queue, and then closed
>> would be enough to unjam it).
>>
>> Basically the state listener is there in case the reason the consumer was
>> unable to consume it was because it was a particularly large message... if
>> it is subsequently acquired by a different consumer, the first consumer
>> needs to be woken up to look at subsequent messages on the queue to see if
>> it can accept them.
>>
>> Let me know if you have a chance to test the patch, and if it solves this
>> issue for you.
>>
>> Again, apologies that you've run into another obscure issue - but thanks
>> for taking the time to work out what was going on - much appreciated!
>>
>> -- Rob
>>
>> On 31 October 2015 at 07:55, Rob Godfrey <ro...@gmail.com> wrote:
>>
>> > Hi Helen,
>> >
>> > thanks for the detailed analysis - I'll look into this today.
>> >
>> > -- Rob
>> >
>> > On 31 October 2015 at 01:26, helenkwong <he...@gmail.com> wrote:
>> >
>> >> Hi Rob,
>> >>
>> >> We've been using the v0.32 Java broker and haven't run into memory leak
>> >> problems for a while, but we've just seen something similar to what we
>> saw
>> >> before happen again and would appreciate your help. See our previous
>> >> thread
>> >> here (from almost exactly a year ago):
>> >>
>> >>
>> http://qpid.2158936.n2.nabble.com/Possible-for-unconsumed-messages-to-prevent-GC-from-reclaiming-memory-held-by-prior-messages-tp7615368p7615933.html
>> >> .
>> >>
>> >> The pattern, like before, is that when there's an unconsumed message
>> that
>> >> has been sitting in a Queue A for a long time, we can end up being
>> unable
>> >> to
>> >> reclaim old messages in another Queue B, even though these messages
>> have
>> >> already been consumed long ago. The link seems to be that at some
>> point in
>> >> the past, a consumer of Queue A and a consumer of Queue B shared the
>> same
>> >> ServerSession. And although the session and both consumers were closed
>> >> long
>> >> ago, Queue A's head entry still refers to the old Queue A consumer via
>> its
>> >> _stateChangeListeners list. Consequently, we're still holding onto the
>> old
>> >> messages that were in Queue B, via the common session.
>> >>
>> >> More specifically, the screenshot below shows how we go from a Queue A
>> >> consumer referring to old Queue B messages: QueueConsumerImpl ->
>> _target
>> >> ->
>> >> _session -> commands table -> MessageTransfer belonging to a Queue B
>> >> message
>> >> consumed long ago -> completionListener -> old PriorityQueueEntry of
>> >> Queue B
>> >> (highlighted in screenshot) -> many more old queue entries.
>> >> <
>> >>
>> http://qpid.2158936.n2.nabble.com/file/n7633133/qpid_broker_heap_queueAConsumerToQueueBEntry.png
>> >> >
>> >>
>> >> But the Queue A QueueConsumerImpl was created long ago and is closed,
>> so
>> >> it
>> >> seems like it should have been cleaned up in the first place. The
>> second
>> >> screenshot below shows a path from a GC root to the consumer. It looks
>> >> like
>> >> the _stateChangeListeners list of the head of the queue (which has been
>> >> sitting in Queue A for a long time) still contains this old
>> >> QueueConsumerImpl. We regularly create and close consumers to our
>> queues,
>> >> so
>> >> it seems like these old consumers can then accumulate over time, along
>> >> with
>> >> old consumers and old queue entries they reference. (Your fix made
>> things
>> >> much better though when the old consumers accumulate.)
>> >> <
>> >>
>> http://qpid.2158936.n2.nabble.com/file/n7633133/qpid_broker_heap_pathToQueueAConsumer.png
>> >> >
>> >>
>> >> So, the problem seems to be that even though a consumer has been
>> closed,
>> >> it
>> >> remains in a queue entry's _stateChangeListeners list. I see
>> >> QueueEntryImpl
>> >> has a removeStateChangeListener() method, but can't see where its
>> caller
>> >> QueueEntryListener.stateChanged() gets called. So I'm not sure how we
>> can
>> >> get into this situation. Would really appreciate your insight on this.
>> >>
>> >> Thanks!
>> >>
>> >> Helen
>> >>
>> >>
>> >>
>> >> --
>> >> View this message in context:
>> >>
>> http://qpid.2158936.n2.nabble.com/Unconsumed-messages-can-still-prevent-GC-of-consumed-messages-tp7633133.html
>> >> Sent from the Apache Qpid users mailing list archive at Nabble.com.
>> >>
>> >> ---------------------------------------------------------------------
>> >> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
>> >> For additional commands, e-mail: users-help@qpid.apache.org
>> >>
>> >>
>> >
>>
>
>
Re: Unconsumed messages can still prevent GC of consumed messages
Posted by Helen Kwong <he...@gmail.com>.
Thanks a lot for looking into this and fixing it so quickly! Will try the
patch and let you know.
Helen
On Sat, Oct 31, 2015 at 4:50 PM, Rob Godfrey <ro...@gmail.com>
wrote:
> I've raised a jira https://issues.apache.org/jira/browse/QPID-6820
> covering
> the issue with state change listeners not being removed - I've also
> attached a patch which I believe should fix the issue.
>
> The issue will only occur where a consumer runs out of credit while there
> are still messages on the queue, and then that consumer is closed... and
> the first (oldest) message it was unable to consume is not itself consumed
> (or even delivered) to any other consumer. The only way to free the
> association is for the message to be acquired by some other party (it
> doesn't need to be dequeued, just acquired - a session that started, opened
> a consumer, prefetched all the messages from the queue, and then closed
> would be enough to unjam it).
>
> Basically the state listener is there in case the reason the consumer was
> unable to consume it was because it was a particularly large message... if
> it is subsequently acquired by a different consumer, the first consumer
> needs to be woken up to look at subsequent messages on the queue to see if
> it can accept them.
>
> Let me know if you have a chance to test the patch, and if it solves this
> issue for you.
>
> Again, apologies that you've run into another obscure issue - but thanks
> for taking the time to work out what was going on - much appreciated!
>
> -- Rob
>
> On 31 October 2015 at 07:55, Rob Godfrey <ro...@gmail.com> wrote:
>
> > Hi Helen,
> >
> > thanks for the detailed analysis - I'll look into this today.
> >
> > -- Rob
> >
> > On 31 October 2015 at 01:26, helenkwong <he...@gmail.com> wrote:
> >
> >> Hi Rob,
> >>
> >> We've been using the v0.32 Java broker and haven't run into memory leak
> >> problems for a while, but we've just seen something similar to what we
> saw
> >> before happen again and would appreciate your help. See our previous
> >> thread
> >> here (from almost exactly a year ago):
> >>
> >>
> http://qpid.2158936.n2.nabble.com/Possible-for-unconsumed-messages-to-prevent-GC-from-reclaiming-memory-held-by-prior-messages-tp7615368p7615933.html
> >> .
> >>
> >> The pattern, like before, is that when there's an unconsumed message
> that
> >> has been sitting in a Queue A for a long time, we can end up being
> unable
> >> to
> >> reclaim old messages in another Queue B, even though these messages have
> >> already been consumed long ago. The link seems to be that at some point
> in
> >> the past, a consumer of Queue A and a consumer of Queue B shared the
> same
> >> ServerSession. And although the session and both consumers were closed
> >> long
> >> ago, Queue A's head entry still refers to the old Queue A consumer via
> its
> >> _stateChangeListeners list. Consequently, we're still holding onto the
> old
> >> messages that were in Queue B, via the common session.
> >>
> >> More specifically, the screenshot below shows how we go from a Queue A
> >> consumer referring to old Queue B messages: QueueConsumerImpl -> _target
> >> ->
> >> _session -> commands table -> MessageTransfer belonging to a Queue B
> >> message
> >> consumed long ago -> completionListener -> old PriorityQueueEntry of
> >> Queue B
> >> (highlighted in screenshot) -> many more old queue entries.
> >> <
> >>
> http://qpid.2158936.n2.nabble.com/file/n7633133/qpid_broker_heap_queueAConsumerToQueueBEntry.png
> >> >
> >>
> >> But the Queue A QueueConsumerImpl was created long ago and is closed, so
> >> it
> >> seems like it should have been cleaned up in the first place. The second
> >> screenshot below shows a path from a GC root to the consumer. It looks
> >> like
> >> the _stateChangeListeners list of the head of the queue (which has been
> >> sitting in Queue A for a long time) still contains this old
> >> QueueConsumerImpl. We regularly create and close consumers to our
> queues,
> >> so
> >> it seems like these old consumers can then accumulate over time, along
> >> with
> >> old consumers and old queue entries they reference. (Your fix made
> things
> >> much better though when the old consumers accumulate.)
> >> <
> >>
> http://qpid.2158936.n2.nabble.com/file/n7633133/qpid_broker_heap_pathToQueueAConsumer.png
> >> >
> >>
> >> So, the problem seems to be that even though a consumer has been closed,
> >> it
> >> remains in a queue entry's _stateChangeListeners list. I see
> >> QueueEntryImpl
> >> has a removeStateChangeListener() method, but can't see where its caller
> >> QueueEntryListener.stateChanged() gets called. So I'm not sure how we
> can
> >> get into this situation. Would really appreciate your insight on this.
> >>
> >> Thanks!
> >>
> >> Helen
> >>
> >>
> >>
> >> --
> >> View this message in context:
> >>
> http://qpid.2158936.n2.nabble.com/Unconsumed-messages-can-still-prevent-GC-of-consumed-messages-tp7633133.html
> >> Sent from the Apache Qpid users mailing list archive at Nabble.com.
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> >> For additional commands, e-mail: users-help@qpid.apache.org
> >>
> >>
> >
>
Re: Unconsumed messages can still prevent GC of consumed messages
Posted by Rob Godfrey <ro...@gmail.com>.
I've raised a jira https://issues.apache.org/jira/browse/QPID-6820 covering
the issue with state change listeners not being removed - I've also
attached a patch which I believe should fix the issue.
The issue will only occur where a consumer runs out of credit while there
are still messages on the queue, and then that consumer is closed... and
the first (oldest) message it was unable to consume is not itself consumed
(or even delivered) to any other consumer. The only way to free the
association is for the message to be acquired by some other party (it
doesn't need to be dequeued, just acquired - a session that started, opened
a consumer, prefetched all the messages from the queue, and then closed
would be enough to unjam it).
Basically the state listener is there in case the reason the consumer was
unable to consume it was because it was a particularly large message... if
it is subsequently acquired by a different consumer, the first consumer
needs to be woken up to look at subsequent messages on the queue to see if
it can accept them.
Let me know if you have a chance to test the patch, and if it solves this
issue for you.
Again, apologies that you've run into another obscure issue - but thanks
for taking the time to work out what was going on - much appreciated!
-- Rob
On 31 October 2015 at 07:55, Rob Godfrey <ro...@gmail.com> wrote:
> Hi Helen,
>
> thanks for the detailed analysis - I'll look into this today.
>
> -- Rob
>
> On 31 October 2015 at 01:26, helenkwong <he...@gmail.com> wrote:
>
>> Hi Rob,
>>
>> We've been using the v0.32 Java broker and haven't run into memory leak
>> problems for a while, but we've just seen something similar to what we saw
>> before happen again and would appreciate your help. See our previous
>> thread
>> here (from almost exactly a year ago):
>>
>> http://qpid.2158936.n2.nabble.com/Possible-for-unconsumed-messages-to-prevent-GC-from-reclaiming-memory-held-by-prior-messages-tp7615368p7615933.html
>> .
>>
>> The pattern, like before, is that when there's an unconsumed message that
>> has been sitting in a Queue A for a long time, we can end up being unable
>> to
>> reclaim old messages in another Queue B, even though these messages have
>> already been consumed long ago. The link seems to be that at some point in
>> the past, a consumer of Queue A and a consumer of Queue B shared the same
>> ServerSession. And although the session and both consumers were closed
>> long
>> ago, Queue A's head entry still refers to the old Queue A consumer via its
>> _stateChangeListeners list. Consequently, we're still holding onto the old
>> messages that were in Queue B, via the common session.
>>
>> More specifically, the screenshot below shows how we go from a Queue A
>> consumer referring to old Queue B messages: QueueConsumerImpl -> _target
>> ->
>> _session -> commands table -> MessageTransfer belonging to a Queue B
>> message
>> consumed long ago -> completionListener -> old PriorityQueueEntry of
>> Queue B
>> (highlighted in screenshot) -> many more old queue entries.
>> <
>> http://qpid.2158936.n2.nabble.com/file/n7633133/qpid_broker_heap_queueAConsumerToQueueBEntry.png
>> >
>>
>> But the Queue A QueueConsumerImpl was created long ago and is closed, so
>> it
>> seems like it should have been cleaned up in the first place. The second
>> screenshot below shows a path from a GC root to the consumer. It looks
>> like
>> the _stateChangeListeners list of the head of the queue (which has been
>> sitting in Queue A for a long time) still contains this old
>> QueueConsumerImpl. We regularly create and close consumers to our queues,
>> so
>> it seems like these old consumers can then accumulate over time, along
>> with
>> old consumers and old queue entries they reference. (Your fix made things
>> much better though when the old consumers accumulate.)
>> <
>> http://qpid.2158936.n2.nabble.com/file/n7633133/qpid_broker_heap_pathToQueueAConsumer.png
>> >
>>
>> So, the problem seems to be that even though a consumer has been closed,
>> it
>> remains in a queue entry's _stateChangeListeners list. I see
>> QueueEntryImpl
>> has a removeStateChangeListener() method, but can't see where its caller
>> QueueEntryListener.stateChanged() gets called. So I'm not sure how we can
>> get into this situation. Would really appreciate your insight on this.
>>
>> Thanks!
>>
>> Helen
>>
>>
>>
>> --
>> View this message in context:
>> http://qpid.2158936.n2.nabble.com/Unconsumed-messages-can-still-prevent-GC-of-consumed-messages-tp7633133.html
>> Sent from the Apache Qpid users mailing list archive at Nabble.com.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
>> For additional commands, e-mail: users-help@qpid.apache.org
>>
>>
>
Re: Unconsumed messages can still prevent GC of consumed messages
Posted by Rob Godfrey <ro...@gmail.com>.
Hi Helen,
thanks for the detailed analysis - I'll look into this today.
-- Rob
On 31 October 2015 at 01:26, helenkwong <he...@gmail.com> wrote:
> Hi Rob,
>
> We've been using the v0.32 Java broker and haven't run into memory leak
> problems for a while, but we've just seen something similar to what we saw
> before happen again and would appreciate your help. See our previous thread
> here (from almost exactly a year ago):
>
> http://qpid.2158936.n2.nabble.com/Possible-for-unconsumed-messages-to-prevent-GC-from-reclaiming-memory-held-by-prior-messages-tp7615368p7615933.html
> .
>
> The pattern, like before, is that when there's an unconsumed message that
> has been sitting in a Queue A for a long time, we can end up being unable
> to
> reclaim old messages in another Queue B, even though these messages have
> already been consumed long ago. The link seems to be that at some point in
> the past, a consumer of Queue A and a consumer of Queue B shared the same
> ServerSession. And although the session and both consumers were closed long
> ago, Queue A's head entry still refers to the old Queue A consumer via its
> _stateChangeListeners list. Consequently, we're still holding onto the old
> messages that were in Queue B, via the common session.
>
> More specifically, the screenshot below shows how we go from a Queue A
> consumer referring to old Queue B messages: QueueConsumerImpl -> _target ->
> _session -> commands table -> MessageTransfer belonging to a Queue B
> message
> consumed long ago -> completionListener -> old PriorityQueueEntry of Queue
> B
> (highlighted in screenshot) -> many more old queue entries.
> <
> http://qpid.2158936.n2.nabble.com/file/n7633133/qpid_broker_heap_queueAConsumerToQueueBEntry.png
> >
>
> But the Queue A QueueConsumerImpl was created long ago and is closed, so it
> seems like it should have been cleaned up in the first place. The second
> screenshot below shows a path from a GC root to the consumer. It looks like
> the _stateChangeListeners list of the head of the queue (which has been
> sitting in Queue A for a long time) still contains this old
> QueueConsumerImpl. We regularly create and close consumers to our queues,
> so
> it seems like these old consumers can then accumulate over time, along with
> old consumers and old queue entries they reference. (Your fix made things
> much better though when the old consumers accumulate.)
> <
> http://qpid.2158936.n2.nabble.com/file/n7633133/qpid_broker_heap_pathToQueueAConsumer.png
> >
>
> So, the problem seems to be that even though a consumer has been closed, it
> remains in a queue entry's _stateChangeListeners list. I see QueueEntryImpl
> has a removeStateChangeListener() method, but can't see where its caller
> QueueEntryListener.stateChanged() gets called. So I'm not sure how we can
> get into this situation. Would really appreciate your insight on this.
>
> Thanks!
>
> Helen
>
>
>
> --
> View this message in context:
> http://qpid.2158936.n2.nabble.com/Unconsumed-messages-can-still-prevent-GC-of-consumed-messages-tp7633133.html
> Sent from the Apache Qpid users mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>
>