You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@qpid.apache.org by "Andrew Kennedy (JIRA)" <qp...@incubator.apache.org> on 2010/06/25 13:14:49 UTC
[jira] Created: (QPID-2694) Memory leak in 0-8/0-9 AMQSessions on
channel close
Memory leak in 0-8/0-9 AMQSessions on channel close
---------------------------------------------------
Key: QPID-2694
URL: https://issues.apache.org/jira/browse/QPID-2694
Project: Qpid
Issue Type: Bug
Components: Java Broker
Affects Versions: 0.6, 0.5, 0.7
Reporter: Andrew Kennedy
Fix For: 0.7, 0.6, 0.5
When closing a 0-8/0-9 AMQSession, the ChannelCloseOk returned by the broker is not handled in the client, leading to a memory leak with AMQProtocolSession objects being left. This is particularly evident with frameworks like Mule which open a session per connection.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org
[jira] Assigned: (QPID-2694) Memory leak in 0-8/0-9 AMQSessions on
channel close
Posted by "Marnie McCormack (JIRA)" <qp...@incubator.apache.org>.
[ https://issues.apache.org/jira/browse/QPID-2694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Marnie McCormack reassigned QPID-2694:
--------------------------------------
Assignee: Marnie McCormack
> Memory leak in 0-8/0-9 AMQSessions on channel close
> ---------------------------------------------------
>
> Key: QPID-2694
> URL: https://issues.apache.org/jira/browse/QPID-2694
> Project: Qpid
> Issue Type: Bug
> Components: Java Broker
> Affects Versions: 0.5, 0.6, 0.7
> Reporter: Andrew Kennedy
> Assignee: Marnie McCormack
> Fix For: 0.5, 0.6, 0.7
>
> Attachments: 0001-QPID-2694-Fix-channel-close-memory-leak.patch
>
>
> When closing a 0-8/0-9 AMQSession, the ChannelCloseOk returned by the broker is not handled in the client, leading to a memory leak with AMQProtocolSession objects being left. This is particularly evident with frameworks like Mule which open a session per connection.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org
[jira] Commented: (QPID-2694) Memory leak in 0-8/0-9 AMQSessions on
channel close
Posted by "Marnie McCormack (JIRA)" <qp...@incubator.apache.org>.
[ https://issues.apache.org/jira/browse/QPID-2694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12885315#action_12885315 ]
Marnie McCormack commented on QPID-2694:
----------------------------------------
Discussed on list/with Andrew how we currently test this type of case i.e. protocol handling where a close-ok should cause an object to be cleaned up. No existing model to use, agreed a new JIRA for testing this type of protocol/rules logic best idea. Will now commit to trunk.
> Memory leak in 0-8/0-9 AMQSessions on channel close
> ---------------------------------------------------
>
> Key: QPID-2694
> URL: https://issues.apache.org/jira/browse/QPID-2694
> Project: Qpid
> Issue Type: Bug
> Components: Java Broker
> Affects Versions: 0.5, 0.6, 0.7
> Reporter: Andrew Kennedy
> Assignee: Marnie McCormack
> Fix For: 0.5, 0.6, 0.7
>
> Attachments: 0001-QPID-2694-Fix-channel-close-memory-leak.patch
>
>
> When closing a 0-8/0-9 AMQSession, the ChannelCloseOk returned by the broker is not handled in the client, leading to a memory leak with AMQProtocolSession objects being left. This is particularly evident with frameworks like Mule which open a session per connection.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org
[jira] Resolved: (QPID-2694) Memory leak in 0-8/0-9 AMQSessions on
channel close
Posted by "Marnie McCormack (JIRA)" <qp...@incubator.apache.org>.
[ https://issues.apache.org/jira/browse/QPID-2694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Marnie McCormack resolved QPID-2694.
------------------------------------
Resolution: Fixed
Patches applied
> Memory leak in 0-8/0-9 AMQSessions on channel close
> ---------------------------------------------------
>
> Key: QPID-2694
> URL: https://issues.apache.org/jira/browse/QPID-2694
> Project: Qpid
> Issue Type: Bug
> Components: Java Broker
> Affects Versions: 0.5, 0.6, 0.7
> Reporter: Andrew Kennedy
> Assignee: Marnie McCormack
> Fix For: 0.7, 0.6, 0.5
>
> Attachments: 0001-QPID-2694-Fix-channel-close-memory-leak.patch
>
>
> When closing a 0-8/0-9 AMQSession, the ChannelCloseOk returned by the broker is not handled in the client, leading to a memory leak with AMQProtocolSession objects being left. This is particularly evident with frameworks like Mule which open a session per connection.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org
[jira] Updated: (QPID-2694) Memory leak in 0-8/0-9 AMQSessions on
channel close
Posted by "Andrew Kennedy (JIRA)" <qp...@incubator.apache.org>.
[ https://issues.apache.org/jira/browse/QPID-2694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrew Kennedy updated QPID-2694:
---------------------------------
Attachment: 0001-QPID-2694-Fix-channel-close-memory-leak.patch
Patch is for both 0.5.x branch and trunk code (same changes required).
> Memory leak in 0-8/0-9 AMQSessions on channel close
> ---------------------------------------------------
>
> Key: QPID-2694
> URL: https://issues.apache.org/jira/browse/QPID-2694
> Project: Qpid
> Issue Type: Bug
> Components: Java Broker
> Affects Versions: 0.5, 0.6, 0.7
> Reporter: Andrew Kennedy
> Fix For: 0.5, 0.6, 0.7
>
> Attachments: 0001-QPID-2694-Fix-channel-close-memory-leak.patch
>
>
> When closing a 0-8/0-9 AMQSession, the ChannelCloseOk returned by the broker is not handled in the client, leading to a memory leak with AMQProtocolSession objects being left. This is particularly evident with frameworks like Mule which open a session per connection.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org
Re: [jira] Commented: (QPID-2694) Memory leak in 0-8/0-9 AMQSessions
on channel close
Posted by Rafael Schloming <ra...@redhat.com>.
Marnie McCormack wrote:
> How do we currently test that for various operations (like Session close)
> the correct messages are dispatched i.e. so we avoid this case where the
> close-ok didn't go ?
>
> That was really what I was driving at on the test - obv we can stress
> test/profile on a one off basis outside of the codebase/
I haven't followed the details of this Jira, but I would say that in
general these sorts of protocol operations effect state changes in some
entity, e.g. the session and/or the conection, and we should test that
the state change has occurred. This probably involves testing against
the implementing class(es) directly as the JMS interface may not expose
the accessors necessary to verify that the state change has occurred.
--Rafael
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org
Re: [jira] Commented: (QPID-2694) Memory leak in 0-8/0-9 AMQSessions
on channel close
Posted by Marnie McCormack <ma...@googlemail.com>.
How do we currently test that for various operations (like Session close)
the correct messages are dispatched i.e. so we avoid this case where the
close-ok didn't go ?
That was really what I was driving at on the test - obv we can stress
test/profile on a one off basis outside of the codebase/
Marnie
On Wed, Jun 30, 2010 at 2:27 PM, Rajith Attapattu <ra...@gmail.com>wrote:
> I agree with Rafi.
> Memory leaks are not the sort that could be verified with a unit test
> that runs under a min.
> You need to run for a reasonably long time to detect trends.
>
> I used to have a nightly cron tasks that runs a producer consumer pair
> with transactions etc for 10 hours and records memory usage.
> We also ran a similar test for over a week before to do wrap around
> testing, but I also recorded memory usage and graphed it.
> If there is a memory leak it will be visible and we did infact find
> one issue where we were stuffing stuff into a map that grew very very
> slowly over time.
> However this wasn't even visible during the 10 hour run, but was only
> detected during the week long test.
> I don't think it's unreasonable run such a test once a release cycle.
>
> Once you identify an issue then maybe you could try and write a test
> that could reproduce it in a short time frame and then analyse using a
> profiler etc..
> But again in order to verify I would still recommend running for at
> least 10 hrs plus.
>
> Just my 2 cents and hope it helps.
>
> Rajith
>
> On Wed, Jun 30, 2010 at 8:00 AM, Rafael Schloming <ra...@redhat.com>
> wrote:
> > Emmanuel Bourg wrote:
> >>
> >> Le 30/06/2010 09:20, Andrew Kennedy a écrit :
> >>
> >>> So, what do people think would be the best and simplest way to test
> this
> >>> leak is fixed? Or am I over complicating this for myself before I've
> had
> >>> enough coffee?
> >>
> >> I admit I don't remember seeing a unit test for a memory leak. This is
> >> usually verified once by a profiler. I see two approaches to write such
> a
> >> test:
> >> - generate a heap dump and inspect it
> >> - if the test can get a reference on the object that leaks, the garbage
> >> collection can be verified by a WeakReference combined with a
> ReferenceQueue
> >>
> >> public void testMemoryLeak() {
> >> Object leakingObject = getLeakingObject();
> >>
> >> ReferenceQueue queue = new ReferenceQueue();
> >> WeakReference reference = new WeakReference(leakingObject, queue);
> >> leakingObject = null;
> >>
> >> // so some operations supposed to dereference the object
> >>
> >> System.gc();
> >>
> >> assertTrue(reference.isEnqueued());
> >> }
> >
> > FWIW, I don't know how portable a test like this would be. According to
> the
> > doc, System.gc() only "suggests" that the JVM runs the garbage collector.
> >
> > It's probably also not that robust since you wouldn't detect leakage of
> any
> > ancillary objects.
> >
> > I think a targeted stress test is likely to be way more robust for this
> sort
> > of thing than a unit test. Of course that's the sort of thing you don't
> > really want to run once per checkin, more like once (hopefully) per
> release.
> >
> > Perhaps it would be useful to start a suite/category of longer running
> > "release" tests.
> >
> > --Rafael
> >
> > ---------------------------------------------------------------------
> > Apache Qpid - AMQP Messaging Implementation
> > Project: http://qpid.apache.org
> > Use/Interact: mailto:dev-subscribe@qpid.apache.org
> >
> >
>
>
>
> --
> Regards,
>
> Rajith Attapattu
> Red Hat
> http://rajith.2rlabs.com/
>
> ---------------------------------------------------------------------
> Apache Qpid - AMQP Messaging Implementation
> Project: http://qpid.apache.org
> Use/Interact: mailto:dev-subscribe@qpid.apache.org
>
>
Re: [jira] Commented: (QPID-2694) Memory leak in 0-8/0-9 AMQSessions
on channel close
Posted by Rajith Attapattu <ra...@gmail.com>.
I agree with Rafi.
Memory leaks are not the sort that could be verified with a unit test
that runs under a min.
You need to run for a reasonably long time to detect trends.
I used to have a nightly cron tasks that runs a producer consumer pair
with transactions etc for 10 hours and records memory usage.
We also ran a similar test for over a week before to do wrap around
testing, but I also recorded memory usage and graphed it.
If there is a memory leak it will be visible and we did infact find
one issue where we were stuffing stuff into a map that grew very very
slowly over time.
However this wasn't even visible during the 10 hour run, but was only
detected during the week long test.
I don't think it's unreasonable run such a test once a release cycle.
Once you identify an issue then maybe you could try and write a test
that could reproduce it in a short time frame and then analyse using a
profiler etc..
But again in order to verify I would still recommend running for at
least 10 hrs plus.
Just my 2 cents and hope it helps.
Rajith
On Wed, Jun 30, 2010 at 8:00 AM, Rafael Schloming <ra...@redhat.com> wrote:
> Emmanuel Bourg wrote:
>>
>> Le 30/06/2010 09:20, Andrew Kennedy a écrit :
>>
>>> So, what do people think would be the best and simplest way to test this
>>> leak is fixed? Or am I over complicating this for myself before I've had
>>> enough coffee?
>>
>> I admit I don't remember seeing a unit test for a memory leak. This is
>> usually verified once by a profiler. I see two approaches to write such a
>> test:
>> - generate a heap dump and inspect it
>> - if the test can get a reference on the object that leaks, the garbage
>> collection can be verified by a WeakReference combined with a ReferenceQueue
>>
>> public void testMemoryLeak() {
>> Object leakingObject = getLeakingObject();
>>
>> ReferenceQueue queue = new ReferenceQueue();
>> WeakReference reference = new WeakReference(leakingObject, queue);
>> leakingObject = null;
>>
>> // so some operations supposed to dereference the object
>>
>> System.gc();
>>
>> assertTrue(reference.isEnqueued());
>> }
>
> FWIW, I don't know how portable a test like this would be. According to the
> doc, System.gc() only "suggests" that the JVM runs the garbage collector.
>
> It's probably also not that robust since you wouldn't detect leakage of any
> ancillary objects.
>
> I think a targeted stress test is likely to be way more robust for this sort
> of thing than a unit test. Of course that's the sort of thing you don't
> really want to run once per checkin, more like once (hopefully) per release.
>
> Perhaps it would be useful to start a suite/category of longer running
> "release" tests.
>
> --Rafael
>
> ---------------------------------------------------------------------
> Apache Qpid - AMQP Messaging Implementation
> Project: http://qpid.apache.org
> Use/Interact: mailto:dev-subscribe@qpid.apache.org
>
>
--
Regards,
Rajith Attapattu
Red Hat
http://rajith.2rlabs.com/
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org
Re: [jira] Commented: (QPID-2694) Memory leak in 0-8/0-9 AMQSessions
on channel close
Posted by Rafael Schloming <ra...@redhat.com>.
Emmanuel Bourg wrote:
> Le 30/06/2010 09:20, Andrew Kennedy a écrit :
>
>> So, what do people think would be the best and simplest way to test this
>> leak is fixed? Or am I over complicating this for myself before I've had
>> enough coffee?
>
> I admit I don't remember seeing a unit test for a memory leak. This is
> usually verified once by a profiler. I see two approaches to write such
> a test:
> - generate a heap dump and inspect it
> - if the test can get a reference on the object that leaks, the garbage
> collection can be verified by a WeakReference combined with a
> ReferenceQueue
>
> public void testMemoryLeak() {
> Object leakingObject = getLeakingObject();
>
> ReferenceQueue queue = new ReferenceQueue();
> WeakReference reference = new WeakReference(leakingObject, queue);
> leakingObject = null;
>
> // so some operations supposed to dereference the object
>
> System.gc();
>
> assertTrue(reference.isEnqueued());
> }
FWIW, I don't know how portable a test like this would be. According to
the doc, System.gc() only "suggests" that the JVM runs the garbage
collector.
It's probably also not that robust since you wouldn't detect leakage of
any ancillary objects.
I think a targeted stress test is likely to be way more robust for this
sort of thing than a unit test. Of course that's the sort of thing you
don't really want to run once per checkin, more like once (hopefully)
per release.
Perhaps it would be useful to start a suite/category of longer running
"release" tests.
--Rafael
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org
Re: [jira] Commented: (QPID-2694) Memory leak in 0-8/0-9 AMQSessions
on channel close
Posted by Emmanuel Bourg <eb...@apache.org>.
Le 30/06/2010 09:20, Andrew Kennedy a écrit :
> So, what do people think would be the best and simplest way to test this
> leak is fixed? Or am I over complicating this for myself before I've had
> enough coffee?
I admit I don't remember seeing a unit test for a memory leak. This is
usually verified once by a profiler. I see two approaches to write such
a test:
- generate a heap dump and inspect it
- if the test can get a reference on the object that leaks, the garbage
collection can be verified by a WeakReference combined with a ReferenceQueue
public void testMemoryLeak() {
Object leakingObject = getLeakingObject();
ReferenceQueue queue = new ReferenceQueue();
WeakReference reference = new WeakReference(leakingObject, queue);
leakingObject = null;
// so some operations supposed to dereference the object
System.gc();
assertTrue(reference.isEnqueued());
}
Emmanuel Bourg
Re: [jira] Commented: (QPID-2694) Memory leak in 0-8/0-9 AMQSessions on channel close
Posted by Andrew Kennedy <an...@gmail.com>.
Hm.
It's not something I've ever had to do before, I think. Can anyone
help me out on how to write a unit test for a (fixed) memory leak in
Java?
I could just create 100K sessions on a connection, close them and say
that's all good, but that's just the happy path. It wouldn't really
be testing the failure case, as you can't expect to do this:
} catch (OuttOfMemoryError oome) {
fail("What does this program have in common with a sieve?");
}
If I start a child JVM with the -XX HeapDumpOnOOME option, and say
10Mb of heap, then run the number of sessions up as far as possible,
that could work. If there's no heap dump file, the test passes?
Or, I could monitor heap usage as I create the (non) leaky sessions,
and make sure the numbers balance out before and after closing a
session, and would also have to make sure GC has run, and so on. Not
exactly as below, but similar. I don't really like doing this sort of
thing in Java...
long free = Runtime.getRuntime().freeMemory();
if (free < HEAP_LIMIT_LWM) fail("Used too much memory");
So, what do people think would be the best and simplest way to test
this leak is fixed? Or am I over complicating this for myself before
I've had enough coffee?
Cheerz,
Andrew.
--
-- andrew d kennedy ? do not fold, bend, spindle, or mutilate ;
-- http://grkvlt.blogspot.com/ ? edinburgh : +44 7941 197 134 ;
On 30 Jun 2010, at 07:38, Marnie McCormack (JIRA) wrote:
>
> [ https://issues.apache.org/jira/browse/QPID-2694?
> page=com.atlassian.jira.plugin.system.issuetabpanels:comment-
> tabpanel&focusedCommentId=12883809#action_12883809 ]
>
> Marnie McCormack commented on QPID-2694:
> ----------------------------------------
>
> Reviewed files, changes look ok - applied to branch. Please add a
> test for this case, thanks.
>
>> Memory leak in 0-8/0-9 AMQSessions on channel close
>> ---------------------------------------------------
>>
>> Key: QPID-2694
>> URL: https://issues.apache.org/jira/browse/QPID-2694
>> Project: Qpid
>> Issue Type: Bug
>> Components: Java Broker
>> Affects Versions: 0.5, 0.6, 0.7
>> Reporter: Andrew Kennedy
>> Fix For: 0.5, 0.6, 0.7
>>
>> Attachments: 0001-QPID-2694-Fix-channel-close-memory-
>> leak.patch
>>
>>
>> When closing a 0-8/0-9 AMQSession, the ChannelCloseOk returned by
>> the broker is not handled in the client, leading to a memory leak
>> with AMQProtocolSession objects being left. This is particularly
>> evident with frameworks like Mule which open a session per
>> connection.
>
> --
> This message is automatically generated by JIRA.
> -
> You can reply to this email to add a comment to the issue online.
>
>
> ---------------------------------------------------------------------
> Apache Qpid - AMQP Messaging Implementation
> Project: http://qpid.apache.org
> Use/Interact: mailto:dev-subscribe@qpid.apache.org
>
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org
[jira] Commented: (QPID-2694) Memory leak in 0-8/0-9 AMQSessions on
channel close
Posted by "Marnie McCormack (JIRA)" <qp...@incubator.apache.org>.
[ https://issues.apache.org/jira/browse/QPID-2694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12883809#action_12883809 ]
Marnie McCormack commented on QPID-2694:
----------------------------------------
Reviewed files, changes look ok - applied to branch. Please add a test for this case, thanks.
> Memory leak in 0-8/0-9 AMQSessions on channel close
> ---------------------------------------------------
>
> Key: QPID-2694
> URL: https://issues.apache.org/jira/browse/QPID-2694
> Project: Qpid
> Issue Type: Bug
> Components: Java Broker
> Affects Versions: 0.5, 0.6, 0.7
> Reporter: Andrew Kennedy
> Fix For: 0.5, 0.6, 0.7
>
> Attachments: 0001-QPID-2694-Fix-channel-close-memory-leak.patch
>
>
> When closing a 0-8/0-9 AMQSession, the ChannelCloseOk returned by the broker is not handled in the client, leading to a memory leak with AMQProtocolSession objects being left. This is particularly evident with frameworks like Mule which open a session per connection.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org