You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@qpid.apache.org by Praveen M <le...@gmail.com> on 2011/10/28 21:01:53 UTC
Java Broker - creating and closing too many connections causes memory leak?
Hi,
I ran the following test a few times and I ran into a OOM exception. Can
someone please tell me if i'm doing something wrong or if this could
potentially be a bug? Thanks.
1) Create 1000 queues.
2) Create 4 consumers -> each consumer uses it's own connection
3) In a loop do the following,
for each queues 1..1000 {
create producer(create new connection) for queue |i|
produce 1 message for that queue.
close producer (close connection)
}
4) Wait till all messages are consumed by the 4 consumers and then close all
4 consumers.
5) Re-Run the test without a broker restart
Each time the test ran, it created and closed 1004 connections (with at most
5 connections living in parallel...we close the earlier queue's producer's
connection before creating a producer for the next queue).
Also, I checked for any stray threads on the client side due to any unclosed
connection, and it looks like everything was closed fine (I didn't see any
stray threads).
On the broker side, I saw that the memory usage creeps up each time i re-run
the test, and eventually hits a OOM. (Running the above test about 5 times
causes a OOM )
I tried profiling the broker to see what's holding on to all that memory and
saw a certain class's(org.apache.qpid.transport.Method[]) instance count
grow up each time my test was run and it was holding the maximum memory in
the broker.
The number of instances of org.apache.qpid.transport.Method[] were trending
as follows:
After first test - 2008
After second test - 4016
... and so on.
There seems to be a direct relation between the number of those objects and
the connections created and closed. ~ Why are these objects not freed on
closing the connection??
I'm using the Java Broker out of trunk and using it with a DerbyMessageStore
for the test.
Or is there something fundamentally wrong by the way I run the test, I guess
opening and closing so many connections isn't a great idea (maybe I should
be dealing with sessions). But still, I see that this can be a potential
problem in a really long running broker system(where we cannot afford to
restart the broker periodically) which can have clients joining/leaving at
regular intervals.
Thank you for all the effort answering our questions,
--
-Praveen
Re: Java Broker - creating and closing too many connections causes
memory leak?
Posted by Danushka Menikkumbura <da...@gmail.com>.
Hi Praveen,
AFAIK what you see is a known issue in the Java broker. I ran into the same
issue when I used the Java broker in a messaging product based on the Java
broker. Our team has submitted a couple of patches too but then the code
base was in the middle of some major refactoring. I think Robbie is the
right person to elaborate on this.
Thanks,
Danushka
On Sat, Oct 29, 2011 at 12:31 AM, Praveen M <le...@gmail.com> wrote:
> Hi,
>
> I ran the following test a few times and I ran into a OOM exception. Can
> someone please tell me if i'm doing something wrong or if this could
> potentially be a bug? Thanks.
>
> 1) Create 1000 queues.
> 2) Create 4 consumers -> each consumer uses it's own connection
> 3) In a loop do the following,
>
> for each queues 1..1000 {
> create producer(create new connection) for queue |i|
> produce 1 message for that queue.
> close producer (close connection)
> }
>
> 4) Wait till all messages are consumed by the 4 consumers and then close
> all
> 4 consumers.
> 5) Re-Run the test without a broker restart
>
> Each time the test ran, it created and closed 1004 connections (with at
> most
> 5 connections living in parallel...we close the earlier queue's producer's
> connection before creating a producer for the next queue).
>
> Also, I checked for any stray threads on the client side due to any
> unclosed
> connection, and it looks like everything was closed fine (I didn't see any
> stray threads).
>
> On the broker side, I saw that the memory usage creeps up each time i
> re-run
> the test, and eventually hits a OOM. (Running the above test about 5 times
> causes a OOM )
>
> I tried profiling the broker to see what's holding on to all that memory
> and
> saw a certain class's(org.apache.qpid.transport.Method[]) instance count
> grow up each time my test was run and it was holding the maximum memory in
> the broker.
>
> The number of instances of org.apache.qpid.transport.Method[] were trending
> as follows:
>
> After first test - 2008
> After second test - 4016
> ... and so on.
>
> There seems to be a direct relation between the number of those objects and
> the connections created and closed. ~ Why are these objects not freed on
> closing the connection??
> I'm using the Java Broker out of trunk and using it with a
> DerbyMessageStore
> for the test.
>
> Or is there something fundamentally wrong by the way I run the test, I
> guess
> opening and closing so many connections isn't a great idea (maybe I should
> be dealing with sessions). But still, I see that this can be a potential
> problem in a really long running broker system(where we cannot afford to
> restart the broker periodically) which can have clients joining/leaving at
> regular intervals.
>
>
> Thank you for all the effort answering our questions,
> --
> -Praveen
>
Re: Java Broker - creating and closing too many connections causes memory leak?
Posted by Praveen <le...@gmail.com>.
Hi,
I had just built from the svn source repo for the latest trunk once Robbie had pushed in the fix.
Praveen
Sent from my iPhone
On Nov 11, 2011, at 6:04 AM, Robbie Gemmell <ro...@gmail.com> wrote:
> I believe that is what Praveen actually did as well, but you can also
> find a nightly release build of the current trunk at the following
> URL:
> https://builds.apache.org/view/M-R/view/Qpid/job/Qpid-Java-Release/lastSuccessfulBuild/artifact/trunk/qpid/java/broker/release/
>
> Robbie
>
> On 11 November 2011 11:08, dizhihui@gmail.com <di...@gmail.com> wrote:
>> I have check out source code from SVN and bulid a new package.
>>
>> -Dee
>>
>> --
>> View this message in context: http://apache-qpid-users.2158936.n2.nabble.com/Java-Broker-creating-and-closing-too-many-connections-causes-memory-leak-tp6941393p6985050.html
>> Sent from the Apache Qpid users mailing list archive at Nabble.com.
>>
>> ---------------------------------------------------------------------
>> Apache Qpid - AMQP Messaging Implementation
>> Project: http://qpid.apache.org
>> Use/Interact: mailto:users-subscribe@qpid.apache.org
>>
>>
>
> ---------------------------------------------------------------------
> Apache Qpid - AMQP Messaging Implementation
> Project: http://qpid.apache.org
> Use/Interact: mailto:users-subscribe@qpid.apache.org
>
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: Java Broker - creating and closing too many connections causes
memory leak?
Posted by Robbie Gemmell <ro...@gmail.com>.
I believe that is what Praveen actually did as well, but you can also
find a nightly release build of the current trunk at the following
URL:
https://builds.apache.org/view/M-R/view/Qpid/job/Qpid-Java-Release/lastSuccessfulBuild/artifact/trunk/qpid/java/broker/release/
Robbie
On 11 November 2011 11:08, dizhihui@gmail.com <di...@gmail.com> wrote:
> I have check out source code from SVN and bulid a new package.
>
> -Dee
>
> --
> View this message in context: http://apache-qpid-users.2158936.n2.nabble.com/Java-Broker-creating-and-closing-too-many-connections-causes-memory-leak-tp6941393p6985050.html
> Sent from the Apache Qpid users mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> Apache Qpid - AMQP Messaging Implementation
> Project: http://qpid.apache.org
> Use/Interact: mailto:users-subscribe@qpid.apache.org
>
>
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: Java Broker - creating and closing too many connections causes
memory leak?
Posted by "dizhihui@gmail.com" <di...@gmail.com>.
I have check out source code from SVN and bulid a new package.
-Dee
--
View this message in context: http://apache-qpid-users.2158936.n2.nabble.com/Java-Broker-creating-and-closing-too-many-connections-causes-memory-leak-tp6941393p6985050.html
Sent from the Apache Qpid users mailing list archive at Nabble.com.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: Java Broker - creating and closing too many connections causes
memory leak?
Posted by "dizhihui@gmail.com" <di...@gmail.com>.
praveen,
I meet the same problem.
Where did you get the updated java broker package?
thanks.
-dee
--
View this message in context: http://apache-qpid-users.2158936.n2.nabble.com/Java-Broker-creating-and-closing-too-many-connections-causes-memory-leak-tp6941393p6984755.html
Sent from the Apache Qpid users mailing list archive at Nabble.com.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: Java Broker - creating and closing too many connections causes
memory leak?
Posted by Praveen M <le...@gmail.com>.
Yo, tested this too. Works really well, the memory leak has disappeared. :)
On Sun, Oct 30, 2011 at 12:03 PM, Praveen M <le...@gmail.com> wrote:
> Great . Thanks a ton. I'll check on this too tomorrow and keep you posted.
>
>
> On Sun, Oct 30, 2011 at 11:55 AM, Robbie Gemmell <robbie.gemmell@gmail.com
> > wrote:
>
>> This was indeed a good and proper leak, introduced by changes made
>> since 0.12. I have made an update to fix the leak, if you rerun your
>> test now it should work better.
>>
>> Robbie
>>
>> On 29 October 2011 03:46, Praveen M <le...@gmail.com> wrote:
>> > Thanks a lot Robbie for the prompt reply and looking into this.
>> >
>> > Praveen
>> >
>> > On Fri, Oct 28, 2011 at 7:41 PM, Robbie Gemmell <
>> robbie.gemmell@gmail.com>wrote:
>> >
>> >> This isn't a current known issue since work was previously undertaken
>> >> to fix similar problems, so this sounds like a new bug or possibly
>> >> something that has been reintroduced. The Method[] objects are part of
>> >> the underlying transport layer backing each JMS Session, so they will
>> >> directly correlate to the number of Connections and Sessions you are
>> >> creating, although there appear to have been exactly twice what you
>> >> indicated you are using per run which is stange. They should be
>> >> getting cleared up, but something is obviously holding onto them.
>> >>
>> >> Opening and closing a connection and session per message is of course
>> >> rather inefficient, but it should of course still work. I'll add this
>> >> to my list of things to look at.
>> >>
>> >> Robbie
>> >>
>> >> On 28 October 2011 20:01, Praveen M <le...@gmail.com> wrote:
>> >> > Hi,
>> >> >
>> >> > I ran the following test a few times and I ran into a OOM
>> exception. Can
>> >> > someone please tell me if i'm doing something wrong or if this could
>> >> > potentially be a bug? Thanks.
>> >> >
>> >> > 1) Create 1000 queues.
>> >> > 2) Create 4 consumers -> each consumer uses it's own connection
>> >> > 3) In a loop do the following,
>> >> >
>> >> > for each queues 1..1000 {
>> >> > create producer(create new connection) for queue |i|
>> >> > produce 1 message for that queue.
>> >> > close producer (close connection)
>> >> > }
>> >> >
>> >> > 4) Wait till all messages are consumed by the 4 consumers and then
>> close
>> >> all
>> >> > 4 consumers.
>> >> > 5) Re-Run the test without a broker restart
>> >> >
>> >> > Each time the test ran, it created and closed 1004 connections (with
>> at
>> >> most
>> >> > 5 connections living in parallel...we close the earlier queue's
>> >> producer's
>> >> > connection before creating a producer for the next queue).
>> >> >
>> >> > Also, I checked for any stray threads on the client side due to any
>> >> unclosed
>> >> > connection, and it looks like everything was closed fine (I didn't
>> see
>> >> any
>> >> > stray threads).
>> >> >
>> >> > On the broker side, I saw that the memory usage creeps up each time i
>> >> re-run
>> >> > the test, and eventually hits a OOM. (Running the above test about 5
>> >> times
>> >> > causes a OOM )
>> >> >
>> >> > I tried profiling the broker to see what's holding on to all that
>> memory
>> >> and
>> >> > saw a certain class's(org.apache.qpid.transport.Method[]) instance
>> count
>> >> > grow up each time my test was run and it was holding the maximum
>> memory
>> >> in
>> >> > the broker.
>> >> >
>> >> > The number of instances of org.apache.qpid.transport.Method[] were
>> >> trending
>> >> > as follows:
>> >> >
>> >> > After first test - 2008
>> >> > After second test - 4016
>> >> > ... and so on.
>> >> >
>> >> > There seems to be a direct relation between the number of those
>> objects
>> >> and
>> >> > the connections created and closed. ~ Why are these objects not
>> freed on
>> >> > closing the connection??
>> >> > I'm using the Java Broker out of trunk and using it with a
>> >> DerbyMessageStore
>> >> > for the test.
>> >> >
>> >> > Or is there something fundamentally wrong by the way I run the test,
>> I
>> >> guess
>> >> > opening and closing so many connections isn't a great idea (maybe I
>> >> should
>> >> > be dealing with sessions). But still, I see that this can be a
>> potential
>> >> > problem in a really long running broker system(where we cannot
>> afford to
>> >> > restart the broker periodically) which can have clients
>> joining/leaving
>> >> at
>> >> > regular intervals.
>> >> >
>> >> >
>> >> > Thank you for all the effort answering our questions,
>> >> > --
>> >> > -Praveen
>> >> >
>> >>
>> >> ---------------------------------------------------------------------
>> >> Apache Qpid - AMQP Messaging Implementation
>> >> Project: http://qpid.apache.org
>> >> Use/Interact: mailto:users-subscribe@qpid.apache.org
>> >>
>> >>
>> >
>> >
>> > --
>> > -Praveen
>> >
>>
>> ---------------------------------------------------------------------
>> Apache Qpid - AMQP Messaging Implementation
>> Project: http://qpid.apache.org
>> Use/Interact: mailto:users-subscribe@qpid.apache.org
>>
>>
>
>
> --
> -Praveen
>
--
-Praveen
Re: Java Broker - creating and closing too many connections causes
memory leak?
Posted by Praveen M <le...@gmail.com>.
Great . Thanks a ton. I'll check on this too tomorrow and keep you posted.
On Sun, Oct 30, 2011 at 11:55 AM, Robbie Gemmell
<ro...@gmail.com>wrote:
> This was indeed a good and proper leak, introduced by changes made
> since 0.12. I have made an update to fix the leak, if you rerun your
> test now it should work better.
>
> Robbie
>
> On 29 October 2011 03:46, Praveen M <le...@gmail.com> wrote:
> > Thanks a lot Robbie for the prompt reply and looking into this.
> >
> > Praveen
> >
> > On Fri, Oct 28, 2011 at 7:41 PM, Robbie Gemmell <
> robbie.gemmell@gmail.com>wrote:
> >
> >> This isn't a current known issue since work was previously undertaken
> >> to fix similar problems, so this sounds like a new bug or possibly
> >> something that has been reintroduced. The Method[] objects are part of
> >> the underlying transport layer backing each JMS Session, so they will
> >> directly correlate to the number of Connections and Sessions you are
> >> creating, although there appear to have been exactly twice what you
> >> indicated you are using per run which is stange. They should be
> >> getting cleared up, but something is obviously holding onto them.
> >>
> >> Opening and closing a connection and session per message is of course
> >> rather inefficient, but it should of course still work. I'll add this
> >> to my list of things to look at.
> >>
> >> Robbie
> >>
> >> On 28 October 2011 20:01, Praveen M <le...@gmail.com> wrote:
> >> > Hi,
> >> >
> >> > I ran the following test a few times and I ran into a OOM exception.
> Can
> >> > someone please tell me if i'm doing something wrong or if this could
> >> > potentially be a bug? Thanks.
> >> >
> >> > 1) Create 1000 queues.
> >> > 2) Create 4 consumers -> each consumer uses it's own connection
> >> > 3) In a loop do the following,
> >> >
> >> > for each queues 1..1000 {
> >> > create producer(create new connection) for queue |i|
> >> > produce 1 message for that queue.
> >> > close producer (close connection)
> >> > }
> >> >
> >> > 4) Wait till all messages are consumed by the 4 consumers and then
> close
> >> all
> >> > 4 consumers.
> >> > 5) Re-Run the test without a broker restart
> >> >
> >> > Each time the test ran, it created and closed 1004 connections (with
> at
> >> most
> >> > 5 connections living in parallel...we close the earlier queue's
> >> producer's
> >> > connection before creating a producer for the next queue).
> >> >
> >> > Also, I checked for any stray threads on the client side due to any
> >> unclosed
> >> > connection, and it looks like everything was closed fine (I didn't see
> >> any
> >> > stray threads).
> >> >
> >> > On the broker side, I saw that the memory usage creeps up each time i
> >> re-run
> >> > the test, and eventually hits a OOM. (Running the above test about 5
> >> times
> >> > causes a OOM )
> >> >
> >> > I tried profiling the broker to see what's holding on to all that
> memory
> >> and
> >> > saw a certain class's(org.apache.qpid.transport.Method[]) instance
> count
> >> > grow up each time my test was run and it was holding the maximum
> memory
> >> in
> >> > the broker.
> >> >
> >> > The number of instances of org.apache.qpid.transport.Method[] were
> >> trending
> >> > as follows:
> >> >
> >> > After first test - 2008
> >> > After second test - 4016
> >> > ... and so on.
> >> >
> >> > There seems to be a direct relation between the number of those
> objects
> >> and
> >> > the connections created and closed. ~ Why are these objects not freed
> on
> >> > closing the connection??
> >> > I'm using the Java Broker out of trunk and using it with a
> >> DerbyMessageStore
> >> > for the test.
> >> >
> >> > Or is there something fundamentally wrong by the way I run the test, I
> >> guess
> >> > opening and closing so many connections isn't a great idea (maybe I
> >> should
> >> > be dealing with sessions). But still, I see that this can be a
> potential
> >> > problem in a really long running broker system(where we cannot afford
> to
> >> > restart the broker periodically) which can have clients
> joining/leaving
> >> at
> >> > regular intervals.
> >> >
> >> >
> >> > Thank you for all the effort answering our questions,
> >> > --
> >> > -Praveen
> >> >
> >>
> >> ---------------------------------------------------------------------
> >> Apache Qpid - AMQP Messaging Implementation
> >> Project: http://qpid.apache.org
> >> Use/Interact: mailto:users-subscribe@qpid.apache.org
> >>
> >>
> >
> >
> > --
> > -Praveen
> >
>
> ---------------------------------------------------------------------
> Apache Qpid - AMQP Messaging Implementation
> Project: http://qpid.apache.org
> Use/Interact: mailto:users-subscribe@qpid.apache.org
>
>
--
-Praveen
Re: Java Broker - creating and closing too many connections causes
memory leak?
Posted by Robbie Gemmell <ro...@gmail.com>.
This was indeed a good and proper leak, introduced by changes made
since 0.12. I have made an update to fix the leak, if you rerun your
test now it should work better.
Robbie
On 29 October 2011 03:46, Praveen M <le...@gmail.com> wrote:
> Thanks a lot Robbie for the prompt reply and looking into this.
>
> Praveen
>
> On Fri, Oct 28, 2011 at 7:41 PM, Robbie Gemmell <ro...@gmail.com>wrote:
>
>> This isn't a current known issue since work was previously undertaken
>> to fix similar problems, so this sounds like a new bug or possibly
>> something that has been reintroduced. The Method[] objects are part of
>> the underlying transport layer backing each JMS Session, so they will
>> directly correlate to the number of Connections and Sessions you are
>> creating, although there appear to have been exactly twice what you
>> indicated you are using per run which is stange. They should be
>> getting cleared up, but something is obviously holding onto them.
>>
>> Opening and closing a connection and session per message is of course
>> rather inefficient, but it should of course still work. I'll add this
>> to my list of things to look at.
>>
>> Robbie
>>
>> On 28 October 2011 20:01, Praveen M <le...@gmail.com> wrote:
>> > Hi,
>> >
>> > I ran the following test a few times and I ran into a OOM exception. Can
>> > someone please tell me if i'm doing something wrong or if this could
>> > potentially be a bug? Thanks.
>> >
>> > 1) Create 1000 queues.
>> > 2) Create 4 consumers -> each consumer uses it's own connection
>> > 3) In a loop do the following,
>> >
>> > for each queues 1..1000 {
>> > create producer(create new connection) for queue |i|
>> > produce 1 message for that queue.
>> > close producer (close connection)
>> > }
>> >
>> > 4) Wait till all messages are consumed by the 4 consumers and then close
>> all
>> > 4 consumers.
>> > 5) Re-Run the test without a broker restart
>> >
>> > Each time the test ran, it created and closed 1004 connections (with at
>> most
>> > 5 connections living in parallel...we close the earlier queue's
>> producer's
>> > connection before creating a producer for the next queue).
>> >
>> > Also, I checked for any stray threads on the client side due to any
>> unclosed
>> > connection, and it looks like everything was closed fine (I didn't see
>> any
>> > stray threads).
>> >
>> > On the broker side, I saw that the memory usage creeps up each time i
>> re-run
>> > the test, and eventually hits a OOM. (Running the above test about 5
>> times
>> > causes a OOM )
>> >
>> > I tried profiling the broker to see what's holding on to all that memory
>> and
>> > saw a certain class's(org.apache.qpid.transport.Method[]) instance count
>> > grow up each time my test was run and it was holding the maximum memory
>> in
>> > the broker.
>> >
>> > The number of instances of org.apache.qpid.transport.Method[] were
>> trending
>> > as follows:
>> >
>> > After first test - 2008
>> > After second test - 4016
>> > ... and so on.
>> >
>> > There seems to be a direct relation between the number of those objects
>> and
>> > the connections created and closed. ~ Why are these objects not freed on
>> > closing the connection??
>> > I'm using the Java Broker out of trunk and using it with a
>> DerbyMessageStore
>> > for the test.
>> >
>> > Or is there something fundamentally wrong by the way I run the test, I
>> guess
>> > opening and closing so many connections isn't a great idea (maybe I
>> should
>> > be dealing with sessions). But still, I see that this can be a potential
>> > problem in a really long running broker system(where we cannot afford to
>> > restart the broker periodically) which can have clients joining/leaving
>> at
>> > regular intervals.
>> >
>> >
>> > Thank you for all the effort answering our questions,
>> > --
>> > -Praveen
>> >
>>
>> ---------------------------------------------------------------------
>> Apache Qpid - AMQP Messaging Implementation
>> Project: http://qpid.apache.org
>> Use/Interact: mailto:users-subscribe@qpid.apache.org
>>
>>
>
>
> --
> -Praveen
>
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: Java Broker - creating and closing too many connections causes
memory leak?
Posted by Praveen M <le...@gmail.com>.
Thanks a lot Robbie for the prompt reply and looking into this.
Praveen
On Fri, Oct 28, 2011 at 7:41 PM, Robbie Gemmell <ro...@gmail.com>wrote:
> This isn't a current known issue since work was previously undertaken
> to fix similar problems, so this sounds like a new bug or possibly
> something that has been reintroduced. The Method[] objects are part of
> the underlying transport layer backing each JMS Session, so they will
> directly correlate to the number of Connections and Sessions you are
> creating, although there appear to have been exactly twice what you
> indicated you are using per run which is stange. They should be
> getting cleared up, but something is obviously holding onto them.
>
> Opening and closing a connection and session per message is of course
> rather inefficient, but it should of course still work. I'll add this
> to my list of things to look at.
>
> Robbie
>
> On 28 October 2011 20:01, Praveen M <le...@gmail.com> wrote:
> > Hi,
> >
> > I ran the following test a few times and I ran into a OOM exception. Can
> > someone please tell me if i'm doing something wrong or if this could
> > potentially be a bug? Thanks.
> >
> > 1) Create 1000 queues.
> > 2) Create 4 consumers -> each consumer uses it's own connection
> > 3) In a loop do the following,
> >
> > for each queues 1..1000 {
> > create producer(create new connection) for queue |i|
> > produce 1 message for that queue.
> > close producer (close connection)
> > }
> >
> > 4) Wait till all messages are consumed by the 4 consumers and then close
> all
> > 4 consumers.
> > 5) Re-Run the test without a broker restart
> >
> > Each time the test ran, it created and closed 1004 connections (with at
> most
> > 5 connections living in parallel...we close the earlier queue's
> producer's
> > connection before creating a producer for the next queue).
> >
> > Also, I checked for any stray threads on the client side due to any
> unclosed
> > connection, and it looks like everything was closed fine (I didn't see
> any
> > stray threads).
> >
> > On the broker side, I saw that the memory usage creeps up each time i
> re-run
> > the test, and eventually hits a OOM. (Running the above test about 5
> times
> > causes a OOM )
> >
> > I tried profiling the broker to see what's holding on to all that memory
> and
> > saw a certain class's(org.apache.qpid.transport.Method[]) instance count
> > grow up each time my test was run and it was holding the maximum memory
> in
> > the broker.
> >
> > The number of instances of org.apache.qpid.transport.Method[] were
> trending
> > as follows:
> >
> > After first test - 2008
> > After second test - 4016
> > ... and so on.
> >
> > There seems to be a direct relation between the number of those objects
> and
> > the connections created and closed. ~ Why are these objects not freed on
> > closing the connection??
> > I'm using the Java Broker out of trunk and using it with a
> DerbyMessageStore
> > for the test.
> >
> > Or is there something fundamentally wrong by the way I run the test, I
> guess
> > opening and closing so many connections isn't a great idea (maybe I
> should
> > be dealing with sessions). But still, I see that this can be a potential
> > problem in a really long running broker system(where we cannot afford to
> > restart the broker periodically) which can have clients joining/leaving
> at
> > regular intervals.
> >
> >
> > Thank you for all the effort answering our questions,
> > --
> > -Praveen
> >
>
> ---------------------------------------------------------------------
> Apache Qpid - AMQP Messaging Implementation
> Project: http://qpid.apache.org
> Use/Interact: mailto:users-subscribe@qpid.apache.org
>
>
--
-Praveen
Re: Java Broker - creating and closing too many connections causes
memory leak?
Posted by Robbie Gemmell <ro...@gmail.com>.
This isn't a current known issue since work was previously undertaken
to fix similar problems, so this sounds like a new bug or possibly
something that has been reintroduced. The Method[] objects are part of
the underlying transport layer backing each JMS Session, so they will
directly correlate to the number of Connections and Sessions you are
creating, although there appear to have been exactly twice what you
indicated you are using per run which is stange. They should be
getting cleared up, but something is obviously holding onto them.
Opening and closing a connection and session per message is of course
rather inefficient, but it should of course still work. I'll add this
to my list of things to look at.
Robbie
On 28 October 2011 20:01, Praveen M <le...@gmail.com> wrote:
> Hi,
>
> I ran the following test a few times and I ran into a OOM exception. Can
> someone please tell me if i'm doing something wrong or if this could
> potentially be a bug? Thanks.
>
> 1) Create 1000 queues.
> 2) Create 4 consumers -> each consumer uses it's own connection
> 3) In a loop do the following,
>
> for each queues 1..1000 {
> create producer(create new connection) for queue |i|
> produce 1 message for that queue.
> close producer (close connection)
> }
>
> 4) Wait till all messages are consumed by the 4 consumers and then close all
> 4 consumers.
> 5) Re-Run the test without a broker restart
>
> Each time the test ran, it created and closed 1004 connections (with at most
> 5 connections living in parallel...we close the earlier queue's producer's
> connection before creating a producer for the next queue).
>
> Also, I checked for any stray threads on the client side due to any unclosed
> connection, and it looks like everything was closed fine (I didn't see any
> stray threads).
>
> On the broker side, I saw that the memory usage creeps up each time i re-run
> the test, and eventually hits a OOM. (Running the above test about 5 times
> causes a OOM )
>
> I tried profiling the broker to see what's holding on to all that memory and
> saw a certain class's(org.apache.qpid.transport.Method[]) instance count
> grow up each time my test was run and it was holding the maximum memory in
> the broker.
>
> The number of instances of org.apache.qpid.transport.Method[] were trending
> as follows:
>
> After first test - 2008
> After second test - 4016
> ... and so on.
>
> There seems to be a direct relation between the number of those objects and
> the connections created and closed. ~ Why are these objects not freed on
> closing the connection??
> I'm using the Java Broker out of trunk and using it with a DerbyMessageStore
> for the test.
>
> Or is there something fundamentally wrong by the way I run the test, I guess
> opening and closing so many connections isn't a great idea (maybe I should
> be dealing with sessions). But still, I see that this can be a potential
> problem in a really long running broker system(where we cannot afford to
> restart the broker periodically) which can have clients joining/leaving at
> regular intervals.
>
>
> Thank you for all the effort answering our questions,
> --
> -Praveen
>
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org