You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by Jerry Cwiklik <cw...@us.ibm.com> on 2012/12/10 15:44:00 UTC
Broker Leak
I am running AMQ broker v.5.6 on linux with 10G Xmx, 5G limit memoryUsage,
producerFlowControl=false, no persistence, vmCursor and AUTO_ACKNOWLEDGE.
There are a hundreds of producers and consumers, a few queues and a couple
of hundred of temp queues. While watching broker memory in jConsole I notice
a steady raise in OldGen. Forcing GC doesnt change anything. The broker
eventually OOMs after a few days. The HeapAnalyzer shows 96% of heap in
VmPendingMessageCursor. There are many messages destined for temp queues in
this cursor.
I created a simple test program where a Producer keeps sending messages to a
Consumer which immediately sends a reply back to the Producer's temp queue.
I kill the Producer with -9, force GC on the broker and dump the heap.
Looking at the heap, I can see that the VMPendingCursor holds messages for
the temp queue which no longer exist! It looks to me that the broker never
cleans up messages destined for temp queues which no longer exist leading to
a memory leak. Attached is a snapshot from the heapAnalyzer showing a
message in VMPendingMessageCursor.
Is there a way to configure the broker to remove such messages? We currently
dont expire messages (TTL=0). The broker knows when a temp queue is deleted
so it should try to clean up messages associated with it.
<http://activemq.2283324.n4.nabble.com/file/n4660437/heap-dump-leak.png>
Regards, Jerry C
--
View this message in context: http://activemq.2283324.n4.nabble.com/Broker-Leak-tp4660437.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Broker Leak
Posted by Christian Posta <ch...@gmail.com>.
Jerry,
Thanks again for demonstrating this.
It's been fixed in the trunk.
https://issues.apache.org/jira/browse/AMQ-4222
Thanks,
Christian
On Thu, Dec 13, 2012 at 11:17 AM, Christian Posta <christian.posta@gmail.com
> wrote:
> Awesome, thanks for coming back and explaining what happened there. I was
> super curious about this one.
> I still think there's room for a unit test to document this and show that
> the reference in the ProducerBrokerExchange still holds onto the "region"
> destinations. This reference could be cleared at the end of a send since
> it's never used again and would allow the broker to reclaim the destination
> completely.
>
> I'll write the test for that and change that reference!!
> Tracked here:
>
> https://issues.apache.org/jira/browse/AMQ-4222
>
>
>
> On Thu, Dec 13, 2012 at 10:17 AM, Jerry Cwiklik <cw...@us.ibm.com>wrote:
>
>> Christian, after much pain and suffering I finally figured out what is
>> going
>> on. Our system is quite complicated and involves many producers that send
>> large messages (600K-1.5M) to a relatively few multi-threaded consumers
>> (services) which run "forever". The producers are transient and can be
>> killed by our custom job scheduler at any time via kill -9 to make room
>> for
>> other producers. We run the broker with 10G heap.
>>
>> The consumer is coded to group and cache Sessions with a Connection which
>> has an inactivity timer associated with it. Every time a message is sent,
>> the timer is restarted. If the timer pops (default 30minutes), the
>> Sessions,
>> MessageProducers and a Connection are closed due to inactivity.
>>
>> This worked perfectly fine until about 4 weeks ago when we started
>> experiencing broker OOM problem. While the broker was running we could
>> see a
>> steady (fast) rise in the heap usage in a jConsole. After a couple of days
>> the broker's jvm would OOM.
>>
>> The problem started happening when we introduced pingers for the
>> Consumers.
>> Every minute a pinger sends a message to a Consumer to make sure its
>> alive.
>> The Consumer replies to the pinger request and restarts inactivity timer.
>> It
>> took me awhile to see the bug in our application, but eventually I
>> determined that our timer behaves incorrectly as it is associated with a
>> Connection not individual Sessions. The Sessions go stale due to producer
>> getting killed, and any messages in the broker referenced by
>> ProducerExchange object are retained indefinitely causing a leak in the
>> broker. As you explained it to me, the broker uses lazy approach to
>> cleanup.
>> Meaning it cleans up on a new message from the Producer. In our case, the
>> Producer never sends anything and thus no cleanup is ever done.
>>
>> The fix for this is to create a timestamp for every Session when it was
>> last
>> used to message to the broker. At fixed intervals a Session Reaper thread
>> wakes up and checks the timestamp of every Session to determine if it has
>> been inactive for a max allowed time and if so, to close it.
>>
>> So the problem was caused by an application bug and the fact that the
>> broker
>> takes a lazy approach to cleanup. As a side note, under the described
>> scenario, I've noticed that the broker memory usage (shown in jConsole)
>> indicated 0 even though there were ton of messages in the heap with valid
>> references (held by ProducerExchange).
>>
>> Thanks Christian for your help
>>
>> -Jerry C
>>
>>
>>
>>
>>
>>
>> --
>> View this message in context:
>> http://activemq.2283324.n4.nabble.com/Broker-Leak-tp4660437p4660618.html
>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>
>
>
>
> --
> *Christian Posta*
> http://www.christianposta.com/blog
> twitter: @christianposta
>
>
--
*Christian Posta*
http://www.christianposta.com/blog
twitter: @christianposta
Re: Broker Leak
Posted by Christian Posta <ch...@gmail.com>.
Awesome, thanks for coming back and explaining what happened there. I was
super curious about this one.
I still think there's room for a unit test to document this and show that
the reference in the ProducerBrokerExchange still holds onto the "region"
destinations. This reference could be cleared at the end of a send since
it's never used again and would allow the broker to reclaim the destination
completely.
I'll write the test for that and change that reference!!
Tracked here:
https://issues.apache.org/jira/browse/AMQ-4222
On Thu, Dec 13, 2012 at 10:17 AM, Jerry Cwiklik <cw...@us.ibm.com> wrote:
> Christian, after much pain and suffering I finally figured out what is
> going
> on. Our system is quite complicated and involves many producers that send
> large messages (600K-1.5M) to a relatively few multi-threaded consumers
> (services) which run "forever". The producers are transient and can be
> killed by our custom job scheduler at any time via kill -9 to make room for
> other producers. We run the broker with 10G heap.
>
> The consumer is coded to group and cache Sessions with a Connection which
> has an inactivity timer associated with it. Every time a message is sent,
> the timer is restarted. If the timer pops (default 30minutes), the
> Sessions,
> MessageProducers and a Connection are closed due to inactivity.
>
> This worked perfectly fine until about 4 weeks ago when we started
> experiencing broker OOM problem. While the broker was running we could see
> a
> steady (fast) rise in the heap usage in a jConsole. After a couple of days
> the broker's jvm would OOM.
>
> The problem started happening when we introduced pingers for the Consumers.
> Every minute a pinger sends a message to a Consumer to make sure its alive.
> The Consumer replies to the pinger request and restarts inactivity timer.
> It
> took me awhile to see the bug in our application, but eventually I
> determined that our timer behaves incorrectly as it is associated with a
> Connection not individual Sessions. The Sessions go stale due to producer
> getting killed, and any messages in the broker referenced by
> ProducerExchange object are retained indefinitely causing a leak in the
> broker. As you explained it to me, the broker uses lazy approach to
> cleanup.
> Meaning it cleans up on a new message from the Producer. In our case, the
> Producer never sends anything and thus no cleanup is ever done.
>
> The fix for this is to create a timestamp for every Session when it was
> last
> used to message to the broker. At fixed intervals a Session Reaper thread
> wakes up and checks the timestamp of every Session to determine if it has
> been inactive for a max allowed time and if so, to close it.
>
> So the problem was caused by an application bug and the fact that the
> broker
> takes a lazy approach to cleanup. As a side note, under the described
> scenario, I've noticed that the broker memory usage (shown in jConsole)
> indicated 0 even though there were ton of messages in the heap with valid
> references (held by ProducerExchange).
>
> Thanks Christian for your help
>
> -Jerry C
>
>
>
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/Broker-Leak-tp4660437p4660618.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
--
*Christian Posta*
http://www.christianposta.com/blog
twitter: @christianposta
Re: Broker Leak
Posted by Jerry Cwiklik <cw...@us.ibm.com>.
Christian, after much pain and suffering I finally figured out what is going
on. Our system is quite complicated and involves many producers that send
large messages (600K-1.5M) to a relatively few multi-threaded consumers
(services) which run "forever". The producers are transient and can be
killed by our custom job scheduler at any time via kill -9 to make room for
other producers. We run the broker with 10G heap.
The consumer is coded to group and cache Sessions with a Connection which
has an inactivity timer associated with it. Every time a message is sent,
the timer is restarted. If the timer pops (default 30minutes), the Sessions,
MessageProducers and a Connection are closed due to inactivity.
This worked perfectly fine until about 4 weeks ago when we started
experiencing broker OOM problem. While the broker was running we could see a
steady (fast) rise in the heap usage in a jConsole. After a couple of days
the broker's jvm would OOM.
The problem started happening when we introduced pingers for the Consumers.
Every minute a pinger sends a message to a Consumer to make sure its alive.
The Consumer replies to the pinger request and restarts inactivity timer. It
took me awhile to see the bug in our application, but eventually I
determined that our timer behaves incorrectly as it is associated with a
Connection not individual Sessions. The Sessions go stale due to producer
getting killed, and any messages in the broker referenced by
ProducerExchange object are retained indefinitely causing a leak in the
broker. As you explained it to me, the broker uses lazy approach to cleanup.
Meaning it cleans up on a new message from the Producer. In our case, the
Producer never sends anything and thus no cleanup is ever done.
The fix for this is to create a timestamp for every Session when it was last
used to message to the broker. At fixed intervals a Session Reaper thread
wakes up and checks the timestamp of every Session to determine if it has
been inactive for a max allowed time and if so, to close it.
So the problem was caused by an application bug and the fact that the broker
takes a lazy approach to cleanup. As a side note, under the described
scenario, I've noticed that the broker memory usage (shown in jConsole)
indicated 0 even though there were ton of messages in the heap with valid
references (held by ProducerExchange).
Thanks Christian for your help
-Jerry C
--
View this message in context: http://activemq.2283324.n4.nabble.com/Broker-Leak-tp4660437p4660618.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Broker Leak
Posted by Christian Posta <ch...@gmail.com>.
Bad choice of words.. not "destined" but could run out of memory if
consumers cannot keep up with producers
On Wed, Dec 12, 2012 at 5:51 PM, Christian Posta
<ch...@gmail.com>wrote:
> So i guess in general i would think that a configuration with no PFC, vm
> cursors, and no persistence is destined to run out of memory.
>
> I wonder if what's happening in your case is these temporary queues are
> getting filled with messages that aren't being consumed and then when the
> reference hangs around in the Producer exchange, it's making a dent in the
> heap space. If this happens enough times it seems like it can OOM. I'll try
> to reproduce that on my side.
>
>
> On Wed, Dec 12, 2012 at 7:55 AM, Jerry Cwiklik <cw...@us.ibm.com> wrote:
>
>> Christian, thanks. The most recent experiment was run with a much smaller
>> broker, Xmx=360M. I am trying to reproduce the problem I am seeing in a
>> larger system which is not easy to debug. I can't share the large heap
>> dump
>> as it may contain proprietary data. But thanks for the offer to help
>> examine
>> it.
>>
>> I know that the large heap dump looks exactly as the screenshot I attached
>> to the initial post. It looks to me that somehow messages in temp queues
>> are
>> retained in broker memory. I watched this broker OOM and I can say for
>> sure
>> that none of the live temp queues showed any build up before the OOM.
>>
>> Assuming for a moment that this problem is induced by an application, what
>> could possibly be causing this build up of ProducerBrokerExchange objects
>> which hold references to messages in temp queues? Again, I know that there
>> is no buildup in live temp queues so this is not the case of a slow
>> consumer
>> not keeping up with processing msg from a temp queue.
>>
>> As mentioned in the original post, we run with producerFlowControl=false,
>> no
>> persistence, and vm cursor. I ran netstat on a machine where the broker is
>> running and dont see any FIN_WAIT nor WAIT_CLOSE. There are no messages in
>> brokers log saying that memory usage is high. In fact, the jconsole broker
>> MBean doesnt show it being short on memory. The memory usage shows as 0!
>> Yet, the brokers hits OOM.
>>
>> -Jerry C
>>
>>
>>
>>
>>
>>
>>
>>
>> --
>> View this message in context:
>> http://activemq.2283324.n4.nabble.com/Broker-Leak-tp4660437p4660538.html
>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>
>
>
>
> --
> *Christian Posta*
> http://www.christianposta.com/blog
> twitter: @christianposta
>
>
--
*Christian Posta*
http://www.christianposta.com/blog
twitter: @christianposta
Re: Broker Leak
Posted by Christian Posta <ch...@gmail.com>.
So i guess in general i would think that a configuration with no PFC, vm
cursors, and no persistence is destined to run out of memory.
I wonder if what's happening in your case is these temporary queues are
getting filled with messages that aren't being consumed and then when the
reference hangs around in the Producer exchange, it's making a dent in the
heap space. If this happens enough times it seems like it can OOM. I'll try
to reproduce that on my side.
On Wed, Dec 12, 2012 at 7:55 AM, Jerry Cwiklik <cw...@us.ibm.com> wrote:
> Christian, thanks. The most recent experiment was run with a much smaller
> broker, Xmx=360M. I am trying to reproduce the problem I am seeing in a
> larger system which is not easy to debug. I can't share the large heap dump
> as it may contain proprietary data. But thanks for the offer to help
> examine
> it.
>
> I know that the large heap dump looks exactly as the screenshot I attached
> to the initial post. It looks to me that somehow messages in temp queues
> are
> retained in broker memory. I watched this broker OOM and I can say for sure
> that none of the live temp queues showed any build up before the OOM.
>
> Assuming for a moment that this problem is induced by an application, what
> could possibly be causing this build up of ProducerBrokerExchange objects
> which hold references to messages in temp queues? Again, I know that there
> is no buildup in live temp queues so this is not the case of a slow
> consumer
> not keeping up with processing msg from a temp queue.
>
> As mentioned in the original post, we run with producerFlowControl=false,
> no
> persistence, and vm cursor. I ran netstat on a machine where the broker is
> running and dont see any FIN_WAIT nor WAIT_CLOSE. There are no messages in
> brokers log saying that memory usage is high. In fact, the jconsole broker
> MBean doesnt show it being short on memory. The memory usage shows as 0!
> Yet, the brokers hits OOM.
>
> -Jerry C
>
>
>
>
>
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/Broker-Leak-tp4660437p4660538.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
--
*Christian Posta*
http://www.christianposta.com/blog
twitter: @christianposta
Re: Broker Leak
Posted by Jerry Cwiklik <cw...@us.ibm.com>.
Christian, thanks. The most recent experiment was run with a much smaller
broker, Xmx=360M. I am trying to reproduce the problem I am seeing in a
larger system which is not easy to debug. I can't share the large heap dump
as it may contain proprietary data. But thanks for the offer to help examine
it.
I know that the large heap dump looks exactly as the screenshot I attached
to the initial post. It looks to me that somehow messages in temp queues are
retained in broker memory. I watched this broker OOM and I can say for sure
that none of the live temp queues showed any build up before the OOM.
Assuming for a moment that this problem is induced by an application, what
could possibly be causing this build up of ProducerBrokerExchange objects
which hold references to messages in temp queues? Again, I know that there
is no buildup in live temp queues so this is not the case of a slow consumer
not keeping up with processing msg from a temp queue.
As mentioned in the original post, we run with producerFlowControl=false, no
persistence, and vm cursor. I ran netstat on a machine where the broker is
running and dont see any FIN_WAIT nor WAIT_CLOSE. There are no messages in
brokers log saying that memory usage is high. In fact, the jconsole broker
MBean doesnt show it being short on memory. The memory usage shows as 0!
Yet, the brokers hits OOM.
-Jerry C
--
View this message in context: http://activemq.2283324.n4.nabble.com/Broker-Leak-tp4660437p4660538.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Broker Leak
Posted by Christian Posta <ch...@gmail.com>.
No, that's not expected behavior. I ran your tests as you described and I
do see memory being consumed and churned, but it all eventually gets
collected by the GC.
Is your heap the 10G heap you mentioned originally? If you can get me the
heapdump, I'll gladly take a look for you. Let me know and I can set up a
share somehwere.
On Tue, Dec 11, 2012 at 1:32 PM, Jerry Cwiklik <cw...@us.ibm.com> wrote:
> Thanks, your reply seems consistent with what I am seeing in the heap dump
> (see heapAnalyzer screenshot attached to the initial post). I am still
> unclear how are the dead messages cleaned up (if ever). I ran a new
> scenario
> where I had 4 SpringConsumerWithReply processes and a single
> ProducerWithReply process which I start in a shell's endless for-loop.
> for i in (1 .. 1000000000)
> do
> java -jar producer-w-consumer.jar tcp://0.0.0.0:61616 serviceQ 100000
> 500000
> done
>
> I run the script and keep killing ProducerWithReply process which gets
> launched again. I dont touch any of the SpringConsumerWithReply processes.
> After a short while the broker OOMs.
>
> Given the above scenario, is it expected that the broker OOMs? I know that
> it OOMs due to messages destined for temp queues which no longer exist. How
> should the ProducerExchange instances be cleaned up? Should the
> SpringConsumerWithRep close a Session or Connection to a destination that
> it
> thinks is no longer used? Is closing the Session enough to purge
> ProducerExchange from the broker?
>
> -Jerry C
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/Broker-Leak-tp4660437p4660513.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
--
*Christian Posta*
http://www.christianposta.com/blog
twitter: @christianposta
Re: Broker Leak
Posted by Jerry Cwiklik <cw...@us.ibm.com>.
Thanks, your reply seems consistent with what I am seeing in the heap dump
(see heapAnalyzer screenshot attached to the initial post). I am still
unclear how are the dead messages cleaned up (if ever). I ran a new scenario
where I had 4 SpringConsumerWithReply processes and a single
ProducerWithReply process which I start in a shell's endless for-loop.
for i in (1 .. 1000000000)
do
java -jar producer-w-consumer.jar tcp://0.0.0.0:61616 serviceQ 100000
500000
done
I run the script and keep killing ProducerWithReply process which gets
launched again. I dont touch any of the SpringConsumerWithReply processes.
After a short while the broker OOMs.
Given the above scenario, is it expected that the broker OOMs? I know that
it OOMs due to messages destined for temp queues which no longer exist. How
should the ProducerExchange instances be cleaned up? Should the
SpringConsumerWithRep close a Session or Connection to a destination that it
thinks is no longer used? Is closing the Session enough to purge
ProducerExchange from the broker?
-Jerry C
--
View this message in context: http://activemq.2283324.n4.nabble.com/Broker-Leak-tp4660437p4660513.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Broker Leak
Posted by Christian Posta <ch...@gmail.com>.
No, it's not related to the half-closed socket. Everything gets cleaned up
properly on that end.
The destination (TempQueue) is removed from the broker as expected, but the
producer connection (on the broker side) from the SprignConsumerWithReply
module has an object (ProducerExchange) that keeps a reference to the
TempQueue because that's where it last sent a message.
The reference unfortunately keeps the entire object representing the
TempQueue around, including the references it has to the queue cursor. If
the producer sends another message, then that reference gets updated to
wherever the new message is being sent.
This is just cleanup that happens within the broker. The destination is no
longer considered there when it's in this state. The messages are not
reachable for dispatching or consuming.
On Tue, Dec 11, 2012 at 11:42 AM, Jerry Cwiklik <cw...@us.ibm.com> wrote:
> Thanks, yes this is true that there is an open connection to a temp queue
> from the the SpringConsumerWithReply service. This is probably a common
> scenario where one deploys a service process which services requests from
> clients which can disappear at any time. Cleaning up unused/dead
> connections
> would be an application concern. No argument there.
>
> What I dont understand is why do I have messages in the broker destined for
> a temp queue which no longer exist. When I kill the ProducerWithConsumer,
> the temp queue is removed but the broker retains messages for it, which
> causes a leak. Is this releated to what Gary mentioned, namely half closed
> socket? We do use soWriteTimeout but not soTimeout. I will try to run the
> test case with that.
>
> -Jerry C
>
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/Broker-Leak-tp4660437p4660508.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
--
*Christian Posta*
http://www.christianposta.com/blog
twitter: @christianposta
Re: Broker Leak
Posted by Jerry Cwiklik <cw...@us.ibm.com>.
Thanks, yes this is true that there is an open connection to a temp queue
from the the SpringConsumerWithReply service. This is probably a common
scenario where one deploys a service process which services requests from
clients which can disappear at any time. Cleaning up unused/dead connections
would be an application concern. No argument there.
What I dont understand is why do I have messages in the broker destined for
a temp queue which no longer exist. When I kill the ProducerWithConsumer,
the temp queue is removed but the broker retains messages for it, which
causes a leak. Is this releated to what Gary mentioned, namely half closed
socket? We do use soWriteTimeout but not soTimeout. I will try to run the
test case with that.
-Jerry C
--
View this message in context: http://activemq.2283324.n4.nabble.com/Broker-Leak-tp4660437p4660508.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Broker Leak
Posted by Christian Posta <ch...@gmail.com>.
So here's what's happening...
There are actually three (3) connections being created:
1.. by your ProducerWithConsumer.java
1.. by the Spring DMLC
1.. by your explicit code in SpringConsumerWithReply (which you use to
produce to the temp destination).
So the connection that I saw laying around was not from the
ProducerWithConsumer.java like I thought. That connection is cleaned up
properly so no need for extra TCP socket monitors (although Gary's
suggestion is always something to keep mind of... this can happen... it
comes from this JIRA: https://issues.apache.org/jira/browse/AMQ-2006)
But the connection for the *producer* in your SpringConsumerWithReply is
kept open and that connection still has the reference to the TempQueue.
This reference will be cleared when the producer goes away or if the
producer sends another message.
On Tue, Dec 11, 2012 at 7:44 AM, Gary Tully <ga...@gmail.com> wrote:
> you are looking at the need for a writeTimeoutFilter or inactivityMonitor
> or OS level tcp keepAlive to detect a half closed socket as the result of
> an abortive close of the client.
>
>
> On 11 December 2012 14:36, Christian Posta <christian.posta@gmail.com
> >wrote:
>
> > I just ran your tests.
> > Actually, this doesn't seem to be the case of the consumers not going
> away.
> > Looks like the connection wasn't properly cleaned up on the broker's side
> > after the kill -9. Let me try with the trunk and see if there's anything
> > different before I dig deeper to see why it wasn't cleaned up.
> >
> >
> > On Mon, Dec 10, 2012 at 2:37 PM, Jerry Cwiklik <cw...@us.ibm.com>
> wrote:
> >
> > > Thanks, so if the subs dont go away the broker thinks it can deliver
> the
> > > messages to them even though the destination is gone? Seems that a temp
> > > queue removal should result/trigger a cleanup of messages associated
> with
> > > that destination.
> > >
> > > I dont have JUnit test. Instead, I attach sources for a simple jms
> > producer
> > > and a consumer which I ran as separate processes. While the producer is
> > > blasting msgs I kill it with -9, force the GC in the broker's jvm via
> > > jConsole and dump the heap to analyze. I ran this scenario multiple
> times
> > > and am seeing dead messages in a VM cursor.
> > >
> > > ProducerWithConsumer.java
> > > <
> > >
> >
> http://activemq.2283324.n4.nabble.com/file/n4660465/ProducerWithConsumer.java
> > > >
> > > SpringConsumerWithReply.java
> > > <
> > >
> >
> http://activemq.2283324.n4.nabble.com/file/n4660465/SpringConsumerWithReply.java
> > > >
> > > Thanks, Jerry C
> > >
> > >
> > >
> > >
> > > --
> > > View this message in context:
> > >
> http://activemq.2283324.n4.nabble.com/Broker-Leak-tp4660437p4660465.html
> > > Sent from the ActiveMQ - User mailing list archive at Nabble.com.
> > >
> >
> >
> >
> > --
> > *Christian Posta*
> > http://www.christianposta.com/blog
> > twitter: @christianposta
> >
>
>
>
> --
> http://redhat.com
> http://blog.garytully.com
>
--
*Christian Posta*
http://www.christianposta.com/blog
twitter: @christianposta
Re: Broker Leak
Posted by Jerry Cwiklik <cw...@us.ibm.com>.
Our broker has been configured with
<transportConnector
name="openwire"
uri="tcp://0.0.0.0:61616?transport.soWriteTimeout=15000"
/>
We are experiencing the problem with the above setting.
Also, the broker occasionally logs the following:
WARN Transport -Async error occured: java.lang.IllegalStateException: Cannot
remove a consumer from a connection that had not been registered: ID: xxx
Not sure if the above is related to sockets not being closed.
-Jerry C
--
View this message in context: http://activemq.2283324.n4.nabble.com/Broker-Leak-tp4660437p4660506.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Broker Leak
Posted by Gary Tully <ga...@gmail.com>.
you are looking at the need for a writeTimeoutFilter or inactivityMonitor
or OS level tcp keepAlive to detect a half closed socket as the result of
an abortive close of the client.
On 11 December 2012 14:36, Christian Posta <ch...@gmail.com>wrote:
> I just ran your tests.
> Actually, this doesn't seem to be the case of the consumers not going away.
> Looks like the connection wasn't properly cleaned up on the broker's side
> after the kill -9. Let me try with the trunk and see if there's anything
> different before I dig deeper to see why it wasn't cleaned up.
>
>
> On Mon, Dec 10, 2012 at 2:37 PM, Jerry Cwiklik <cw...@us.ibm.com> wrote:
>
> > Thanks, so if the subs dont go away the broker thinks it can deliver the
> > messages to them even though the destination is gone? Seems that a temp
> > queue removal should result/trigger a cleanup of messages associated with
> > that destination.
> >
> > I dont have JUnit test. Instead, I attach sources for a simple jms
> producer
> > and a consumer which I ran as separate processes. While the producer is
> > blasting msgs I kill it with -9, force the GC in the broker's jvm via
> > jConsole and dump the heap to analyze. I ran this scenario multiple times
> > and am seeing dead messages in a VM cursor.
> >
> > ProducerWithConsumer.java
> > <
> >
> http://activemq.2283324.n4.nabble.com/file/n4660465/ProducerWithConsumer.java
> > >
> > SpringConsumerWithReply.java
> > <
> >
> http://activemq.2283324.n4.nabble.com/file/n4660465/SpringConsumerWithReply.java
> > >
> > Thanks, Jerry C
> >
> >
> >
> >
> > --
> > View this message in context:
> > http://activemq.2283324.n4.nabble.com/Broker-Leak-tp4660437p4660465.html
> > Sent from the ActiveMQ - User mailing list archive at Nabble.com.
> >
>
>
>
> --
> *Christian Posta*
> http://www.christianposta.com/blog
> twitter: @christianposta
>
--
http://redhat.com
http://blog.garytully.com
Re: Broker Leak
Posted by Christian Posta <ch...@gmail.com>.
I just ran your tests.
Actually, this doesn't seem to be the case of the consumers not going away.
Looks like the connection wasn't properly cleaned up on the broker's side
after the kill -9. Let me try with the trunk and see if there's anything
different before I dig deeper to see why it wasn't cleaned up.
On Mon, Dec 10, 2012 at 2:37 PM, Jerry Cwiklik <cw...@us.ibm.com> wrote:
> Thanks, so if the subs dont go away the broker thinks it can deliver the
> messages to them even though the destination is gone? Seems that a temp
> queue removal should result/trigger a cleanup of messages associated with
> that destination.
>
> I dont have JUnit test. Instead, I attach sources for a simple jms producer
> and a consumer which I ran as separate processes. While the producer is
> blasting msgs I kill it with -9, force the GC in the broker's jvm via
> jConsole and dump the heap to analyze. I ran this scenario multiple times
> and am seeing dead messages in a VM cursor.
>
> ProducerWithConsumer.java
> <
> http://activemq.2283324.n4.nabble.com/file/n4660465/ProducerWithConsumer.java
> >
> SpringConsumerWithReply.java
> <
> http://activemq.2283324.n4.nabble.com/file/n4660465/SpringConsumerWithReply.java
> >
> Thanks, Jerry C
>
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/Broker-Leak-tp4660437p4660465.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
--
*Christian Posta*
http://www.christianposta.com/blog
twitter: @christianposta
Re: Broker Leak
Posted by Jerry Cwiklik <cw...@us.ibm.com>.
Thanks, so if the subs dont go away the broker thinks it can deliver the
messages to them even though the destination is gone? Seems that a temp
queue removal should result/trigger a cleanup of messages associated with
that destination.
I dont have JUnit test. Instead, I attach sources for a simple jms producer
and a consumer which I ran as separate processes. While the producer is
blasting msgs I kill it with -9, force the GC in the broker's jvm via
jConsole and dump the heap to analyze. I ran this scenario multiple times
and am seeing dead messages in a VM cursor.
ProducerWithConsumer.java
<http://activemq.2283324.n4.nabble.com/file/n4660465/ProducerWithConsumer.java>
SpringConsumerWithReply.java
<http://activemq.2283324.n4.nabble.com/file/n4660465/SpringConsumerWithReply.java>
Thanks, Jerry C
--
View this message in context: http://activemq.2283324.n4.nabble.com/Broker-Leak-tp4660437p4660465.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Broker Leak
Posted by Christian Posta <ch...@gmail.com>.
That could be the case. When a temp dest goes away, it's subs do not
automatically go away. This is something we've been discussing recently.
Can you post the configs you used in your test, or better yet a unit test
that shows this?
On Mon, Dec 10, 2012 at 7:44 AM, Jerry Cwiklik <cw...@us.ibm.com> wrote:
> I am running AMQ broker v.5.6 on linux with 10G Xmx, 5G limit memoryUsage,
> producerFlowControl=false, no persistence, vmCursor and AUTO_ACKNOWLEDGE.
> There are a hundreds of producers and consumers, a few queues and a couple
> of hundred of temp queues. While watching broker memory in jConsole I
> notice
> a steady raise in OldGen. Forcing GC doesnt change anything. The broker
> eventually OOMs after a few days. The HeapAnalyzer shows 96% of heap in
> VmPendingMessageCursor. There are many messages destined for temp queues in
> this cursor.
>
> I created a simple test program where a Producer keeps sending messages to
> a
> Consumer which immediately sends a reply back to the Producer's temp queue.
> I kill the Producer with -9, force GC on the broker and dump the heap.
> Looking at the heap, I can see that the VMPendingCursor holds messages for
> the temp queue which no longer exist! It looks to me that the broker never
> cleans up messages destined for temp queues which no longer exist leading
> to
> a memory leak. Attached is a snapshot from the heapAnalyzer showing a
> message in VMPendingMessageCursor.
>
> Is there a way to configure the broker to remove such messages? We
> currently
> dont expire messages (TTL=0). The broker knows when a temp queue is deleted
> so it should try to clean up messages associated with it.
>
> <http://activemq.2283324.n4.nabble.com/file/n4660437/heap-dump-leak.png>
>
> Regards, Jerry C
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/Broker-Leak-tp4660437.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
--
*Christian Posta*
http://www.christianposta.com/blog
twitter: @christianposta