You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by RuralHunter <ru...@gmail.com> on 2016/07/19 08:57:56 UTC
Producer count keeps increasing
Hi,
I'm using activemq 5.13.3. I'm seeing the producer count on the broker side
keeps increasing and never shrinks. I inspected the value TotalProducerCount
of the broker via jmx. And it is also verified by the class count of
org.apache.activemq.broker.jmx.ProducerView in jvm heap class histogram.
We found the session.createProducer() method got slower and slower at the
client side if we keep activemq running for long time. We checked all other
metrics both activemq and the server where it runs and could not find any
other thing abnormal. The stack of the activemq server shows many of the
threads are doing this:
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00007fcfdb2edeb8> (a
java.util.concurrent.locks.ReentrantLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:867)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1197)
at
java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:214)
at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:290)
at
java.util.concurrent.CopyOnWriteArrayList.addIfAbsent(CopyOnWriteArrayList.java:585)
at
java.util.concurrent.CopyOnWriteArraySet.add(CopyOnWriteArraySet.java:248)
at
org.apache.activemq.broker.jmx.ManagedRegionBroker.registerProducer(ManagedRegionBroker.java:384)
at
org.apache.activemq.broker.jmx.ManagedRegionBroker.addProducer(ManagedRegionBroker.java:268)
at
org.apache.activemq.broker.CompositeDestinationBroker.addProducer(CompositeDestinationBroker.java:56)
at
org.apache.activemq.broker.BrokerFilter.addProducer(BrokerFilter.java:108)
at
org.apache.activemq.broker.MutableBrokerFilter.addProducer(MutableBrokerFilter.java:113)
at
org.apache.activemq.broker.TransportConnection.processAddProducer(TransportConnection.java:618)
at org.apache.activemq.command.ProducerInfo.visit(ProducerInfo.java:108)
at
org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:338)
at
org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:188)
at
org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)
at
org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:125)
at
org.apache.activemq.transport.AbstractInactivityMonitor.onCommand(AbstractInactivityMonitor.java:300)
at
org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)
at
org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:233)
at
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:215)
at java.lang.Thread.run(Thread.java:745)
The TotalProducerCount is more than 120000 when I checked it last time.
If I restart the activemq, the session.createProducer() becomes very quicks
at client side. I also noticed the TotalConnectionsCount also increases with
the TotalProducerCount. So looks there is some connection/producer leak at
server side. How should I handle this?
--
View this message in context: http://activemq.2283324.n4.nabble.com/Producer-count-keeps-increasing-tp4714087.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Producer count keeps increasing
Posted by RuralHunter <ru...@gmail.com>.
Yes, we have many clients produces messages and it's possible that some of
the clients didn't close the producer properly. We will check them closely.
Meantime, is there any way to prevent this at the server side? For example,
if the client network connection is broken, will the orphan producer get
closed at the server side?
--
View this message in context: http://activemq.2283324.n4.nabble.com/Producer-count-keeps-increasing-tp4714087p4714105.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Producer count keeps increasing
Posted by RuralHunter <ru...@gmail.com>.
Hi Tim,
Thanks. After we fixed the problematic client, ActiveMQ is working as
expected. The producer count stays stable even client exits abnormally.
ActiveMQ is doing the cleanup correctly.
--
View this message in context: http://activemq.2283324.n4.nabble.com/Producer-count-keeps-increasing-tp4714087p4714292.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Producer count keeps increasing
Posted by Tim Bain <tb...@alumni.duke.edu>.
Glad you found the problem.
The answer to your question about the behavior if the TCP connection is
broken is that yes, I believe that will cause a cleanup. But keep in mind
that hard-killing a process with an open TCP connection doesn't result in
the connection closing till the OS times it out, so depending on how you do
the close, there might be a delay before it happens (and a reconnect
attempt before the timeout happens might or might not prevent the cleanup
from happening). I haven't tested this, so if it's something you're
concerned about, I recommend you test it out to be sure it does what you
expect (and then report back to let us know what you found).
Tim
On Tue, Jul 19, 2016 at 6:45 PM, RuralHunter <ru...@gmail.com> wrote:
> We'v found the leaking client by checking the producer count of the queues
> one by one in JConsole.
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/Producer-count-keeps-increasing-tp4714087p4714138.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Re: Producer count keeps increasing
Posted by RuralHunter <ru...@gmail.com>.
We'v found the leaking client by checking the producer count of the queues
one by one in JConsole.
--
View this message in context: http://activemq.2283324.n4.nabble.com/Producer-count-keeps-increasing-tp4714087p4714138.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Producer count keeps increasing
Posted by Tim Bain <tb...@alumni.duke.edu>.
In your client code, it sounds like you're calling session.createProducer()
many times, possibly in a loop. Are you closing each producer when you're
done with it, before you create the next one? If not, the resource leak is
in your code, not ActiveMQ's.
Tim
On Jul 19, 2016 5:41 AM, "RuralHunter" <ru...@gmail.com> wrote:
> Hi,
> I'm using activemq 5.13.3. I'm seeing the producer count on the broker side
> keeps increasing and never shrinks. I inspected the value
> TotalProducerCount
> of the broker via jmx. And it is also verified by the class count of
> org.apache.activemq.broker.jmx.ProducerView in jvm heap class histogram.
> We found the session.createProducer() method got slower and slower at the
> client side if we keep activemq running for long time. We checked all other
> metrics both activemq and the server where it runs and could not find any
> other thing abnormal. The stack of the activemq server shows many of the
> threads are doing this:
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00007fcfdb2edeb8> (a
> java.util.concurrent.locks.ReentrantLock$NonfairSync)
> at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> at
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
> at
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:867)
> at
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1197)
> at
>
> java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:214)
> at
> java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:290)
> at
>
> java.util.concurrent.CopyOnWriteArrayList.addIfAbsent(CopyOnWriteArrayList.java:585)
> at
> java.util.concurrent.CopyOnWriteArraySet.add(CopyOnWriteArraySet.java:248)
> at
>
> org.apache.activemq.broker.jmx.ManagedRegionBroker.registerProducer(ManagedRegionBroker.java:384)
> at
>
> org.apache.activemq.broker.jmx.ManagedRegionBroker.addProducer(ManagedRegionBroker.java:268)
> at
>
> org.apache.activemq.broker.CompositeDestinationBroker.addProducer(CompositeDestinationBroker.java:56)
> at
> org.apache.activemq.broker.BrokerFilter.addProducer(BrokerFilter.java:108)
> at
>
> org.apache.activemq.broker.MutableBrokerFilter.addProducer(MutableBrokerFilter.java:113)
> at
>
> org.apache.activemq.broker.TransportConnection.processAddProducer(TransportConnection.java:618)
> at
> org.apache.activemq.command.ProducerInfo.visit(ProducerInfo.java:108)
> at
>
> org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:338)
> at
>
> org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:188)
> at
>
> org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)
> at
>
> org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:125)
> at
>
> org.apache.activemq.transport.AbstractInactivityMonitor.onCommand(AbstractInactivityMonitor.java:300)
> at
>
> org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)
> at
> org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:233)
> at
> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:215)
> at java.lang.Thread.run(Thread.java:745)
>
> The TotalProducerCount is more than 120000 when I checked it last time.
> If I restart the activemq, the session.createProducer() becomes very quicks
> at client side. I also noticed the TotalConnectionsCount also increases
> with
> the TotalProducerCount. So looks there is some connection/producer leak at
> server side. How should I handle this?
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/Producer-count-keeps-increasing-tp4714087.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>