Broker unable to send messages to topic

[cart_chiu <ca...@gmail.com>]

hi, we are experiencing an issue in our production environment when we
publish about 10k small messages.  I see 9 BLOCKED threads in the thread
dump:

priority:10 - threadId:0x00007f34b0070800 - nativeId:0x19c20 - nativeId
(decimal):105504 - state:BLOCKED
stackTrace:
java.lang.Thread.State: BLOCKED (on object monitor)
at org.apache.activemq.broker.region.Topic.doMessageSend(Topic.java:423)
- waiting to lock <0x000000076c328620> (a
org.apache.activemq.broker.region.Topic)
at org.apache.activemq.broker.region.Topic.send(Topic.java:404)
at
org.apache.activemq.broker.region.AbstractRegion.send(AbstractRegion.java:365)
at
org.apache.activemq.broker.region.RegionBroker.send(RegionBroker.java:523)
at org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:129)
at
org.apache.activemq.broker.CompositeDestinationBroker.send(CompositeDestinationBroker.java:96)
at
org.apache.activemq.broker.TransactionBroker.send(TransactionBroker.java:227)
at org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:129)
at
org.apache.activemq.broker.MutableBrokerFilter.send(MutableBrokerFilter.java:135)
at
org.apache.activemq.broker.TransportConnection.processMessage(TransportConnection.java:458)
at
org.apache.activemq.command.ActiveMQMessage.visit(ActiveMQMessage.java:681)
at
org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:306)
at
org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:179)
at
org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:69)
at
org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113)
at
org.apache.activemq.transport.InactivityMonitor.onCommand(InactivityMonitor.java:227)
- locked <0x00000007214c0ec8> (a
org.apache.activemq.transport.InactivityMonitor$1)
at
org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)
at
org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:220)
at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:202)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- None

Any ideas how we can tune our ActiveMQ settings to address this? 
NabbleThreadDump.txt
<http://activemq.2283324.n4.nabble.com/file/t379693/NabbleThreadDump.txt>  
NabbleActiveMQ.txt
<http://activemq.2283324.n4.nabble.com/file/t379693/NabbleActiveMQ.txt>  
Attached is our full thread dump and activemq.xml.  Thanks in advance!



--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html

Re: Broker unable to send messages to topic

[Tim Bain <tb...@alumni.duke.edu>]

Your subject implies that you're unable to send (which I read to mean
"unable to send any messages, at all, to this topic"), but looking at the
thread dump, I see the nine threads in BLOCKED state waiting to acquire the
same lock and a tenth thread that's holding that lock while doing work (see
thread dump below). That tenth thread doesn't show any signs of being
unable to make forward progress, which implies that it will eventually
release the lock, and one of the other threads will then be able to make
progress. If that's accurate, then there would be thread contention and
slow performance, but publishers would still be able to make (slow)
progress on sending messages. Is that what you're in fact seeing?

What would give us more information would be for you to take a CPU sampling
snapshot from JVisualVM while you're in this state and share it here, to
help us figure out where the time is being spent by the thread that's
holding the lock. Operations done while holding locks should always be as
fast as possible, but maybe something in your specific use case is causing
the operations that occur while holding the lock for that topic to be
slower than expected, and the CPU sampling snapshot would let one of us dig
into whether that's the case and what it is (and what might be the root
cause of it).

And what more can you tell us about the load that's being placed on the
broker and the patterns of publication and consumption: the number of
consumers on the topic in question, the rate of message publication to the
topic, whether selectors are in use on this topic's consumers, etc.? Is
this thread dump captured during a load test (where you're explicitly
trying to drive your system faster than it would normally go) or during
normal operations?

Thanks,
Tim

On Fri, Oct 25, 2019, 5:23 PM cart_chiu <ca...@gmail.com> wrote:

> hi, we are experiencing an issue in our production environment when we
> publish about 10k small messages.  I see 9 BLOCKED threads in the thread
> dump:
>
> priority:10 - threadId:0x00007f34b0070800 - nativeId:0x19c20 - nativeId
> (decimal):105504 - state:BLOCKED
> stackTrace:
> java.lang.Thread.State: BLOCKED (on object monitor)
> at org.apache.activemq.broker.region.Topic.doMessageSend(Topic.java:423)
> - waiting to lock <0x000000076c328620> (a
> org.apache.activemq.broker.region.Topic)
> at org.apache.activemq.broker.region.Topic.send(Topic.java:404)
> at
>
> org.apache.activemq.broker.region.AbstractRegion.send(AbstractRegion.java:365)
> at
> org.apache.activemq.broker.region.RegionBroker.send(RegionBroker.java:523)
> at org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:129)
> at
>
> org.apache.activemq.broker.CompositeDestinationBroker.send(CompositeDestinationBroker.java:96)
> at
>
> org.apache.activemq.broker.TransactionBroker.send(TransactionBroker.java:227)
> at org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:129)
> at
>
> org.apache.activemq.broker.MutableBrokerFilter.send(MutableBrokerFilter.java:135)
> at
>
> org.apache.activemq.broker.TransportConnection.processMessage(TransportConnection.java:458)
> at
> org.apache.activemq.command.ActiveMQMessage.visit(ActiveMQMessage.java:681)
> at
>
> org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:306)
> at
>
> org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:179)
> at
>
> org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:69)
> at
>
> org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113)
> at
>
> org.apache.activemq.transport.InactivityMonitor.onCommand(InactivityMonitor.java:227)
> - locked <0x00000007214c0ec8> (a
> org.apache.activemq.transport.InactivityMonitor$1)
> at
>
> org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)
> at
> org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:220)
> at
> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:202)
> at java.lang.Thread.run(Thread.java:745)
> Locked ownable synchronizers:
> - None
>
> Any ideas how we can tune our ActiveMQ settings to address this?
> NabbleThreadDump.txt
> <http://activemq.2283324.n4.nabble.com/file/t379693/NabbleThreadDump.txt>
>
> NabbleActiveMQ.txt
> <http://activemq.2283324.n4.nabble.com/file/t379693/NabbleActiveMQ.txt>
> Attached is our full thread dump and activemq.xml.  Thanks in advance!
>
>
>
> --
> Sent from:
> http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
>