You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@activemq.apache.org by "Ranjit Nethi (JIRA)" <ji...@apache.org> on 2016/09/06 20:30:21 UTC
[jira] [Created] (AMQ-6419) Durable Subscribers going offline -
Deadlock - Network Of Brokers
Ranjit Nethi created AMQ-6419:
---------------------------------
Summary: Durable Subscribers going offline - Deadlock - Network Of Brokers
Key: AMQ-6419
URL: https://issues.apache.org/jira/browse/AMQ-6419
Project: ActiveMQ
Issue Type: Bug
Components: Broker
Affects Versions: 5.10.0
Environment: Linux - Centos 5.8
Active MQ Version 5.10.0
Container - Karaf 3.0.3
Camel 2.14.1
Reporter: Ranjit Nethi
We are using ActiveMQ, using Network of Brokers in Hub-Spoke topology.
Active MQ is deployed in Karaf Container
Software Stack Used
Active MQ Version 5.10.0
Container - Karaf 3.0.3
Camel 2.14.1
Hub is having 2 brokers- Master Active MQ and Slave Active MQ, configured using shared file system (GFS) and Kaha DB Data Store. Operate as failover. Spokes or Site connect to Hub using network connectors configuration below-
<networkConnectors>
<networkConnector
name="T:broker1->broker2"
uri="masterslave:(tcp://masterbroker.com:61616,tcp://slavebroker.com:61616)
duplex="true"
decreaseNetworkConsumerPriority="true"
networkTTL="2"
dynamicOnly="true">
</networkConnector>
<networkConnector
name="Q:broker1->broker2"
uri="masterslave:(tcp://masterbroker.com:61616,tcp://slavebroker.com:61616)
duplex="true"
decreaseNetworkConsumerPriority="true"
networkTTL="2"
dynamicOnly="true">
</networkConnector>
</networkConnectors>
Issue
We have topic on HUB AMQ broker called Shop.Canonical.WorkOrder.Notify, and the producers are at the spoke level. The producers write to the local topic Shop.Canonical.WorkOrder.Notify (This is the same topic AMQ creates when using network of brokers).
We have Durable Topic Subscribers on Central (HUB ) broker, camel client routes subscribed to this topic.
Apparently due to network connections lost or force shutdowns of servers hosting these karaf containers at spoke level, we are ending up having offline durable subscribers.
It looks like we had network outage and network failover tried to stop the network bridge and at the same time active mq tried to start another connection causing deadlock. I see the following exception from log file. Also the details of the threads causing deadlock towards the end of this message.
javax.jms.InvalidClientIDException: Broker: taxxx-amq-broker - Client: T:broker1->broker2_central-amq-broker_inbound_taxxx-amq-broker already connected from vm://taxxx-amq-broker#4548
at org.apache.activemq.broker.region.RegionBroker.addConnection(RegionBroker.java:246)[83:org.apache.activemq.activemq-osgi:5.10.0]
at org.apache.activemq.broker.jmx.ManagedRegionBroker.addConnection(ManagedRegionBroker.java:231)[83:org.apache.activemq.activemq-osgi:5.10.0]
at org.apache.activemq.broker.BrokerFilter.addConnection(BrokerFilter.java:92)[83:org.apache.activemq.activemq-osgi:5.10.0]
at org.apache.activemq.advisory.AdvisoryBroker.addConnection(AdvisoryBroker.java:89)[83:org.apache.activemq.activemq-osgi:5.10.0]
at org.apache.activemq.broker.BrokerFilter.addConnection(BrokerFilter.java:92)[83:org.apache.activemq.activemq-osgi:5.10.0]
at org.apache.activemq.broker.BrokerFilter.addConnection(BrokerFilter.java:92)[83:org.apache.activemq.activemq-osgi:5.10.0]
at org.apache.activemq.broker.BrokerFilter.addConnection(BrokerFilter.java:92)[83:org.apache.activemq.activemq-osgi:5.10.0]
at org.apache.activemq.security.JaasAuthenticationBroker.addConnection(JaasAuthenticationBroker.java:87)[83:org.apache.activemq.activemq-osgi:5.10.0]
at org.apache.activemq.broker.MutableBrokerFilter.addConnection(MutableBrokerFilter.java:97)[83:org.apache.activemq.activemq-osgi:5.10.0]
at org.apache.activemq.broker.TransportConnection.processAddConnection(TransportConnection.java:764)[83:org.apache.activemq.activemq-osgi:5.10.0]
at org.apache.activemq.command.ConnectionInfo.visit(ConnectionInfo.java:139)[83:org.apache.activemq.activemq-osgi:5.10.0]
at org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:294)[83:org.apache.activemq.activemq-osgi:5.10.0]
at org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:148)[83:org.apache.activemq.activemq-osgi:5.10.0]
at org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:116)[83:org.apache.activemq.activemq-osgi:5.10.0]
at org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)[83:org.apache.activemq.activemq-osgi:5.10.0]
at org.apache.activemq.transport.vm.VMTransport.doDispatch(VMTransport.java:138)[83:org.apache.activemq.activemq-osgi:5.10.0]
at org.apache.activemq.transport.vm.VMTransport.dispatch(VMTransport.java:130)[83:org.apache.activemq.activemq-osgi:5.10.0]
at org.apache.activemq.transport.vm.VMTransport.oneway(VMTransport.java:107)[83:org.apache.activemq.activemq-osgi:5.10.0]
at org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:68)[83:org.apache.activemq.activemq-osgi:5.10.0]
at org.apache.activemq.transport.ResponseCorrelator.asyncRequest(ResponseCorrelator.java:81)[83:org.apache.activemq.activemq-osgi:5.10.0]
at org.apache.activemq.transport.ResponseCorrelator.request(ResponseCorrelator.java:86)[83:org.apache.activemq.activemq-osgi:5.10.0]
at org.apache.activemq.network.DemandForwardingBridgeSupport.startLocalBridge(DemandForwardingBridgeSupport.java:453)[83:org.apache.activemq.activemq-osgi:5.10.0]
at org.apache.activemq.network.DemandForwardingBridgeSupport.doStartLocalAndRemoteBridges(DemandForwardingBridgeSupport.java:420)[83:org.apache.activemq.activemq-osgi:5.10.0]
at org.apache.activemq.network.DemandForwardingBridgeSupport.access$500(DemandForwardingBridgeSupport.java:105)[83:org.apache.activemq.activemq-osgi:5.10.0]
at org.apache.activemq.network.DemandForwardingBridgeSupport$5.run(DemandForwardingBridgeSupport.java:335)[83:org.apache.activemq.activemq-osgi:5.10.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)[:1.7.0_75]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)[:1.7.0_75]
at java.lang.Thread.run(Thread.java:745)[:1.7.0_75]
Above scenario also creates a dead lock condition - 3 threads
Thread 1 - Name: ActiveMQ BrokerService[taxxx-amq-broker] Task-17676
State: WAITING on java.util.concurrent.locks.ReentrantLock$NonfairSync@138f39e owned by: ActiveMQ Transport: tcp://shopesb1-prd-sl/10.98.1.92:61616@54480
Total blocked: 5 Total waited: 10
Stack trace:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:867)
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1197)
java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:214)
java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:290)
org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:66)
org.apache.activemq.transport.ResponseCorrelator.oneway(ResponseCorrelator.java:60)
org.apache.activemq.network.DemandForwardingBridgeSupport.serviceLocalCommand(DemandForwardingBridgeSupport.java:970)
org.apache.activemq.network.DemandForwardingBridgeSupport$2.onCommand(DemandForwardingBridgeSupport.java:206)
org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:116)
org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)
org.apache.activemq.transport.vm.VMTransport.doDispatch(VMTransport.java:138)
org.apache.activemq.transport.vm.VMTransport.dispatch(VMTransport.java:130)
- locked java.util.concurrent.atomic.AtomicBoolean@1e5f8fb
org.apache.activemq.transport.vm.VMTransport.oneway(VMTransport.java:107)
org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:68)
org.apache.activemq.transport.ResponseCorrelator.oneway(ResponseCorrelator.java:60)
org.apache.activemq.broker.TransportConnection.dispatch(TransportConnection.java:1370)
org.apache.activemq.broker.TransportConnection.processDispatch(TransportConnection.java:889)
org.apache.activemq.broker.TransportConnection.iterate(TransportConnection.java:935)
org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:133)
org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:48)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
java.lang.Thread.run(Thread.java:745)
Thread 2 -
Name: ActiveMQ Transport: tcp://shopesb1-prd-sl/10.98.1.92:61616@54480
State: WAITING on java.util.concurrent.locks.ReentrantLock$NonfairSync@1c83e84 owned by: ActiveMQ BrokerService[taxxx-amq-broker] Task-17676
Total blocked: 0 Total waited: 2
Stack trace:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:867)
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1197)
java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:214)
java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:290)
org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:66)
org.apache.activemq.transport.ResponseCorrelator.oneway(ResponseCorrelator.java:60)
org.apache.activemq.broker.TransportConnection.dispatch(TransportConnection.java:1370)
org.apache.activemq.broker.TransportConnection.processDispatch(TransportConnection.java:889)
org.apache.activemq.broker.TransportConnection.dispatchSync(TransportConnection.java:849)
org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:150)
org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:116)
org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)
org.apache.activemq.transport.vm.VMTransport.doDispatch(VMTransport.java:138)
org.apache.activemq.transport.vm.VMTransport.dispatch(VMTransport.java:130)
- locked java.util.concurrent.atomic.AtomicBoolean@12796af
org.apache.activemq.transport.vm.VMTransport.oneway(VMTransport.java:107)
org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:68)
org.apache.activemq.transport.ResponseCorrelator.asyncRequest(ResponseCorrelator.java:81)
org.apache.activemq.transport.ResponseCorrelator.request(ResponseCorrelator.java:86)
org.apache.activemq.network.DemandForwardingBridgeSupport.addSubscription(DemandForwardingBridgeSupport.java:905)
org.apache.activemq.network.DemandForwardingBridgeSupport.addConsumerInfo(DemandForwardingBridgeSupport.java:1178)
org.apache.activemq.network.DemandForwardingBridgeSupport.serviceRemoteConsumerAdvisory(DemandForwardingBridgeSupport.java:763)
- locked java.net.URI@15e6d6
org.apache.activemq.network.DemandForwardingBridgeSupport.serviceRemoteCommand(DemandForwardingBridgeSupport.java:614)
org.apache.activemq.network.DemandForwardingBridgeSupport$3.onCommand(DemandForwardingBridgeSupport.java:224)
org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:116)
org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)
org.apache.activemq.transport.failover.FailoverTransport$3.onCommand(FailoverTransport.java:208)
org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113)
org.apache.activemq.transport.AbstractInactivityMonitor.onCommand(AbstractInactivityMonitor.java:270)
org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)
org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:214)
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:196)
java.lang.Thread.run(Thread.java:745)
Thread 3 - Name: ActiveMQ BrokerService[taxxx-amq-broker] Task-18436
State: WAITING on java.util.concurrent.locks.ReentrantLock$NonfairSync@138f39e owned by: ActiveMQ Transport: tcp://shopesb1-prd-sl/10.98.1.92:61616@54480
Total blocked: 0 Total waited: 44
Stack trace:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:867)
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1197)
java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:214)
java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:290)
org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:66)
org.apache.activemq.transport.ResponseCorrelator.oneway(ResponseCorrelator.java:60)
org.apache.activemq.network.DemandForwardingBridgeSupport$4.run(DemandForwardingBridgeSupport.java:288)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
java.lang.Thread.run(Thread.java:745)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)