You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@activemq.apache.org by "matteo rulli (JIRA)" <ji...@apache.org> on 2014/09/15 12:06:35 UTC
[jira] [Comment Edited] (AMQ-5260) Cross talk over duplex network
connection can lead to blocking (alternative take)
[ https://issues.apache.org/jira/browse/AMQ-5260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14133752#comment-14133752 ]
matteo rulli edited comment on AMQ-5260 at 9/15/14 10:06 AM:
-------------------------------------------------------------
Ok, we managed to systematically reproduce the issue in a dedicated test-case (please find it as an attachment to this issue, *AMQ5260AdvancedTest.java*). This junit must be inserted in the _org.apache.activemq.bugs_ package within the activemq official src distro (version 5.10.0). In order to reproduce the deadlock in few seconds, the following configuration must be applied (VMArgs for the runnable jar of the junit test launch config):
{noformat}
-Dorg.apache.activemq.UseDedicatedTaskRunner=false
-Dbroker.ip.address=<broker ip goes here>
-Dnum.producers.spec.topic=5
-Dnum.consumers.spec.topic=5
-Dnum.producers.gen.topic=5
-Dnum.consumers.gen.topic=5
-Dnum.messages.sec=200
{noformat}
At the time being the problem seems related to a concurrence issue triggered by an DemandForwardingBridgeSupport#addSubscription and an DemandForwardingBridgeSupport#onCommand invocations.
To reproduce the issue it is necessary to create a test forcing such an invocations:
# create two brokers each one with a duplex network connector to the other one
# create a generic topic and create some producers publishing on this topic. Moreover, create some consumers consuming from this topic. Half of the producers/consumers are connected to one broker, the other half to the other one. This scenario should trigger the DemandForwardingBridgeSupport#onCommand invocation
# create some specific topic with one producer and some consumers. The producer is connected to one broker, the consumers to the other one. This scenario should trigger the DemandForwardingBridgeSupport#addSubscription invocation.
After few seconds after the junit start, just press the _detect deadlock_ button in jconsole.
As a side note, the deadlock shows up both with two duplex net connectors (A to B and B to A) and with a single duplex net connector (A to B only) between the two brokers.
was (Author: matteor):
Ok, we managed to systematically reproduce the issue in a dedicated test-case (please find it as an attachment to this issue, *AMQ5260AdvancedTest.java*). This junit must be inserted in the _org.apache.activemq.bugs_ package within the activemq official src distro (version 5.10.0). In order to reproduce the deadlock in few seconds, the following configuration must be applied (VMArgs for the runnable jar of the junit test launch config):
{noformat}
-Dorg.apache.activemq.UseDedicatedTaskRunner=false
-Dbroker.ip.address=<broker ip goes here>
-Dnum.producers.spec.topic=5
-Dnum.consumers.spec.topic=5
-Dnum.producers.gen.topic=5
-Dnum.consumers.gen.topic=5
-Dnum.messages.sec=200
{noformat}
At the time being the problem seems related to a concurrence issue triggered by an DemandForwardingBridgeSupport#addSubscription and an DemandForwardingBridgeSupport#onCommand invocations.
To reproduce the issue it is necessary to create a test forcing such an invocations:
# create two brokers each one with a duplex network connector to the other one
# create a generic topic and create some producers publishing on this topic. Moreover, create some consumers consuming from this topic. Half of the producers/consumers are connected to one broker, the other half to the other one. This scenario should trigger the DemandForwardingBridgeSupport#onCommand invocation
# create some specific topic with one producer and some consumers. The producer is connected to one broker, the consumers to the other one. This scenario should trigger the DemandForwardingBridgeSupport#addSubscription invocation.
After few seconds after the junit start, just press the _detect deadlock_ button in jconsole.
> Cross talk over duplex network connection can lead to blocking (alternative take)
> ---------------------------------------------------------------------------------
>
> Key: AMQ-5260
> URL: https://issues.apache.org/jira/browse/AMQ-5260
> Project: ActiveMQ
> Issue Type: Bug
> Affects Versions: 5.9.0, 5.10.0
> Reporter: matteo rulli
> Attachments: AMQ5260AdvancedTest.java, AMQ_5260.patch, AMQ_5260_2.patch, AMQ_5260_3.patch, deadlock.jpg, debug.jpg
>
>
> Pretty the same description with respect to AMQ-4328.
> ----
> !deadlock.jpg!
> h2. Stacktraces:
> Stacktrace no.1:
> {noformat}
> Name: ActiveMQ NIO Worker 12
> State: BLOCKED on java.net.URI@1bae2b28 owned by: ActiveMQ Transport: tcp:///10.0.1.219:61616@57789
> Total blocked: 2 Total waited: 67
> Stack trace:
> org.apache.activemq.network.DemandForwardingBridgeSupport.serviceRemoteConsumerAdvisory(DemandForwardingBridgeSupport.java:714)
> org.apache.activemq.network.DemandForwardingBridgeSupport.serviceRemoteCommand(DemandForwardingBridgeSupport.java:581)
> org.apache.activemq.network.DemandForwardingBridgeSupport$3.onCommand(DemandForwardingBridgeSupport.java:191)
> org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:116)
> org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)
> org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113)
> org.apache.activemq.transport.AbstractInactivityMonitor.onCommand(AbstractInactivityMonitor.java:270)
> org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)
> org.apache.activemq.transport.nio.NIOTransport.serviceRead(NIOTransport.java:138)
> org.apache.activemq.transport.nio.NIOTransport$1.onSelect(NIOTransport.java:69)
> org.apache.activemq.transport.nio.SelectorSelection.onSelect(SelectorSelection.java:94)
> org.apache.activemq.transport.nio.SelectorWorker$1.run(SelectorWorker.java:119)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> java.lang.Thread.run(Unknown Source)
> {noformat}
> ----
> stack trace no.2
> {noformat}
> Name: ActiveMQ Transport: tcp:///10.0.1.219:61616@57789
> State: WAITING on java.util.concurrent.locks.ReentrantLock$NonfairSync@3cdbfa3e owned by: ActiveMQ BrokerService[master2] Task-4
> Total blocked: 19 Total waited: 3
> Stack trace:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(Unknown Source)
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(Unknown Source)
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(Unknown Source)
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(Unknown Source)
> java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(Unknown Source)
> java.util.concurrent.locks.ReentrantLock.lock(Unknown Source)
> org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:66)
> org.apache.activemq.transport.ResponseCorrelator.oneway(ResponseCorrelator.java:60)
> org.apache.activemq.broker.TransportConnection.dispatch(TransportConnection.java:1339)
> org.apache.activemq.broker.TransportConnection.processDispatch(TransportConnection.java:858)
> org.apache.activemq.broker.TransportConnection.dispatchSync(TransportConnection.java:818)
> org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:151)
> org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:116)
> org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)
> org.apache.activemq.transport.vm.VMTransport.doDispatch(VMTransport.java:138)
> org.apache.activemq.transport.vm.VMTransport.dispatch(VMTransport.java:127)
> - locked java.util.concurrent.atomic.AtomicBoolean@689389da
> org.apache.activemq.transport.vm.VMTransport.oneway(VMTransport.java:104)
> org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:68)
> org.apache.activemq.transport.ResponseCorrelator.asyncRequest(ResponseCorrelator.java:81)
> org.apache.activemq.transport.ResponseCorrelator.request(ResponseCorrelator.java:86)
> org.apache.activemq.network.DemandForwardingBridgeSupport.addSubscription(DemandForwardingBridgeSupport.java:856)
> org.apache.activemq.network.DemandForwardingBridgeSupport.addConsumerInfo(DemandForwardingBridgeSupport.java:1128)
> org.apache.activemq.network.DemandForwardingBridgeSupport.serviceRemoteConsumerAdvisory(DemandForwardingBridgeSupport.java:714)
> - locked java.net.URI@1bae2b28
> org.apache.activemq.network.DemandForwardingBridgeSupport.serviceRemoteCommand(DemandForwardingBridgeSupport.java:581)
> org.apache.activemq.network.DemandForwardingBridgeSupport$3.onCommand(DemandForwardingBridgeSupport.java:191)
> org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:116)
> org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)
> org.apache.activemq.transport.failover.FailoverTransport$3.onCommand(FailoverTransport.java:196)
> org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113)
> org.apache.activemq.transport.AbstractInactivityMonitor.onCommand(AbstractInactivityMonitor.java:270)
> org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)
> org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:214)
> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:196)
> java.lang.Thread.run(Unknown Source)
> {noformat}
> ----
> stack trace no.3
> {noformat}
> Name: ActiveMQ BrokerService[master2] Task-4
> State: WAITING on java.util.concurrent.locks.ReentrantLock$NonfairSync@717bb9c owned by: ActiveMQ Transport: tcp:///10.0.1.219:61616@57789
> Total blocked: 19 Total waited: 117
> Stack trace:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(Unknown Source)
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(Unknown Source)
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(Unknown Source)
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(Unknown Source)
> java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(Unknown Source)
> java.util.concurrent.locks.ReentrantLock.lock(Unknown Source)
> org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:66)
> org.apache.activemq.transport.ResponseCorrelator.oneway(ResponseCorrelator.java:60)
> org.apache.activemq.network.DemandForwardingBridgeSupport.serviceLocalCommand(DemandForwardingBridgeSupport.java:921)
> org.apache.activemq.network.DemandForwardingBridgeSupport$2.onCommand(DemandForwardingBridgeSupport.java:173)
> org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:116)
> org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)
> org.apache.activemq.transport.vm.VMTransport.doDispatch(VMTransport.java:138)
> org.apache.activemq.transport.vm.VMTransport.dispatch(VMTransport.java:127)
> - locked java.util.concurrent.atomic.AtomicBoolean@453bb109
> org.apache.activemq.transport.vm.VMTransport.oneway(VMTransport.java:104)
> org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:68)
> org.apache.activemq.transport.ResponseCorrelator.oneway(ResponseCorrelator.java:60)
> org.apache.activemq.broker.TransportConnection.dispatch(TransportConnection.java:1339)
> org.apache.activemq.broker.TransportConnection.processDispatch(TransportConnection.java:858)
> org.apache.activemq.broker.TransportConnection.iterate(TransportConnection.java:904)
> org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:129)
> org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:47)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> java.lang.Thread.run(Unknown Source)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)