ActiveMQ duplex network connector dead lock [5.13.1, 5.11.1]

["yang.yang.zz" <ya...@outlook.com>]

Hi：

We ran into this dead lock since upgrade to 5.11.1. In short, this dead lock
happens on network connector when "duplex" is set to "true". For details of
replicating this defect, see my earlier post  here
<http://activemq.2283324.n4.nabble.com/5-13-1-message-blocked-td4708592.html>  

I also found an unresolved defect  AMQ-5260
<https://issues.apache.org/jira/browse/AMQ-5260>   which has the same
symptom. It looks like this issue could also be found in 5.9.x and 5.10.x.

The issue is exactly as described in AMQ-5260. But we'd like to limit the
code change impact within serviceRemoteConsumerAdvisory only. So below is
the code change we made, 

Inside the addConsumerInfo call at
DemandForwardingBridgeSupport.serviceRemoteConsumerAdvisory, we added a new
parameter for timeout

            // in a cyclic network there can be multiple bridges per broker
that can propagate
            // a network subscription so there is a need to synchronize on a
shared entity
            synchronized (brokerService.getVmConnectorURI()) {
            	addConsumerInfo(info*, 15000*);
            }

And then use this timeout under

    protected void addSubscription(DemandSubscription sub*, int timout*)
throws IOException {
        if (sub != null) {
            if (isDuplex()) {
                // async vm transport, need to wait for completion
                localBroker.request(sub.getLocalInfo()*, timout*);
            } else {
                localBroker.oneway(sub.getLocalInfo());
            }
        }
    }

So the difference is, when the dead lock happened, the localBroker may throw
RequestTimedOutIOException if there is a dead lock.

What could be the impact if addSubscription() hit timeout? why it is set to
be forever wait at the first place?


Thanks,
-Yang



--
View this message in context: http://activemq.2283324.n4.nabble.com/ActiveMQ-duplex-network-connector-dead-lock-5-13-1-5-11-1-tp4708953.html
Sent from the ActiveMQ - Dev mailing list archive at Nabble.com.

Re: ActiveMQ duplex network connector dead lock [5.13.1, 5.11.1]

["yang.yang.zz" <ya...@outlook.com>]

I tried the 3 patches matteo rulli provided in AMQ5260. It seems reduced the
chance of running into deadlock but it does not completely eliminate it. I
can still see FutureResponse.getResult being waiting forever. Therefore I
added another patch to make getResult can timeout after 30 secs and throw
Exception.

With these 4 patches, it passed our network failure test. All the
connections are fully recovered after the network failure. It takes 25-30
mins though, but it's much better than being blocked forever.

 


Name: ActiveMQ Transport: tcp:///10.130.156.205:46138@61616
State: WAITING on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@482aaa1a
Total blocked: 5  Total waited: 6

Stack trace:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403)
org.apache.activemq.transport.FutureResponse.getResult(FutureResponse.java:40)
org.apache.activemq.transport.ResponseCorrelator.request(ResponseCorrelator.java:87)
org.apache.activemq.network.DemandForwardingBridgeSupport.addSubscription(DemandForwardingBridgeSupport.java:911)
org.apache.activemq.network.DemandForwardingBridgeSupport.addConsumerInfo(DemandForwardingBridgeSupport.java:1184)
org.apache.activemq.network.DemandForwardingBridgeSupport.serviceRemoteConsumerAdvisory(DemandForwardingBridgeSup



--
View this message in context: http://activemq.2283324.n4.nabble.com/ActiveMQ-duplex-network-connector-dead-lock-5-13-1-5-11-1-tp4708953p4708972.html
Sent from the ActiveMQ - Dev mailing list archive at Nabble.com.