You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@activemq.apache.org by "Arthur Naseef (JIRA)" <ji...@apache.org> on 2011/04/08 23:05:05 UTC

[jira] [Created] (AMQ-3277) Deadlock with static:(failover:(...)) configuration

Deadlock with static:(failover:(...)) configuration
---------------------------------------------------

                 Key: AMQ-3277
                 URL: https://issues.apache.org/jira/browse/AMQ-3277
             Project: ActiveMQ
          Issue Type: Bug
          Components: Broker
    Affects Versions: 5.4.2
         Environment: 5.4.2 with back-ported patches for AMQ-3176, AMQ-2736, AMQ-3124, AMQ-3125, AMQ-3167.

Two brokers, each with a single duplex network connection to the other, using static:(failover:(first,second)) URI format.
            Reporter: Arthur Naseef


Using failover transports with static transports to create a network of brokers with H/A pairs, the failovers lead to deadlocks.

This appears related to AMQ-3176, but the patch for that problem is included in our 5.4.2 build.

                <networkConnector name="nw1_from_nw2" uri="static:(failover:(tcp://127.0.0.1:60992,tcp://127.0.0.1:60002))"
                                  duplex="true" conduitSubscriptions="false" dynamicOnly="true"
                                  networkTTL="7" suppressDuplicateQueueSubscriptions="true"
                                  decreaseNetworkConsumerPriority="true"

                                  bridgeTempDestinations="true"
                                  >
                </networkConnector>

The deadlock occurs after stopping and restarting each broker once.  Here's the steps:

    - start broker1
    - start broker2
    * test connectivity
    - sthudown broker1
    - restart broker1
    * test connectivity
    - shutdown broker2
    - restart broker2
    * test connectivity

Here is the deadlock information from the JVM:

Java stack information for the threads listed above:
===================================================
"StartRemotelBridge: localBroker=vm://peer-nw2#2":
        at org.apache.activemq.transport.failover.FailoverTransport.oneway(FailoverTransport.java:477)
        - waiting to lock <0x00002aaac2df83c8> (a java.lang.Object)
        at org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:40)
        - locked <0x00002aaac2e050f8> (a java.lang.Object)
        at org.apache.activemq.transport.ResponseCorrelator.oneway(ResponseCorrelator.java:60)
        at org.apache.activemq.network.DemandForwardingBridgeSupport.startRemoteBridge(DemandForwardingBridgeSupport.java:352)
        - locked <0x00002aaac2df7c50> (a org.apache.activemq.network.DurableConduitBridge)
        at org.apache.activemq.network.DemandForwardingBridgeSupport$4.run(DemandForwardingBridgeSupport.java:277)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)
"ActiveMQ Failover Worker: 169776139":
        at org.apache.activemq.network.DemandForwardingBridgeSupport.startLocalBridge(DemandForwardingBridgeSupport.java:290)
        - waiting to lock <0x00002aaac2df7c50> (a org.apache.activemq.networFound one Java-level deadlock:
=============================
"StartRemotelBridge: localBroker=vm://peer-nw2#2":
  waiting to lock monitor 0x0000000044796938 (object 0x00002aaac2df83c8, a java.lang.Object),
  which is held by "ActiveMQ Failover Worker: 169776139"
"ActiveMQ Failover Worker: 169776139":
  waiting to lock monitor 0x00002aaac470f0b0 (object 0x00002aaac2df7c50, a org.apache.activemq.network.DurableConduitBridge),
  which is held by k.DurableConduitBridge)
        at org.apache.activemq.network.DemandForwardingBridgeSupport$2.transportResumed(DemandForwardingBridgeSupport.java:220)
        at org.apache.activemq.transport.TransportFilter.transportResumed(TransportFilter.java:109)
        at org.apache.activemq.transport.TransportFilter.transportResumed(TransportFilter.java:109)
        at org.apache.activemq.transport.failover.FailoverTransport.doReconnect(FailoverTransport.java:870)
        - locked <0x00002aaac2df83c8> (a java.lang.Object)
        at org.apache.activemq.transport.failover.FailoverTransport$2.iterate(FailoverTransport.java:135)
        - locked <0x00002aaac2df83d8> (a java.lang.Object)
        at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:98)
        at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)

Found 1 deadlock.


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

Re: [jira] [Created] (AMQ-3277) Deadlock with static:(failover:(...)) configuration

Posted by artnaseef <Ar...@Ticketmaster.com>.
Looking at the code, I see the "reconnectMutex" is one of the two objects
being locked.  Any tips on what the locks are protecting and order of
operations would greatly help diagnosing and fixing this issue.

--
View this message in context: http://activemq.2283324.n4.nabble.com/jira-Created-AMQ-3277-Deadlock-with-static-failover-configuration-tp3437438p3437443.html
Sent from the ActiveMQ - Dev mailing list archive at Nabble.com.

[jira] [Commented] (AMQ-3277) Deadlock with static:(failover:(...)) configuration

Posted by "Arthur Naseef (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/AMQ-3277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13038216#comment-13038216 ] 

Arthur Naseef commented on AMQ-3277:
------------------------------------

To use this new transport, just change "failover" to "single".  Here's an example for use in a NetworkConnector:

static:(single:(ssl://broker1.node1.bus:61616,ssl://broker2.node1.bus:61616))


> Deadlock with static:(failover:(...)) configuration
> ---------------------------------------------------
>
>                 Key: AMQ-3277
>                 URL: https://issues.apache.org/jira/browse/AMQ-3277
>             Project: ActiveMQ
>          Issue Type: Bug
>          Components: Broker
>    Affects Versions: 5.4.2
>         Environment: 5.4.2 with back-ported patches for AMQ-3176, AMQ-2736, AMQ-3124, AMQ-3125, AMQ-3167.
> Two brokers, each with a single duplex network connection to the other, using static:(failover:(first,second)) URI format.
>            Reporter: Arthur Naseef
>         Attachments: singleTransport.patch
>
>
> Using failover transports with static transports to create a network of brokers with H/A pairs, the failovers lead to deadlocks.
> This appears related to AMQ-3176, but the patch for that problem is included in our 5.4.2 build.
>                 <networkConnector name="nw1_from_nw2" uri="static:(failover:(tcp://127.0.0.1:60992,tcp://127.0.0.1:60002))"
>                                   duplex="true" conduitSubscriptions="false" dynamicOnly="true"
>                                   networkTTL="7" suppressDuplicateQueueSubscriptions="true"
>                                   decreaseNetworkConsumerPriority="true"
>                                   bridgeTempDestinations="true"
>                                   >
>                 </networkConnector>
> The deadlock occurs after stopping and restarting each broker once.  Here's the steps:
>     - start broker1
>     - start broker2
>     * test connectivity
>     - sthudown broker1
>     - restart broker1
>     * test connectivity
>     - shutdown broker2
>     - restart broker2
>     * test connectivity
> Here is the deadlock information from the JVM:
> Java stack information for the threads listed above:
> ===================================================
> "StartRemotelBridge: localBroker=vm://peer-nw2#2":
>         at org.apache.activemq.transport.failover.FailoverTransport.oneway(FailoverTransport.java:477)
>         - waiting to lock <0x00002aaac2df83c8> (a java.lang.Object)
>         at org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:40)
>         - locked <0x00002aaac2e050f8> (a java.lang.Object)
>         at org.apache.activemq.transport.ResponseCorrelator.oneway(ResponseCorrelator.java:60)
>         at org.apache.activemq.network.DemandForwardingBridgeSupport.startRemoteBridge(DemandForwardingBridgeSupport.java:352)
>         - locked <0x00002aaac2df7c50> (a org.apache.activemq.network.DurableConduitBridge)
>         at org.apache.activemq.network.DemandForwardingBridgeSupport$4.run(DemandForwardingBridgeSupport.java:277)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>         at java.lang.Thread.run(Thread.java:662)
> "ActiveMQ Failover Worker: 169776139":
>         at org.apache.activemq.network.DemandForwardingBridgeSupport.startLocalBridge(DemandForwardingBridgeSupport.java:290)
>         - waiting to lock <0x00002aaac2df7c50> (a org.apache.activemq.networFound one Java-level deadlock:
> =============================
> "StartRemotelBridge: localBroker=vm://peer-nw2#2":
>   waiting to lock monitor 0x0000000044796938 (object 0x00002aaac2df83c8, a java.lang.Object),
>   which is held by "ActiveMQ Failover Worker: 169776139"
> "ActiveMQ Failover Worker: 169776139":
>   waiting to lock monitor 0x00002aaac470f0b0 (object 0x00002aaac2df7c50, a org.apache.activemq.network.DurableConduitBridge),
>   which is held by k.DurableConduitBridge)
>         at org.apache.activemq.network.DemandForwardingBridgeSupport$2.transportResumed(DemandForwardingBridgeSupport.java:220)
>         at org.apache.activemq.transport.TransportFilter.transportResumed(TransportFilter.java:109)
>         at org.apache.activemq.transport.TransportFilter.transportResumed(TransportFilter.java:109)
>         at org.apache.activemq.transport.failover.FailoverTransport.doReconnect(FailoverTransport.java:870)
>         - locked <0x00002aaac2df83c8> (a java.lang.Object)
>         at org.apache.activemq.transport.failover.FailoverTransport$2.iterate(FailoverTransport.java:135)
>         - locked <0x00002aaac2df83d8> (a java.lang.Object)
>         at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:98)
>         at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)
> Found 1 deadlock.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (AMQ-3277) Deadlock with static:(failover:(...)) configuration

Posted by "Arthur Naseef (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/AMQ-3277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13017647#comment-13017647 ] 

Arthur Naseef commented on AMQ-3277:
------------------------------------

Re-pasting the deadlock information; it was garbled in the last copy-and-paste.

Found one Java-level deadlock:
=============================
"StartRemotelBridge: localBroker=vm://peer-nw2#2":
  waiting to lock monitor 0x0000000044796938 (object 0x00002aaac2df83c8, a java.lang.Object),
  which is held by "ActiveMQ Failover Worker: 169776139"
"ActiveMQ Failover Worker: 169776139":
  waiting to lock monitor 0x00002aaac470f0b0 (object 0x00002aaac2df7c50, a org.apache.activemq.network.DurableConduitBridge),
  which is held by "StartRemotelBridge: localBroker=vm://peer-nw2#2"

Java stack information for the threads listed above:
===================================================
"StartRemotelBridge: localBroker=vm://peer-nw2#2":
        at org.apache.activemq.transport.failover.FailoverTransport.oneway(FailoverTransport.java:477)
        - waiting to lock <0x00002aaac2df83c8> (a java.lang.Object)
        at org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:40)
        - locked <0x00002aaac2e050f8> (a java.lang.Object)
        at org.apache.activemq.transport.ResponseCorrelator.oneway(ResponseCorrelator.java:60)
        at org.apache.activemq.network.DemandForwardingBridgeSupport.startRemoteBridge(DemandForwardingBridgeSupport.java:352)
        - locked <0x00002aaac2df7c50> (a org.apache.activemq.network.DurableConduitBridge)
        at org.apache.activemq.network.DemandForwardingBridgeSupport$4.run(DemandForwardingBridgeSupport.java:277)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)
"ActiveMQ Failover Worker: 169776139":
        at org.apache.activemq.network.DemandForwardingBridgeSupport.startLocalBridge(DemandForwardingBridgeSupport.java:290)
        - waiting to lock <0x00002aaac2df7c50> (a org.apache.activemq.network.DurableConduitBridge)
        at org.apache.activemq.network.DemandForwardingBridgeSupport$2.transportResumed(DemandForwardingBridgeSupport.java:220)
        at org.apache.activemq.transport.TransportFilter.transportResumed(TransportFilter.java:109)
        at org.apache.activemq.transport.TransportFilter.transportResumed(TransportFilter.java:109)
        at org.apache.activemq.transport.failover.FailoverTransport.doReconnect(FailoverTransport.java:870)
        - locked <0x00002aaac2df83c8> (a java.lang.Object)
        at org.apache.activemq.transport.failover.FailoverTransport$2.iterate(FailoverTransport.java:135)
        - locked <0x00002aaac2df83d8> (a java.lang.Object)
        at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:98)
        at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)

Found 1 deadlock.


> Deadlock with static:(failover:(...)) configuration
> ---------------------------------------------------
>
>                 Key: AMQ-3277
>                 URL: https://issues.apache.org/jira/browse/AMQ-3277
>             Project: ActiveMQ
>          Issue Type: Bug
>          Components: Broker
>    Affects Versions: 5.4.2
>         Environment: 5.4.2 with back-ported patches for AMQ-3176, AMQ-2736, AMQ-3124, AMQ-3125, AMQ-3167.
> Two brokers, each with a single duplex network connection to the other, using static:(failover:(first,second)) URI format.
>            Reporter: Arthur Naseef
>
> Using failover transports with static transports to create a network of brokers with H/A pairs, the failovers lead to deadlocks.
> This appears related to AMQ-3176, but the patch for that problem is included in our 5.4.2 build.
>                 <networkConnector name="nw1_from_nw2" uri="static:(failover:(tcp://127.0.0.1:60992,tcp://127.0.0.1:60002))"
>                                   duplex="true" conduitSubscriptions="false" dynamicOnly="true"
>                                   networkTTL="7" suppressDuplicateQueueSubscriptions="true"
>                                   decreaseNetworkConsumerPriority="true"
>                                   bridgeTempDestinations="true"
>                                   >
>                 </networkConnector>
> The deadlock occurs after stopping and restarting each broker once.  Here's the steps:
>     - start broker1
>     - start broker2
>     * test connectivity
>     - sthudown broker1
>     - restart broker1
>     * test connectivity
>     - shutdown broker2
>     - restart broker2
>     * test connectivity
> Here is the deadlock information from the JVM:
> Java stack information for the threads listed above:
> ===================================================
> "StartRemotelBridge: localBroker=vm://peer-nw2#2":
>         at org.apache.activemq.transport.failover.FailoverTransport.oneway(FailoverTransport.java:477)
>         - waiting to lock <0x00002aaac2df83c8> (a java.lang.Object)
>         at org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:40)
>         - locked <0x00002aaac2e050f8> (a java.lang.Object)
>         at org.apache.activemq.transport.ResponseCorrelator.oneway(ResponseCorrelator.java:60)
>         at org.apache.activemq.network.DemandForwardingBridgeSupport.startRemoteBridge(DemandForwardingBridgeSupport.java:352)
>         - locked <0x00002aaac2df7c50> (a org.apache.activemq.network.DurableConduitBridge)
>         at org.apache.activemq.network.DemandForwardingBridgeSupport$4.run(DemandForwardingBridgeSupport.java:277)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>         at java.lang.Thread.run(Thread.java:662)
> "ActiveMQ Failover Worker: 169776139":
>         at org.apache.activemq.network.DemandForwardingBridgeSupport.startLocalBridge(DemandForwardingBridgeSupport.java:290)
>         - waiting to lock <0x00002aaac2df7c50> (a org.apache.activemq.networFound one Java-level deadlock:
> =============================
> "StartRemotelBridge: localBroker=vm://peer-nw2#2":
>   waiting to lock monitor 0x0000000044796938 (object 0x00002aaac2df83c8, a java.lang.Object),
>   which is held by "ActiveMQ Failover Worker: 169776139"
> "ActiveMQ Failover Worker: 169776139":
>   waiting to lock monitor 0x00002aaac470f0b0 (object 0x00002aaac2df7c50, a org.apache.activemq.network.DurableConduitBridge),
>   which is held by k.DurableConduitBridge)
>         at org.apache.activemq.network.DemandForwardingBridgeSupport$2.transportResumed(DemandForwardingBridgeSupport.java:220)
>         at org.apache.activemq.transport.TransportFilter.transportResumed(TransportFilter.java:109)
>         at org.apache.activemq.transport.TransportFilter.transportResumed(TransportFilter.java:109)
>         at org.apache.activemq.transport.failover.FailoverTransport.doReconnect(FailoverTransport.java:870)
>         - locked <0x00002aaac2df83c8> (a java.lang.Object)
>         at org.apache.activemq.transport.failover.FailoverTransport$2.iterate(FailoverTransport.java:135)
>         - locked <0x00002aaac2df83d8> (a java.lang.Object)
>         at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:98)
>         at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)
> Found 1 deadlock.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (AMQ-3277) Deadlock with static:(failover:(...)) configuration

Posted by "Gary Tully (Resolved) (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/AMQ-3277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Gary Tully resolved AMQ-3277.
-----------------------------

       Resolution: Fixed
    Fix Version/s: 5.6.0
         Assignee: Gary Tully

maxReconnectAttempts=1 on the failover uri sorts this.
                
> Deadlock with static:(failover:(...)) configuration
> ---------------------------------------------------
>
>                 Key: AMQ-3277
>                 URL: https://issues.apache.org/jira/browse/AMQ-3277
>             Project: ActiveMQ
>          Issue Type: Bug
>          Components: Broker
>    Affects Versions: 5.4.2
>         Environment: 5.4.2 with back-ported patches for AMQ-3176, AMQ-2736, AMQ-3124, AMQ-3125, AMQ-3167.
> Two brokers, each with a single duplex network connection to the other, using static:(failover:(first,second)) URI format.
>            Reporter: Arthur Naseef
>            Assignee: Gary Tully
>             Fix For: 5.6.0
>
>         Attachments: singleTransport.patch
>
>
> Using failover transports with static transports to create a network of brokers with H/A pairs, the failovers lead to deadlocks.
> This appears related to AMQ-3176, but the patch for that problem is included in our 5.4.2 build.
>                 <networkConnector name="nw1_from_nw2" uri="static:(failover:(tcp://127.0.0.1:60992,tcp://127.0.0.1:60002))"
>                                   duplex="true" conduitSubscriptions="false" dynamicOnly="true"
>                                   networkTTL="7" suppressDuplicateQueueSubscriptions="true"
>                                   decreaseNetworkConsumerPriority="true"
>                                   bridgeTempDestinations="true"
>                                   >
>                 </networkConnector>
> The deadlock occurs after stopping and restarting each broker once.  Here's the steps:
>     - start broker1
>     - start broker2
>     * test connectivity
>     - sthudown broker1
>     - restart broker1
>     * test connectivity
>     - shutdown broker2
>     - restart broker2
>     * test connectivity
> Here is the deadlock information from the JVM:
> Java stack information for the threads listed above:
> ===================================================
> "StartRemotelBridge: localBroker=vm://peer-nw2#2":
>         at org.apache.activemq.transport.failover.FailoverTransport.oneway(FailoverTransport.java:477)
>         - waiting to lock <0x00002aaac2df83c8> (a java.lang.Object)
>         at org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:40)
>         - locked <0x00002aaac2e050f8> (a java.lang.Object)
>         at org.apache.activemq.transport.ResponseCorrelator.oneway(ResponseCorrelator.java:60)
>         at org.apache.activemq.network.DemandForwardingBridgeSupport.startRemoteBridge(DemandForwardingBridgeSupport.java:352)
>         - locked <0x00002aaac2df7c50> (a org.apache.activemq.network.DurableConduitBridge)
>         at org.apache.activemq.network.DemandForwardingBridgeSupport$4.run(DemandForwardingBridgeSupport.java:277)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>         at java.lang.Thread.run(Thread.java:662)
> "ActiveMQ Failover Worker: 169776139":
>         at org.apache.activemq.network.DemandForwardingBridgeSupport.startLocalBridge(DemandForwardingBridgeSupport.java:290)
>         - waiting to lock <0x00002aaac2df7c50> (a org.apache.activemq.networFound one Java-level deadlock:
> =============================
> "StartRemotelBridge: localBroker=vm://peer-nw2#2":
>   waiting to lock monitor 0x0000000044796938 (object 0x00002aaac2df83c8, a java.lang.Object),
>   which is held by "ActiveMQ Failover Worker: 169776139"
> "ActiveMQ Failover Worker: 169776139":
>   waiting to lock monitor 0x00002aaac470f0b0 (object 0x00002aaac2df7c50, a org.apache.activemq.network.DurableConduitBridge),
>   which is held by k.DurableConduitBridge)
>         at org.apache.activemq.network.DemandForwardingBridgeSupport$2.transportResumed(DemandForwardingBridgeSupport.java:220)
>         at org.apache.activemq.transport.TransportFilter.transportResumed(TransportFilter.java:109)
>         at org.apache.activemq.transport.TransportFilter.transportResumed(TransportFilter.java:109)
>         at org.apache.activemq.transport.failover.FailoverTransport.doReconnect(FailoverTransport.java:870)
>         - locked <0x00002aaac2df83c8> (a java.lang.Object)
>         at org.apache.activemq.transport.failover.FailoverTransport$2.iterate(FailoverTransport.java:135)
>         - locked <0x00002aaac2df83d8> (a java.lang.Object)
>         at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:98)
>         at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)
> Found 1 deadlock.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (AMQ-3277) Deadlock with static:(failover:(...)) configuration

Posted by "Gary Tully (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/AMQ-3277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13038486#comment-13038486 ] 

Gary Tully commented on AMQ-3277:
---------------------------------

Using failover: with a network connector is limited to choosing one of the composite urls to connect to. It should no be used to implement failover of the transport because the network connector handles failover itself.
To achieve what arthur describes as "single" transport, the maxReconnectAttempts=1 url parameter must be used in the failover url so that it will report a transport failure back to the networkConnector such that it can initiate a recreation of the bridge.
{code}<networkConnector uri="static:(failover:(tcp://xxx:32258,tcp://yyy:32259)?randomize=false&amp;maxReconnectAttempts=1)"/>{code}
see: https://issues.apache.org/jira/browse/AMQ-3222 

> Deadlock with static:(failover:(...)) configuration
> ---------------------------------------------------
>
>                 Key: AMQ-3277
>                 URL: https://issues.apache.org/jira/browse/AMQ-3277
>             Project: ActiveMQ
>          Issue Type: Bug
>          Components: Broker
>    Affects Versions: 5.4.2
>         Environment: 5.4.2 with back-ported patches for AMQ-3176, AMQ-2736, AMQ-3124, AMQ-3125, AMQ-3167.
> Two brokers, each with a single duplex network connection to the other, using static:(failover:(first,second)) URI format.
>            Reporter: Arthur Naseef
>         Attachments: singleTransport.patch
>
>
> Using failover transports with static transports to create a network of brokers with H/A pairs, the failovers lead to deadlocks.
> This appears related to AMQ-3176, but the patch for that problem is included in our 5.4.2 build.
>                 <networkConnector name="nw1_from_nw2" uri="static:(failover:(tcp://127.0.0.1:60992,tcp://127.0.0.1:60002))"
>                                   duplex="true" conduitSubscriptions="false" dynamicOnly="true"
>                                   networkTTL="7" suppressDuplicateQueueSubscriptions="true"
>                                   decreaseNetworkConsumerPriority="true"
>                                   bridgeTempDestinations="true"
>                                   >
>                 </networkConnector>
> The deadlock occurs after stopping and restarting each broker once.  Here's the steps:
>     - start broker1
>     - start broker2
>     * test connectivity
>     - sthudown broker1
>     - restart broker1
>     * test connectivity
>     - shutdown broker2
>     - restart broker2
>     * test connectivity
> Here is the deadlock information from the JVM:
> Java stack information for the threads listed above:
> ===================================================
> "StartRemotelBridge: localBroker=vm://peer-nw2#2":
>         at org.apache.activemq.transport.failover.FailoverTransport.oneway(FailoverTransport.java:477)
>         - waiting to lock <0x00002aaac2df83c8> (a java.lang.Object)
>         at org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:40)
>         - locked <0x00002aaac2e050f8> (a java.lang.Object)
>         at org.apache.activemq.transport.ResponseCorrelator.oneway(ResponseCorrelator.java:60)
>         at org.apache.activemq.network.DemandForwardingBridgeSupport.startRemoteBridge(DemandForwardingBridgeSupport.java:352)
>         - locked <0x00002aaac2df7c50> (a org.apache.activemq.network.DurableConduitBridge)
>         at org.apache.activemq.network.DemandForwardingBridgeSupport$4.run(DemandForwardingBridgeSupport.java:277)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>         at java.lang.Thread.run(Thread.java:662)
> "ActiveMQ Failover Worker: 169776139":
>         at org.apache.activemq.network.DemandForwardingBridgeSupport.startLocalBridge(DemandForwardingBridgeSupport.java:290)
>         - waiting to lock <0x00002aaac2df7c50> (a org.apache.activemq.networFound one Java-level deadlock:
> =============================
> "StartRemotelBridge: localBroker=vm://peer-nw2#2":
>   waiting to lock monitor 0x0000000044796938 (object 0x00002aaac2df83c8, a java.lang.Object),
>   which is held by "ActiveMQ Failover Worker: 169776139"
> "ActiveMQ Failover Worker: 169776139":
>   waiting to lock monitor 0x00002aaac470f0b0 (object 0x00002aaac2df7c50, a org.apache.activemq.network.DurableConduitBridge),
>   which is held by k.DurableConduitBridge)
>         at org.apache.activemq.network.DemandForwardingBridgeSupport$2.transportResumed(DemandForwardingBridgeSupport.java:220)
>         at org.apache.activemq.transport.TransportFilter.transportResumed(TransportFilter.java:109)
>         at org.apache.activemq.transport.TransportFilter.transportResumed(TransportFilter.java:109)
>         at org.apache.activemq.transport.failover.FailoverTransport.doReconnect(FailoverTransport.java:870)
>         - locked <0x00002aaac2df83c8> (a java.lang.Object)
>         at org.apache.activemq.transport.failover.FailoverTransport$2.iterate(FailoverTransport.java:135)
>         - locked <0x00002aaac2df83d8> (a java.lang.Object)
>         at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:98)
>         at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)
> Found 1 deadlock.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (AMQ-3277) Deadlock with static:(failover:(...)) configuration

Posted by "Arthur Naseef (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/AMQ-3277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Arthur Naseef updated AMQ-3277:
-------------------------------

    Attachment: singleTransport.patch

Patch which creates a new transport "single".  This transport connects to the first composite URL in its list and returns that connection, failing if none connect.

It has no failover features.

Used with NetworkConnectors, this transport operates well with NetworkConnectors since the network bridges are automatically recreated as-needed.  It appears the redundancy - reconnection at the network bridge level and at the failover transport level - may have lead to the deadlock.

> Deadlock with static:(failover:(...)) configuration
> ---------------------------------------------------
>
>                 Key: AMQ-3277
>                 URL: https://issues.apache.org/jira/browse/AMQ-3277
>             Project: ActiveMQ
>          Issue Type: Bug
>          Components: Broker
>    Affects Versions: 5.4.2
>         Environment: 5.4.2 with back-ported patches for AMQ-3176, AMQ-2736, AMQ-3124, AMQ-3125, AMQ-3167.
> Two brokers, each with a single duplex network connection to the other, using static:(failover:(first,second)) URI format.
>            Reporter: Arthur Naseef
>         Attachments: singleTransport.patch
>
>
> Using failover transports with static transports to create a network of brokers with H/A pairs, the failovers lead to deadlocks.
> This appears related to AMQ-3176, but the patch for that problem is included in our 5.4.2 build.
>                 <networkConnector name="nw1_from_nw2" uri="static:(failover:(tcp://127.0.0.1:60992,tcp://127.0.0.1:60002))"
>                                   duplex="true" conduitSubscriptions="false" dynamicOnly="true"
>                                   networkTTL="7" suppressDuplicateQueueSubscriptions="true"
>                                   decreaseNetworkConsumerPriority="true"
>                                   bridgeTempDestinations="true"
>                                   >
>                 </networkConnector>
> The deadlock occurs after stopping and restarting each broker once.  Here's the steps:
>     - start broker1
>     - start broker2
>     * test connectivity
>     - sthudown broker1
>     - restart broker1
>     * test connectivity
>     - shutdown broker2
>     - restart broker2
>     * test connectivity
> Here is the deadlock information from the JVM:
> Java stack information for the threads listed above:
> ===================================================
> "StartRemotelBridge: localBroker=vm://peer-nw2#2":
>         at org.apache.activemq.transport.failover.FailoverTransport.oneway(FailoverTransport.java:477)
>         - waiting to lock <0x00002aaac2df83c8> (a java.lang.Object)
>         at org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:40)
>         - locked <0x00002aaac2e050f8> (a java.lang.Object)
>         at org.apache.activemq.transport.ResponseCorrelator.oneway(ResponseCorrelator.java:60)
>         at org.apache.activemq.network.DemandForwardingBridgeSupport.startRemoteBridge(DemandForwardingBridgeSupport.java:352)
>         - locked <0x00002aaac2df7c50> (a org.apache.activemq.network.DurableConduitBridge)
>         at org.apache.activemq.network.DemandForwardingBridgeSupport$4.run(DemandForwardingBridgeSupport.java:277)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>         at java.lang.Thread.run(Thread.java:662)
> "ActiveMQ Failover Worker: 169776139":
>         at org.apache.activemq.network.DemandForwardingBridgeSupport.startLocalBridge(DemandForwardingBridgeSupport.java:290)
>         - waiting to lock <0x00002aaac2df7c50> (a org.apache.activemq.networFound one Java-level deadlock:
> =============================
> "StartRemotelBridge: localBroker=vm://peer-nw2#2":
>   waiting to lock monitor 0x0000000044796938 (object 0x00002aaac2df83c8, a java.lang.Object),
>   which is held by "ActiveMQ Failover Worker: 169776139"
> "ActiveMQ Failover Worker: 169776139":
>   waiting to lock monitor 0x00002aaac470f0b0 (object 0x00002aaac2df7c50, a org.apache.activemq.network.DurableConduitBridge),
>   which is held by k.DurableConduitBridge)
>         at org.apache.activemq.network.DemandForwardingBridgeSupport$2.transportResumed(DemandForwardingBridgeSupport.java:220)
>         at org.apache.activemq.transport.TransportFilter.transportResumed(TransportFilter.java:109)
>         at org.apache.activemq.transport.TransportFilter.transportResumed(TransportFilter.java:109)
>         at org.apache.activemq.transport.failover.FailoverTransport.doReconnect(FailoverTransport.java:870)
>         - locked <0x00002aaac2df83c8> (a java.lang.Object)
>         at org.apache.activemq.transport.failover.FailoverTransport$2.iterate(FailoverTransport.java:135)
>         - locked <0x00002aaac2df83d8> (a java.lang.Object)
>         at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:98)
>         at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)
> Found 1 deadlock.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (AMQ-3277) Deadlock with static:(failover:(...)) configuration

Posted by "Arthur Naseef (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/AMQ-3277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13038635#comment-13038635 ] 

Arthur Naseef commented on AMQ-3277:
------------------------------------

I will test with Failover using maxReconnectAttempts=1 and post an update.  Thank you.

> Deadlock with static:(failover:(...)) configuration
> ---------------------------------------------------
>
>                 Key: AMQ-3277
>                 URL: https://issues.apache.org/jira/browse/AMQ-3277
>             Project: ActiveMQ
>          Issue Type: Bug
>          Components: Broker
>    Affects Versions: 5.4.2
>         Environment: 5.4.2 with back-ported patches for AMQ-3176, AMQ-2736, AMQ-3124, AMQ-3125, AMQ-3167.
> Two brokers, each with a single duplex network connection to the other, using static:(failover:(first,second)) URI format.
>            Reporter: Arthur Naseef
>         Attachments: singleTransport.patch
>
>
> Using failover transports with static transports to create a network of brokers with H/A pairs, the failovers lead to deadlocks.
> This appears related to AMQ-3176, but the patch for that problem is included in our 5.4.2 build.
>                 <networkConnector name="nw1_from_nw2" uri="static:(failover:(tcp://127.0.0.1:60992,tcp://127.0.0.1:60002))"
>                                   duplex="true" conduitSubscriptions="false" dynamicOnly="true"
>                                   networkTTL="7" suppressDuplicateQueueSubscriptions="true"
>                                   decreaseNetworkConsumerPriority="true"
>                                   bridgeTempDestinations="true"
>                                   >
>                 </networkConnector>
> The deadlock occurs after stopping and restarting each broker once.  Here's the steps:
>     - start broker1
>     - start broker2
>     * test connectivity
>     - sthudown broker1
>     - restart broker1
>     * test connectivity
>     - shutdown broker2
>     - restart broker2
>     * test connectivity
> Here is the deadlock information from the JVM:
> Java stack information for the threads listed above:
> ===================================================
> "StartRemotelBridge: localBroker=vm://peer-nw2#2":
>         at org.apache.activemq.transport.failover.FailoverTransport.oneway(FailoverTransport.java:477)
>         - waiting to lock <0x00002aaac2df83c8> (a java.lang.Object)
>         at org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:40)
>         - locked <0x00002aaac2e050f8> (a java.lang.Object)
>         at org.apache.activemq.transport.ResponseCorrelator.oneway(ResponseCorrelator.java:60)
>         at org.apache.activemq.network.DemandForwardingBridgeSupport.startRemoteBridge(DemandForwardingBridgeSupport.java:352)
>         - locked <0x00002aaac2df7c50> (a org.apache.activemq.network.DurableConduitBridge)
>         at org.apache.activemq.network.DemandForwardingBridgeSupport$4.run(DemandForwardingBridgeSupport.java:277)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>         at java.lang.Thread.run(Thread.java:662)
> "ActiveMQ Failover Worker: 169776139":
>         at org.apache.activemq.network.DemandForwardingBridgeSupport.startLocalBridge(DemandForwardingBridgeSupport.java:290)
>         - waiting to lock <0x00002aaac2df7c50> (a org.apache.activemq.networFound one Java-level deadlock:
> =============================
> "StartRemotelBridge: localBroker=vm://peer-nw2#2":
>   waiting to lock monitor 0x0000000044796938 (object 0x00002aaac2df83c8, a java.lang.Object),
>   which is held by "ActiveMQ Failover Worker: 169776139"
> "ActiveMQ Failover Worker: 169776139":
>   waiting to lock monitor 0x00002aaac470f0b0 (object 0x00002aaac2df7c50, a org.apache.activemq.network.DurableConduitBridge),
>   which is held by k.DurableConduitBridge)
>         at org.apache.activemq.network.DemandForwardingBridgeSupport$2.transportResumed(DemandForwardingBridgeSupport.java:220)
>         at org.apache.activemq.transport.TransportFilter.transportResumed(TransportFilter.java:109)
>         at org.apache.activemq.transport.TransportFilter.transportResumed(TransportFilter.java:109)
>         at org.apache.activemq.transport.failover.FailoverTransport.doReconnect(FailoverTransport.java:870)
>         - locked <0x00002aaac2df83c8> (a java.lang.Object)
>         at org.apache.activemq.transport.failover.FailoverTransport$2.iterate(FailoverTransport.java:135)
>         - locked <0x00002aaac2df83d8> (a java.lang.Object)
>         at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:98)
>         at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)
> Found 1 deadlock.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira