[jira] [Created] (AMQ-6457) Durable Subscribers going offline - Over Network Of Brokers - Duplex Connection

["Ranjit Nethi (JIRA)" <ji...@apache.org>]

Ranjit Nethi created AMQ-6457:
---------------------------------

             Summary: Durable Subscribers going offline - Over Network Of Brokers - Duplex Connection
                 Key: AMQ-6457
                 URL: https://issues.apache.org/jira/browse/AMQ-6457
             Project: ActiveMQ
          Issue Type: Bug
          Components: activemq-camel, networkbridge
    Affects Versions: 5.14.0, 5.13.3
         Environment: Active MQ Version 5.13.3
Karaf 4.0.6 
Camel 2.16.3 

-Active MQ brokers are deployed in Karaf container. 
-Cluster of Active MQ Brokers (Network of brokers, configured as HUB and SPOKES).
-Broker(s) at Hub (Master/Slave configuration using Shared File System)
- Broker(s) at spoke(s)- Connected to HUB using Duplex Network Connector ( Openwire/NIO) 
            Reporter: Ranjit Nethi


-Active MQ brokers are deployed in Karaf container. 
-Cluster of Active MQ Brokers (Network of brokers, configured as HUB and SPOKES).
-Broker(s) at Hub (Master/Slave configuration using Shared File System)
- Broker(s) at spoke(s)- Connected to HUB using Duplex Network Connector ( Openwire/NIO), one for topic and one for queue

Issue: We have durable subscriptions to topics on HUB and producers on Spokes. We are aware of network outages between the HUB and SPOKE brokers, sometimes we are experiencing multiple outages and as a result we see the topic subscribers end up as offline  and continue to store messages as the spoke broker. We are need a restart of the broker to reestablish the network bridge.

Relevant Logs: 

2016-09-22 14:21:40,105 | WARN  | tyMonitor Worker | DemandForwardingBridgeSupport    | 20 - org.apache.activemq.activemq-osgi - 5.13.3 | Network connection between vm://central-amq-broker#44 and tcp:///xx.xxx.xx.xx:53103@61616 shutdown due to a remote error: org.apache.activemq.transport.InactivityIOException: Channel was inactive for too (>30000) long: tcp:///xx.xxx.xx.xx:53103

2016-09-22 14:21:40,117 | INFO  | -broker] Task-21 | DemandForwardingBridgeSupport    | 20 - org.apache.activemq.activemq-osgi - 5.13.3 | central-amq-broker bridge to T1-amq-broker stopped

2016-09-22 14:23:44,246 | INFO  | eMQ NIO Worker 4 | TransportConnection              | 20 - org.apache.activemq.activemq-osgi - 5.13.3 | Started responder end of duplex bridge T:broker1->broker2@ID:T1-35496-1474555572634-0:1
2016-09-22 14:23:44,255 | INFO  | al-amq-broker#52 | DemandForwardingBridgeSupport    | 20 - org.apache.activemq.activemq-osgi - 5.13.3 | Network connection between vm://central-amq-broker#52 and tcp:///xx.xxx.xx.xx:53162@61616 (T1-amq-broker) has been established.

Network Connector on Spoke Broker:


<networkConnectors>
         	<networkConnector 
	            name="T:broker1->broker2"
				userName="karaf"
		        password="karaf"
		        uri="masterslave:(tcp://${ACTIVEMQ_HEAD_1_IP_ADDRESS}:61616,tcp://${ACTIVEMQ_HEAD_2_IP_ADDRESS}:61616)"
        	    duplex="true"
	            decreaseNetworkConsumerPriority="true"
	            networkTTL="2"
        	    dynamicOnly="true">
	            <excludedDestinations>
        	          <queue physicalName="&gt;" />
        	          <topic physicalName="Shop.Pos.Api.Outbound" />
	            </excludedDestinations>
	         </networkConnector>
	         <networkConnector 
	            name="Q:broker1->broker2" 
				userName="karaf"
	            password="karaf"
	            uri="masterslave:(tcp://${ACTIVEMQ_HEAD_1_IP_ADDRESS}:61616,tcp://${ACTIVEMQ_HEAD_2_IP_ADDRESS}:61616)"
	            duplex="true"
	            decreaseNetworkConsumerPriority="true"
	            networkTTL="2"
	            dynamicOnly="true">
	            <excludedDestinations>
	                  <topic physicalName="&gt;" />
	            </excludedDestinations>
	         </networkConnector>
		</networkConnectors>


Test to Recreate:
To recreate this issue, we can block the network between the two hosts while simulating some load





--
This message was sent by Atlassian JIRA
(v6.3.4#6332)