Network of brokers setup error

[mtod <mt...@thetods.net>]

I have some issues with the network of brokers configuration I keep getting a
failure to connect.

The setup for the network of brokers is as follows:

Broker1 Master/Slave
	Node 1 (Master)
	Node 2 (Slave)
	
Broker 2 Master/Slave
	Node 1 (Master)
	Node 2 (Slave)
	
Both clusters work fine.	
		
The activemq.xml broker setup:

Broker1:

		<networkConnectors>
			<networkConnector 
				name="broker1->broker2" 
			
uri="static:(failover:(tcp://192.168.10.5:61616,tcp://192.168.10.6:61616)?randomize=false)" 
				userName="xxxxx" 
				password="xxxxx"
				duplex="true" 
				networkTTL="3" 
				dynamicOnly="true"
				/> 

Broker2:

		<networkConnectors>
			<networkConnector 
				name="broker2->broker1" 
			
uri="static:(failover:(tcp://192.168.10.3:61616,tcp://192.168.10.4:61616)?randomize=false)" 
				userName="xxxxx" 
				password="xxxxx"
				duplex="true" 				
				networkTTL="3" 
				dynamicOnly="true"
				/>
		</networkConnectors>
		
Section of the activeMQ log:

2016-08-17 13:25:58,599 | DEBUG | Attempting 2th connect to:
tcp://192.168.10.4:61616 |
org.apache.activemq.transport.failover.FailoverTransport | ActiveMQ Failover
Worker: 236039765
2016-08-17 13:25:59,616 | DEBUG | Connect fail to: tcp://192.168.10.4:61616,
reason: java.net.ConnectException: Connection refused: connect |
org.apache.activemq.transport.failover.FailoverTransport | ActiveMQ Failover
Worker: 236039765
2016-08-17 13:25:59,616 | DEBUG | Stopping transport
tcp://192.168.10.4:61616 | org.apache.activemq.transport.tcp.TcpTransport |
ActiveMQ Failover Worker: 236039765
2016-08-17 13:25:59,616 | DEBUG | Initialized TaskRunnerFactory[ActiveMQ
Task] using ExecutorService: null |
org.apache.activemq.thread.TaskRunnerFactory | ActiveMQ Failover Worker:
236039765
2016-08-17 13:25:59,616 | DEBUG | Closed socket Socket[unconnected] |
org.apache.activemq.transport.tcp.TcpTransport | ActiveMQ Task-1
2016-08-17 13:25:59,616 | DEBUG | Waiting 40 ms before attempting connection
| org.apache.activemq.transport.failover.FailoverTransport | ActiveMQ
Failover Worker: 236039765

Thanks for the help

Mike



--
View this message in context: http://activemq.2283324.n4.nabble.com/Network-of-brokers-setup-error-tp4715603.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: Network of brokers setup error

[Tim Bain <tb...@alumni.duke.edu>]

Mike,

The biggest issue with geographically distributed systems is the latency
introduced by the distances between your locations.

You'll definitely want to tune your TCP stack, including computing an
adequate TCP buffer size for the bandwidth-delay product
<https://en.wikipedia.org/wiki/Bandwidth-delay_product>.  You may also want
to tell the kernel not to change the TCP congestion window during idle
periods, as I found when I investigated some undesirable behavior in
http://stackoverflow.com/questions/25494929/3x-latency-observed-in-activemq-only-for-higher-wan-latencies.
And you should absolutely brush up on the details of how TCP works, because
understanding (or remembering, if like me it had been a while since you
learned it) the intricacies of the TCP protocol can help you understand
what's going on when you start having performance problems.

Also, you're going to want to consider geography when making decisions
about how to architect the ActiveMQ portions of your overall system
(brokers, consumers, and producers).  Your performance will generally be
best when you can avoid having to jump across a high-latency link, so where
possible you should try to localize processing to ensure that producer,
consumer, and broker(s) are all in the same part of the world.  You may
want to use the priorityBackup feature of the failover protocol to ensure
that you always connect to the nearby broker(s) whenever they are
available.  Also, where you need to cross a high-latency link, you'll have
the best results if you can do it asynchronously (so your program can go on
processing in the meantime), so ideally you'd never want a client
(especially a producer) to have to cross a high-latency link to connect to
a broker; if anything, make the brokers communicate across the high-latency
links, so the clients can be working on the next message while the transfer
is in progress.  If some of your clients (and especially your producers)
aren't located close to any of your brokers, you might want to consider
co-locating an additional broker with those clients, so the client
connection to the first broker is fast.  That could be done by deploying a
standalone broker that's always running, or by running an embedded broker
within the client process.  But be careful of the latter option; it means
that when the client is stopped, all messages on the embedded broker at
that time are no longer available to the network of brokers, which can lead
to undesirable results.  Make sure you think through all the scenarios very
carefully before you choose the embedded broker option.

The final thing to think about is that in this type of configuration,
network partitions become a very real possibility, so you should make sure
that your overall system will behave acceptably in the face of a network
partition (or that it's able to detect one and stop processing until the
problem is resolved).  Whether this is a problem and how hard it is to
resolve depends heavily on the domain of your particular problem.

Tim

On Aug 18, 2016 9:09 AM, "mtod" <mt...@thetods.net> wrote:

> Thanks Tim
>
> I have it working now with the masterslave option.
>
> Question: I'm setting up a masterslave cluster in Germany, China and other
> US Based locations is there any best practices for distributed clusters?
>
> Mike
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.nab
> ble.com/Network-of-brokers-setup-error-tp4715603p4715639.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>

Re: Network of brokers setup error

[mtod <mt...@thetods.net>]

Thanks Tim

I have it working now with the masterslave option.

Question: I'm setting up a masterslave cluster in Germany, China and other
US Based locations is there any best practices for distributed clusters?

Mike



--
View this message in context: http://activemq.2283324.n4.nabble.com/Network-of-brokers-setup-error-tp4715603p4715639.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: Network of brokers setup error

[Tim Bain <tb...@alumni.duke.edu>]

The static:failover: transport pair only works for networkConnectors when
you specify the maxReconnectAttempts=0 option on the failover transport.
This is not needed for client connections, only for broker-to-broker
networkConnectors.  Or you could just use the masterslave: transport, which
is syntactic sugar for static:failover: with maxReconnectAttempts=0.

More details are in the links I quoted in
http://activemq.2283324.n4.nabble.com/Static-network-connectors-and-maxReconnectAttempts-td4685835.html,
if you're interested.

We really should log a warning when someone does this, so I've submitted
AMQ-6399 <https://issues.apache.org/jira/browse/AMQ-6399> to propose that
we do that.

Tim

On Wed, Aug 17, 2016 at 3:05 PM, mtod <mt...@thetods.net> wrote:

> Additional info I left out.
>
> This is running ActiveMQ 5.14 in a SQL lock / Kahadb configuration on
> Windows 2012 core servers.
>
>
>
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.
> nabble.com/Network-of-brokers-setup-error-tp4715603p4715604.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>

Re: Network of brokers setup error

[mtod <mt...@thetods.net>]

Additional info I left out.

This is running ActiveMQ 5.14 in a SQL lock / Kahadb configuration on
Windows 2012 core servers.






--
View this message in context: http://activemq.2283324.n4.nabble.com/Network-of-brokers-setup-error-tp4715603p4715604.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.