Excessive number of connections by failover transport

[Andrei Shakirin <as...@talend.com>]

Hi,

Could you give any suggestion how to workaround / investigate following problem on ActiveMQ communication:

1) Versions: AMQ client 5.14.1; AMQ server 5.13.4
2) Occurrence: only sometimes
3) Connection: failover protocol using url: failover:(tcp://serverA:61616,tcp://serverB:61616)?randomize=false&priorityBackup=true
4) Client details: producers use PooledConnectionFactory, consumers - native ActiveMQConnectionFactory

5) Symptoms:
Suddenly AMQ Server log explodes with the messages like:

2018-01-26 09:26:16,909 | WARN  | Failed to register MBean org.apache.activemq :type=Broker,brokerName=activemq-vm-primary,connector=clientConnectors,connect
orName=default,connectionViewType=clientId,connectionName=ID_ca8f70e115d0-3708
7-1516883370639-0_22 | org.apache.activemq.broker.jmx.ManagedTransportConnecti
on | ActiveMQ Transport: tcp:///172.16.6.56:55548@61616

2018-01-26 09:26:21,375 | WARN  | Ignoring ack received before dispatch; result of failover with an outstanding ack. Acked messages will be replayed if present on this broker. Ignored ack: MessageAck {commandId = 157, responseRequired = false, ackType = 2, consumerId = ID:ca8f70e115d0-37087-1516883370639-1:22:10:1, firstMessageId = ID:a95345a9c0df-33771-1516883685728-1:17:5:1:23, lastMessageId = ID:a95345a9c0df-33771-1516883685728-1:17:5:1:23, destination = queue://Q.CHECKOUT.AUFTRAG_OUT, transactionId = null, messageCount = 1, poisonCause = null} | org.apache.activemq.broker.region.PrefetchSubscription | ActiveMQ Transport: tcp:///172.16.6.56:55464@61616

2018-01-26 09:26:39,211 | WARN  | Transport Connection to: tcp://172.16.6.56:55860 failed: java.net.SocketException: Connection reset | org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ InactivityMonitor Worker

2018-01-26 09:26:47,175 | WARN  | Transport Connection to: tcp://172.16.6.56:57012 failed: java.net.SocketException: Broken pipe (Write failed) | org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ InactivityMonitor Worker

After short period of time AMQ server comes out of resources with "java.lang.OutOfMemoryError: unable to create new native thread" error. The AMQ service process in this case has a huge number of threads (some thousands) 

The client side log contains a lot of reconnection attempts messages like:
2018-01-26 00:10:31,387 WARN    [{{bundle.name,org.apache.activemq.activemq-osgi}{bundle.version,5.14.1}{bundle.id,181}}]     [null]  org.apache.activemq.transport.failover.FailoverTransport      Failed to connect to [tcp://activemq-vm-primary:61616, tcp://activemq-vm-secondary:61616] after: 810 attempt(s) continuing to retry.

It seems that client creates a huge number of connections by failover retry and after some time kills the server. 
Issue looks very similar to described in https://issues.apache.org/jira/browse/AMQ-6603, however server isn't configured with access control settings.
I found the description of similar problem into http://activemq.2283324.n4.nabble.com/ActiveMQ-5-2-OutOfMemoryError-unable-to-create-new-native-thread-td2366585.html  ,  but without concrete suggestion.

The server log file is attached.

Could you please provide any suggestion how to deal with the problem? 

Regards,
Andrei.