Retry queue message delivery to another node after timeout or connection failure

[Scott Van Wart <sc...@indosoft.com>]

I can't for the life of me figure out how to get ActiveMQ Artemis to try
another node for message delivery.  I have a domain-managed wildfly cluster
with 3 nodes:

Wildfly 12 (Artemis 1.5.5) and JDK 1.8.0_131 running on Ubuntu 16.04.  I
started with the defaults.  I deployed an EAR with a single MDB that
listens to a durable queue.  I then connect to a node and send a test
message every 250ms.  I can see the messages appearing round-robin on all
nodes (and JMSDeliveryType is PERSISTENT).

The MDB is configured with dups-ok-acknowledge.  I changed some settings
from the defaults that ship with Wildfly for cluster-connection:

check-period=500
connection-ttl=1000
reconnect-attempts=2
call-timeout=1000
call-failover-timeout=500
use-duplicate-detection=false

Other relevant settings:

retry-interval=500
retry-interval-multiplier=1.5
initial-connect-attempts=-1
message-load-balancing=ON_DEMAND
notification-interval=1000
notification-attempts=2

While the test is going on, I unplug the network cable to one of the
nodes.  The other two nodes fail their 3rd node connection in about a
second and start distributing the messages across only the 2 remaining
nodes, which is fine.  But I "lose" about 2-3 messages during this time to
the failed node.  I can leave that failed node unplugged for as long as I
want.  I can even plug the failed node back in and it won't retransmit
these 2-3 messages.  Finally, I restart all the nodes and the 2-3 "lost"
messages are then transmitted (much later) and only to the failed node.

What I really want is for ActiveMQ to quickly retry delivery to another
node.  So if it attempts delivery and the message isn't acknowledged for
750-1000ms, try another node.  I can handle duplicates just fine. Am I
going about this the right way?

Thanks,
  Scott